An on-board oriented mixed reality display system and method

Abstract

The application relates to an airborne mixed reality display system and method, which comprises a laser positioning base station for sending horizontal-axis laser rays and vertical-axis laser rays; a first laser positioning sensor fixedly arranged on a cabin; a second laser positioning sensor arranged on a head-mounted device; an inertial measurement sensor arranged on the head-mounted device; and a computer for fusing first position information and a motion track with second position information through a Kalman filtering algorithm to obtain real-time position of the head-mounted device; the head-mounted device is used for unifying the real-time position of the head-mounted device, current pose information of the cabin and world coordinates of a virtual target in the same coordinate. The laser positioning result and the IMU positioning result are fused through the Kalman filtering technology, so that real-time and accurate positioning of the mixed reality head-mounted device is realized. The error existing in head tracking of the airborne AR is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mixed reality application, in particular to an airborne mixed reality display system and method. BACKGROUND

[0002] In airborne engineering application, the wide field of view of the outside scene cannot be obtained in the airborne cabin environment, and in order to obtain enhanced situational awareness, the wide field of view of the outside scene obtained by the external image sensor is needed. The US military uses augmented reality in the display of the pilot's cabin, superimposes the enhanced target map in the augmented reality display helmet to form a panoramic extended reality perspective display system that moves with the pilot's helmet, and superimposes the vector graphics into the pilot's field of view, which not only provides navigation information to the pilot, but also provides enhanced battlefield information including hidden enemy forces, greatly enhancing the pilot's situational awareness. In addition, in terms of aviation confrontation training, augmented reality can be used for confrontation simulation, and through augmented reality technology, virtual enemy aircraft is synthesized during pilot training to replace professional dummy aircraft, greatly reducing the cost of air combat training.

[0003] The key technologies in the augmented reality system include tracking and positioning technology and three-dimensional registration technology, wherein the tracking and positioning technology is mainly used to determine the relative position and direction between the user and the surrounding real environment, and to realize the "alignment" problem between the virtual scene and the real environment. The three-dimensional registration mainly completes the work of estimating the camera pose, and superimposes the virtual information to be added into the augmented real scene in real time, to realize the seamless fusion of the virtual environment and the real environment. The three-dimensional registration process mainly includes the spatial positioning tracking of the user's head (camera) and the positioning of the virtual object in the real space.

[0004] According to the SAE G-10HWD (Head-Mounted Display) committee, a new proposal based on ARP 5288 (HUD) has been studied, namely Aerospace Recommended Practice 6377, which proposes a geographical consistency accuracy requirement of 5 milliradians (about 0.28°) related to flight safety, so as to realize the augmented reality display of the airborne environment, the augmented reality display system must have accurate positioning technology and three-dimensional registration display. However, unlike the use environment of most civil AR products, since the airborne AR system involves tracking and positioning among the outside target, the cabin and the pilot, accurate processing is needed step by step, first the system needs to accurately calculate the attitude of the outside target relative to the airborne camera system, and then calculate the attitude information of the pilot relative to the cabin, and finally calculate the attitude information of the outside target relative to the pilot through step-by-step derivation. However, the error of the existing airborne AR augmented display mainly comes from three aspects: aircraft attitude error, head tracking error, and three-dimensional registration display error. SUMMARY

[0005] In view of the above problems, the present application provides an airborne mixed reality display system and method, which solves the problem of head tracking error in the existing AR augmented display.

[0006] To achieve the above-mentioned purpose, the present application provides an airborne mixed reality display system, comprising:

[0007] two laser positioning base stations for sending horizontal-axis laser rays and vertical-axis laser rays;

[0008] a cockpit;

[0009] a first laser positioning sensor fixedly arranged on the cockpit, the first laser positioning sensor being configured to calculate the current cockpit pose of the cockpit according to the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base stations;

[0010] a head-mounted device;

[0011] a second laser positioning sensor arranged on the head-mounted device, the second laser positioning sensor being configured to obtain first position information and a motion trajectory of the head-mounted device according to the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base stations;

[0012] an inertial measurement sensor arranged on the head-mounted device, the inertial measurement sensor being configured to obtain second position information of the head-mounted device;

[0013] a computer configured to fuse the first position information and the motion trajectory of the head-mounted device obtained by the second laser positioning sensor and the second position information of the head-mounted device obtained by the inertial measurement sensor by using a Kalman filtering algorithm to obtain a real-time position of the head-mounted device, and send the world coordinates of the virtual target, the current pose of the cockpit and the real-time position of the head-mounted device to the head-mounted device;

[0014] the head-mounted device is configured to unify the real-time position of the head-mounted device, the current pose information of the cockpit and the world coordinates of the virtual target in the same coordinate by comparing and calculating the world coordinates of the virtual target with the current pose of the cockpit.

[0015] In some embodiments, the computer is further configured to initialize the current position and the covariance error of the head-mounted device using the initial position of the head-mounted device, and use the motion trajectory of the head-mounted device collected by the second laser positioning sensor as the system input u k at the current time. kThe value pair predicts the position of the next moment to obtain a system prediction value, and a predicted covariance error is calculated, the second position information acquired by the inertial measurement sensor is taken as a measurement value of the system, and a Kalman gain K is calculated k Then, the system prediction value is corrected using the system measurement value for data fusion, and the Kalman gain K is used k The current position is updated to obtain the real-time position of the head-mounted device.

[0016] In some embodiments, the second laser positioning sensor is configured to acquire a horizontal-axis angle and a vertical-axis angle relative to the laser positioning base station according to measured times of arrival of the horizontal-axis laser ray and the vertical-axis laser ray transmitted by the laser positioning base station.

[0017] The computer is configured to calculate the first position information and the motion trajectory of the head-mounted device according to a position difference between the first laser positioning sensor and the second laser positioning sensor.

[0018] In some embodiments, the computer is further configured to send the local flight speed, the local flight height, the local six-dimensional pose, the task performance time, and the Beijing Time to the head-mounted device.

[0019] The head-mounted device is configured to generate a flight visualization interface according to the data sent by the computer.

[0020] In some embodiments, the head-mounted device is further configured to calculate a real-time distance between the virtual target and the cockpit according to the world coordinates of the virtual target and the position coordinates of the cockpit, and mark the calculated real-time distance above the virtual graphics corresponding to the virtual target.

[0021] Another technical solution is also provided, which is a mixed reality display method for an airborne device, including the following steps:

[0022] A first laser positioning sensor arranged above the cockpit receives horizontal-axis laser rays and vertical-axis laser rays transmitted by a laser positioning base station to calculate a current cockpit pose of the cockpit.

[0023] A second laser positioning sensor arranged on the head-mounted device receives the horizontal-axis laser rays and the vertical-axis laser rays transmitted by the laser positioning base station to acquire first position information and a motion trajectory of the head-mounted device.

[0024] An inertial measurement sensor arranged on the head-mounted device acquires second position information of the head-mounted device.

[0025] The first position information and motion trajectory of the head-mounted device acquired by the second laser positioning sensor are fused with the second position information of the head-mounted device acquired by the inertial measurement sensor through a Kalman filtering algorithm to obtain a real-time position of the head-mounted device, and the world coordinates of the virtual target, the current pose of the cockpit and the real-time position of the head-mounted device are sent to the head-mounted device.

[0026] The head-mounted device compares and calculates the world coordinates of the virtual target with the current pose of the cockpit, and unifies the real-time position of the head-mounted device, the current pose information of the cockpit and the world coordinates of the virtual target in the same coordinate.

[0027] In some embodiments, the step of "fusing the first position information and motion trajectory of the head-mounted device acquired by the second laser positioning sensor with the second position information of the head-mounted device acquired by the inertial measurement sensor through a Kalman filtering algorithm to obtain a real-time position of the head-mounted device" specifically includes the following steps:

[0028] The current position and covariance error of the head-mounted device are initialized using the initial position of the head-mounted device;

[0029] The motion trajectory of the head-mounted device collected by the second laser positioning sensor is taken as the system input u k at the current time, and the position at the next time is predicted using the u k value to obtain a system prediction value;

[0030] The predicted covariance error is calculated at the same time, the second position information acquired by the inertial measurement sensor is taken as the measurement value of the system, and the Kalman gain K k is calculated;

[0031] Then, the system prediction value is corrected using the system measurement value to perform data fusion, and the Kalman gain K k is used to update the current position to obtain the real-time position of the head-mounted device.

[0032] In some embodiments, the step of "receiving the horizontal-axis laser rays and vertical-axis laser rays transmitted by the laser positioning base station through the second laser positioning sensor arranged on the head-mounted device to acquire the first position information and motion trajectory of the head-mounted device" specifically includes the following steps:

[0033] The second laser positioning sensor acquires the horizontal-axis angle and vertical-axis angle relative to the laser positioning base station according to the time of measuring the horizontal-axis laser rays and vertical-axis laser rays transmitted by the laser positioning base station;

[0034] The first position information and motion trajectory of the head-mounted device are calculated according to the position difference between the first laser positioning sensor and the second laser positioning sensor.

[0035] In some embodiments, the method further comprises the following steps:

[0036] The local flight speed, the local flight height, the local six-dimensional pose, the task time and the Beijing time are sent to the head-mounted device.

[0037] The head-mounted device generates a flight visualization interface according to the data sent by the computer.

[0038] In some embodiments, the method further comprises the following steps:

[0039] The head-mounted device calculates the real-time distance between the virtual target and the cockpit according to the world coordinates of the virtual target and the position coordinates of the cockpit, and labels the calculated real-time distance above the virtual graphics corresponding to the virtual target.

[0040] Unlike the prior art, the above technical solution sends horizontal axis laser rays and vertical axis laser rays through a laser positioning base station, receives the horizontal axis laser rays and the vertical axis laser rays sent by the laser positioning base station through the first laser positioning sensor arranged above the cockpit to solve the current cockpit pose of the cockpit, and receives the horizontal axis laser rays and the vertical axis laser rays sent by the laser positioning base station through the second laser positioning sensor arranged on the head-mounted device to obtain the first position information and the motion trajectory of the head-mounted device. The second position information of the head-mounted device is obtained through the inertial measurement sensor arranged on the head-mounted device, and then the computer fuses the first position information and the motion trajectory of the head-mounted device obtained by the second laser positioning sensor with the second position information of the head-mounted device obtained by the inertial measurement sensor through Kalman filtering algorithm to obtain the real-time position of the head-mounted device, and sends the world coordinates of the virtual target, the current pose of the cockpit and the real-time position of the head-mounted device to the head-mounted device. The head-mounted device compares and calculates the real-time position of the head-mounted device, the current pose information of the cockpit and the world coordinates of the virtual target in the same coordinate. By using the tracking and positioning method based on Lighthouse, the tracking and positioning among the external target, the cockpit of the aircraft and the visual system of the pilot are realized. The Kalman filtering technology is used to fuse the Lighthouse positioning result and the IMU positioning result, so as to realize the real-time and accurate positioning of the mixed reality head-mounted device. The error existing in the head tracking of the airborne AR is reduced.

[0041] The above content is only a summary of the technical solution of the present application. In order to enable those skilled in the art to more clearly understand the technical solution of the present application, and then implement the content recorded in the specification and drawings, and in order to make the above and other purposes, characteristics and advantages of the present application more easily understood, the following will be described in combination with the specific embodiments of the present application and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0042] The accompanying drawings are only used to illustrate the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and cannot be considered as limitations to the present application.

[0043] In the drawings:

[0044] Figure 1 A structural schematic diagram of the airborne mixed reality display system according to the specific embodiments;

[0045] Figure 2 Another structural schematic diagram of the airborne mixed reality display system according to the specific embodiments;

[0046] Figure 3 Another structural schematic diagram of the airborne mixed reality display system according to the specific embodiments;

[0047] Figure 4 Another structural schematic diagram of the airborne mixed reality display system according to the specific embodiments;

[0048] Figure 5 A scene diagram of the scene application interface native position information display according to the specific embodiments;

[0049] Figure 6 A scene diagram of the scene application interface enemy information display according to the specific embodiments;

[0050] Figure 7 A scene diagram of the scene application interface task time information display according to the specific embodiments;

[0051] Figure 8 A scene diagram of the scene application interface native state information display according to the specific embodiments;

[0052] Figure 9 A scene diagram of the scene application interface eye movement tracking function according to the specific embodiments;

[0053] Figure 10 A flowchart of the airborne mixed reality display method according to the specific embodiments.

[0054] The reference signs involved in the above drawings are explained as follows:

[0055] 110, laser positioning base station,

[0056] 120, cockpit,

[0057] 130, first laser positioning sensor;

[0058] 140, head-mounted device,

[0059] 150, second laser positioning sensor;

[0060] 160, inertial measurement sensor;

[0061] 170, computer. DETAILED DESCRIPTION

[0062] To make the possible application scenarios, technical principles, specific schemes that can be implemented, purposes and effects achieved, etc. of the present application clear, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical schemes of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0063] In this paper, the term "embodiment" means that the specific features, structures or characteristics described in combination with the embodiment can be included in at least one embodiment of the present application. The term "embodiment" appearing at various positions in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, each technical feature mentioned in each embodiment can be combined in any way to form a corresponding implementable technical scheme.

[0064] Unless otherwise defined, the meaning of the technical terms used herein is the same as that generally understood by those skilled in the art to which the present application belongs; the use of related terms herein is only for the purpose of describing specific embodiments, and is not intended to limit the present application.

[0065] In the description of the present application, the phrase "and / or" is a description of the logical relationship between the objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in this paper generally represents that the associated objects before and after are a kind of "or" logical relationship.

[0066] In the present application, such as "first" and "second", the terms are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0067] In the present application, the terms "comprise", "contain", "include", or other similar forms are intended to cover non-exclusive inclusions, and do not exclude the presence of additional elements in the process, method or product comprising the stated elements, so that the process, method or product comprising a series of elements can not only include those defined elements, but also include other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0068] In the present application, the terms "greater than", "less than", "exceed" and the like are understood as not including the number itself; the terms "above", "below", "within" and the like are understood as including the number itself. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly specified.

[0069] In the description of the embodiments of the present application, the spatial-related expressions used, such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "perpendicular", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like, indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or the drawings, and are only for the convenience of describing the specific embodiments of the present application or for the reader to understand, and do not indicate or imply that the indicated device or component must have a specific position, a specific orientation, or be constructed or operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0070] Unless otherwise explicitly specified or limited, in the description of the embodiments of the present application, the terms "mount", "connect", "connect", "fix", "set" and the like should be interpreted broadly. For example, the "connection" can be a fixed connection, or a detachable connection, or an integral setting; it can be a mechanical connection, or an electrical connection, or a communication connection; it can be directly connected, or indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art to which the present application belongs, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0071] Head-mounted device is the abbreviation of head-mounted display device, and all head-mounted display devices can be called head-mounted. Through various head-mounted display devices, optical signals are sent to the eyes in different ways, which can achieve different effects such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0072] Referring to Figures 1-3 The embodiment provides an airborne mixed reality display system, comprising:

[0073] laser positioning base stations 110, two in number, for sending horizontal-axis laser rays and vertical-axis laser rays;

[0074] a cockpit 120;

[0075] a first laser positioning sensor 130 fixedly arranged on the cockpit 120, the first laser positioning sensor 130 being used for solving a current cockpit pose of the cockpit 120 according to the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base stations 110;

[0076] a head-mounted device 140;

[0077] a second laser positioning sensor 150 arranged on the head-mounted device 140, the second laser positioning sensor being used for acquiring first position information and a motion trail of the head-mounted device 140 according to the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base stations 110;

[0078] an inertial measurement sensor 160 arranged on the head-mounted device 140, the inertial measurement sensor being used for acquiring second position information of the head-mounted device 140; wherein the inertial measurement sensor 160 is connected to a computer 170 through a USB data line, the USB data line can be a USB-C data line, in other embodiments, a USB-B data line can also be used, or the inertial measurement sensor 160 is connected to the computer 170 through a wireless receiver.

[0079] the computer 170 being used for fusing the first position information and the motion trail of the head-mounted device 140 acquired by the second laser positioning sensor 150 and the second position information of the head-mounted device 140 acquired by the inertial measurement sensor 160 through a Kalman filtering algorithm, obtaining a real-time position of the head-mounted device 140, and sending a world coordinate of a virtual target, a current pose of the cockpit 120 and the real-time position of the head-mounted device 140 to the head-mounted device 140;

[0080] the head-mounted device 140 being used for comparing and calculating the real-time position of the head-mounted device 140, the current pose information of the cockpit 120 and the world coordinate of the virtual target in the same coordinate according to comparison and calculation of the world coordinate of the virtual target and the current pose of the cockpit 120.

[0081] The laser positioning base station 110 sends horizontal and vertical laser beams. A first laser positioning sensor 130 mounted on the cockpit 120 receives these beams and calculates the current cockpit pose. A second laser positioning sensor 150 mounted on the head-mounted display 140 receives these beams and obtains the first position information and motion trajectory of the head-mounted display 140. An inertial measurement sensor 160 mounted on the head-mounted display 140 obtains the second position information of the head-mounted display 140. Then, a computer 1... 70 uses a Kalman filter algorithm to fuse the first position information and motion trajectory of the head-mounted display device 140 obtained by the second laser positioning sensor 150 with the second position information of the head-mounted display device 140 obtained by the inertial measurement sensor 160, thereby obtaining the real-time position of the head-mounted display device 140. The world coordinates of the virtual target, the current pose of the cockpit 120, and the real-time position of the head-mounted display device 140 are then sent to the head-mounted display device 140. The head-mounted display device 140 compares and calculates the world coordinates of the virtual target with the current pose of the cockpit 120, unifying the real-time position of the head-mounted display device 140, the current pose of the cockpit 120, and the world coordinates of the virtual target under the same coordinate system. By using a Lighthouse-based tracking and positioning method, tracking and positioning among external targets, the aircraft cockpit 120, and the pilot's visual system are achieved. Kalman filtering technology is used to fuse the Lighthouse positioning results and the IMU positioning results, thereby achieving real-time and accurate positioning of the mixed reality head-mounted device. This reduces errors in head tracking for airborne AR.

[0082] In some embodiments, the computer 170 is further configured to initialize the current position and covariance error of the head-mounted display device 140 using the initial position of the head-mounted display device 140, and use the motion trajectory of the head-mounted display device 140 collected by the second laser positioning sensor 150 as the system input u at the current moment. k , with u k The system predicts the position at the next moment, obtains the system prediction value, and calculates the prediction covariance error. Using the second position information acquired by the inertial measurement sensor 160 as the system measurement value, the Kalman gain K is calculated. k Then, the system measurements are used to correct the system predictions, and data fusion is performed, while using Kalman gain K. k The current location is updated to obtain the real-time location of the head-mounted display device 140.

[0083] The extended Kalman filter algorithm is used for positioning the mixed reality head-mounted device 140. First, the initial position of the mixed reality head-mounted device 140 is used to initialize the current position and covariance error; in the prediction part, the motion trajectory value of the mixed reality head-mounted device 140 collected by the first laser positioning sensor 130 laser scanning is taken as the system input u k of the current time, the position of the next time is predicted with this value, the system prediction value is obtained, and the predicted covariance error is calculated; then, the second position information obtained by the IMU is taken as the measurement value of the system, and the Kalman gain K k is calculated; then, the system prediction value is corrected using the system measurement value, that is, the positioning data obtained by laser scanning is corrected using the position information obtained by the IMU, data fusion is performed, and the current position information is updated by the Kalman gain K k .

[0084] In some embodiments, the second laser positioning sensor 150 is configured to obtain the horizontal axis angle and the vertical axis angle relative to the laser positioning base station 110 according to the time when the horizontal axis laser ray and the vertical axis laser ray transmitted by the laser positioning base station 110 arrive.

[0085] The computer 170 is configured to calculate the first position information and the motion trajectory of the head-mounted device 140 according to the position difference between the first laser positioning sensor 130 and the second laser positioning sensor 150.

[0086] After the LED light of the laser positioning base station 110 flashes, the second laser positioning sensor 150 distributed on the head-mounted device 140 can measure the time when the horizontal axis laser and the vertical axis laser respectively arrive at the sensor. This time is exactly the time when the horizontal axis laser and the vertical axis laser reach the angle of this specific second laser positioning sensor 150, so as to obtain the horizontal axis angle and the vertical axis angle of the second laser positioning sensor 150 relative to the laser positioning base station 110. The position of the second laser positioning sensor 150 distributed on the head-mounted device is also known. Then, the position information and the motion trajectory of the head-mounted device 140 can be calculated through the position difference between the first laser positioning sensor 130 and the second laser positioning sensor 150.

[0087] In some embodiments, the computer 170 is further configured to send the local flight speed, the local flight height, the local six-dimensional pose, the task performance time, and the Beijing time to the head-mounted device 140.

[0088] The head-mounted device 140 is configured to generate a flight visualization interface according to the data sent by the computer 170.

[0089] The computer 170 obtains the world coordinates of the virtual target, the flight speed of the aircraft, the flight height of the aircraft, the six-dimensional pose of the aircraft, the task time, the Beijing time, and the like in real time by using the internal interface of the air combat simulation software, and sends the data to the mixed reality head-mounted device 140 through the UDP communication protocol; the virtual display control software deployed in the head-mounted device 140 processes the received data, generates a flight visualization interface, and provides the pilot with real-time battlefield situation information.

[0090] In some embodiments, the head-mounted device 140 is further configured to calculate the real-time distance between the virtual target and the cockpit 120 according to the world coordinates of the virtual target and the position coordinates of the cockpit 120, and mark the calculated real-time distance above the virtual graphics corresponding to the virtual target.

[0091] The positioning and three-dimensional registration software for airborne mixed reality display deployed in the head-mounted device 140 calculates the real-time distance between the virtual cockpit and the virtual target by means of the coordinates of the virtual target and the virtual cockpit, and displays the real-time distance above the virtual graphics marking the position of the virtual target.

[0092] In some embodiments, the head-mounted device 140 is further configured to perform audio pre-warning according to the received own-ship lock information. The positioning and three-dimensional registration software for airborne mixed reality display deployed in the head-mounted device 140 performs audio warning according to the received own-ship lock information.

[0093] The aircraft attitude information and the battlefield situation information are obtained in real time by using the internal interface of the air combat simulation software, the information is sent to the mixed reality head-mounted device through the local area network data communication protocol, and the real-time display of the own-ship information and the battlefield situation information is realized in the virtual display control system. The pilot in a complex simulated air combat battlefield is not only provided with navigation information, but also provided with enhanced battlefield information including hidden enemy forces, which greatly enhances the situational awareness ability of the pilot.

[0094] In some embodiments, the IMU (i.e. inertial measurement sensor 160) is connected to the computer 170 through a USB-C data line; the Lighthouse positioning sensor (the first laser positioning sensor 130 and the second laser positioning sensor 150) is positioned by receiving the infrared signal of the Lighthouse base station (laser positioning base station 110), which needs to be connected to a power supply and placed at a height of about two meters with a support, and the computer 170 receives the signal of the Lighthouse base station and the data of the Lighthouse positioning sensor through a wireless receiver, and the Lighthouse positioning sensor, the Lighthouse base station and the wireless receiver should be placed in accordance with the principle that they should be as far away from each other as possible without being blocked. The IMU (inertial measurement sensor 160) and the Lighthouse positioning sensor (second laser positioning sensor 150) are fixed on the mixed reality head-mounted device (head-mounted device 140) through a metal skeleton, the IMU is placed in a plastic box on the skeleton, and the Lighthouse positioning sensor (second laser positioning sensor 150) is fixed on the skeleton through a special screw, and the whole skeleton is adjusted in tightness by a special screw and clamped on the headband of the mixed reality head-mounted device (head-mounted device 140).

[0095] Please refer to Figures 1-4 In some embodiments, a mixed reality display system for airborne, i.e. a multi-source fusion positioning and three-dimensional registration system for airborne mixed reality display, is composed of a multi-source fusion positioning sensor unit, a multi-source fusion positioning computing unit and a head-mounted unit.

[0096] The multi-source fusion positioning sensor unit includes two Lighthouse positioning base stations (laser positioning base station 110) fixed above the cabin, which continuously send X-axis and Y-axis laser beams outward, and the Lighthouse positioning sensor can calculate the current pose in real time according to the laser scanning rays in space. Three Lighthouse positioning sensors (i.e. three first laser positioning sensors 130) are fixed on the surface of the cabin, which are responsible for calculating the current pose of the cabin. One Lighthouse positioning sensor (second laser positioning sensor 150) and one IMU positioning sensor (inertial measurement sensor 160) are fixed on the head-mounted device 140, which are responsible for calculating the current pose of the head-mounted device; the data of the four laser positioning sensors (i.e. three first laser positioning sensors 130 and one second laser positioning sensor 150) is received through a wireless receiver, and the wireless receiver and the inertial measurement sensor 160 are connected to the multi-source fusion computer unit through a USB line.

[0097] The multi-source fusion positioning calculation unit comprises a multi-source data fusion software and a three-dimensional registration software. The multi-source data fusion software receives raw data of three cabin Lighthouse positioning sensors (first laser positioning sensor 130), one head-mounted Lighthouse positioning sensor (second laser positioning sensor 150), and one head-mounted IMU positioning sensor (inertial measurement sensor 160), and then performs filtering and fusion processing. The three-dimensional registration software performs coordinate conversion and outputs high-precision relative coordinate data of the cabin, the head-mounted device, and the virtual target.

[0098] The head-mounted unit (i.e., the head-mounted device 140) comprises a mixed reality head-mounted device and a head-mounted display control software disposed thereon. The head-mounted display control software analyzes flight simulation data through the mixed reality head-mounted device and outputs an augmented reality picture of the virtual target. The user can see the final augmented reality picture through the head-mounted unit.

[0099] In another embodiment, the HoloLens2 is selected as the mixed reality head-mounted device (head-mounted device 140). Two Lighthouse positioning base stations (laser positioning base stations 110) are fixed above the cabin and continuously send X-axis and Y-axis laser beams. Three Tracker positioning sensors (first laser positioning sensor 130) are fixed on the cabin (calibration is completed before testing), one Tracker positioning sensor (second laser positioning sensor 150) and one IMU positioning sensor (inertial measurement sensor 160) are fixed on the head-mounted device. The four Tracker positioning sensors are connected to the Tracker wireless receiver through Bluetooth, and the wireless receiver and the IMU positioning sensor are connected to the test computer A through a USB line. The test computer A and the test computer B are connected to the interactive machine through network cables, respectively, to form a local area network. The simulated air combat data is derived from the DCS-World simulated air combat software.

[0100] The preparation work to be completed is as follows:

[0101] (1) The IMU is connected to the computer through a USB-C data line.

[0102] (2) The tracker is positioned through the infrared signal of the receiving base station. The base station needs to be connected to the power supply and placed at a height of about two meters with the help of a support. The computer receives the signal of the base station and the data of the tracker through the wireless receiver. The placement of the tracker, the base station, and the wireless receiver should follow the principle that there should be no obstacles between any two of them. The wireless receiver cannot be more than 2 meters away from the tracker and the base station.

[0103] (3) The tracker needs to complete device pairing in SteamVR on the computer. When the connection is complete, the AR device, base station and tracking device icons in the SteamVR interface on the computer will light up and not flash.

[0104] (4) The IMU and tracker are fixed to the HoloLens 2 via a metal frame. The IMU is placed in a plastic box on the frame, and the tracker is fixed to the frame with special screws. The entire frame is clamped to the HoloLens 2's headband using special screws for adjustment. Figure 5 , Figure 6 As shown;

[0105] (5) A fully charged HoloLens 2 that has been eye-tracked and calibrated. If the battery is low, connect the HoloLens 2 to a power source using a USB-C cable or connect it to a computer to charge.

[0106] (6) The DCS-World air combat simulation software is running and has started combat, with at least one enemy aircraft set. UDP communication is established between the computer and HoloLens2. The world coordinates of the virtual target, the flight speed of the host, the flight altitude of the host, the six-dimensional pose of the host, the mission duration, and the East 8 time zone are transmitted to the mixed reality head-mounted device in real time via the UDP communication protocol.

[0107] Wearing HoloLens 2, open the mixed reality headset's multi-source fusion-based localization, 3D registration, and eye-tracking software, enable eye-tracking input, and enter the software's scene application interface. Use the keyboard's arrow keys to control the aircraft's flight attitude in the DCS client: such as... Figure 5 The scene application interface shown displays the local pose information scene diagram, and the corresponding local pitch angle, yaw angle, and roll angle information in the scene are displayed correctly; for example... Figure 6 The scene shown is an application interface displaying enemy aircraft information. The corresponding enemy target information in the scene is visualized. Figure 6 In the diagram, the triangle represents the location of the enemy target, the number next to the triangle indicates the distance between the enemy target and our target, and the arrow in the middle indicates the direction of the enemy target within our field of vision; for example... Figure 7 The scene diagram shown illustrates the task time information display in the application interface. The corresponding task time information is visualized, including the time in East Eighth Time Zone (UTC+8) and the time the task has been in progress (in the diagram, the task has just started 1 second, and the current UTC+8 time is 21:28:05). Figure 8The scene application interface shown displays scene graph information, and the corresponding native state information in the scene is visually displayed, including the native height and speed (the left side of the figure represents the speed, and the right side represents the height); for example Figure 9 The scene application interface eye tracking function scene diagram shown, the user gazes at the triangle on the display interface that represents the position of the enemy target, and the small ball moves to the gazed triangle. If they coincide, the triangle (orange) will turn yellow.

[0108] Please refer to Figure 10 An airborne-oriented mixed reality display method, comprising the following steps:

[0109] Step S210: receiving the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base station through the first laser positioning sensor arranged above the cockpit to solve the current cockpit pose of the cockpit;

[0110] Step S220: receiving the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base station through the second laser positioning sensor arranged on the head-mounted device to obtain the first position information and the motion trajectory of the head-mounted device;

[0111] Step S230: obtaining the second position information of the head-mounted device through the inertial measurement sensor arranged on the head-mounted device;

[0112] Step S240: fusing the first position information and the motion trajectory of the head-mounted device obtained by the second laser positioning sensor and the second position information of the head-mounted device obtained by the inertial measurement sensor through the Kalman filtering algorithm to obtain the real-time position of the head-mounted device, and sending the world coordinates of the virtual target, the current pose of the cockpit, and the real-time position of the head-mounted device to the head-mounted device;

[0113] Step S250: the head-mounted device compares and calculates the world coordinates of the virtual target with the current pose of the cockpit, and unifies the real-time position of the head-mounted device, the current pose information of the cockpit, and the world coordinates of the virtual target in the same coordinate.

[0114] The horizontal axis laser ray and the vertical axis laser ray are sent by a laser positioning base station, the horizontal axis laser ray and the vertical axis laser ray sent by the laser positioning base station are received by a first laser positioning sensor arranged above the cockpit to solve the current cockpit pose of the cockpit, the horizontal axis laser ray and the vertical axis laser ray sent by the laser positioning base station are received by a second laser positioning sensor arranged on the head-mounted device to obtain the first position information and the motion trajectory of the head-mounted device; the second position information of the head-mounted device is obtained by an inertial measurement sensor arranged on the head-mounted device, then the first position information and the motion trajectory of the head-mounted device obtained by the second laser positioning sensor and the second position information of the head-mounted device obtained by the inertial measurement sensor are fused by a computer through a Kalman filtering algorithm to obtain the real-time position of the head-mounted device, and the world coordinates of the virtual target, the current pose of the cockpit and the real-time position of the head-mounted device are sent to the head-mounted device; the real-time position of the head-mounted device, the current pose information of the cockpit and the world coordinates of the virtual target are unified in the same coordinate by the head-mounted device according to the comparison and calculation of the world coordinates of the virtual target and the current pose of the cockpit. By using the tracking and positioning method based on Lighthouse, the tracking and positioning among the external target, the cockpit of the aircraft and the visual system of the pilot are realized. The Lighthouse positioning result and the IMU positioning result are fused by using the Kalman filtering technology, so that the real-time and accurate positioning of the mixed reality head-mounted device is realized. The error existing in the head tracking of the airborne AR is reduced.

[0115] In some embodiments, the step of "fusing the first position information and the motion trajectory of the head-mounted device obtained by the second laser positioning sensor and the second position information of the head-mounted device obtained by the inertial measurement sensor through the Kalman filtering algorithm to obtain the real-time position of the head-mounted device" specifically comprises the following steps:

[0116] The initial position of the head-mounted device is used to initialize the current position and the covariance error of the head-mounted device;

[0117] The motion trajectory of the head-mounted device collected by the second laser positioning sensor is taken as the system input u k at the current time; k The position at the next time is predicted with the value of u

[0118] The covariance error of the prediction is calculated at the same time, the second position information obtained by the inertial measurement sensor is taken as the measurement value of the system, and the Kalman gain K k is calculated.

[0119] Then the system measurement value is used to correct the system prediction value, the data is fused, and the Kalman gain K k is used to update the current position to obtain the real-time position of the head-mounted device.

[0120] The extended Kalman filter algorithm is used for positioning of the mixed reality head-mounted device. First, the initial position of the mixed reality head-mounted device is used to initialize the current position and covariance error; in the prediction part, the motion trajectory value of the mixed reality head-mounted device collected by the first laser positioning sensor laser scanning is taken as the system input u k of the current time, the position of the next time is predicted by using the value, the system prediction value is obtained, and the predicted covariance error is calculated; then, the second position information obtained by the IMU is taken as the measurement value of the system, and the Kalman gain K k is calculated; then, the system measurement value is used to correct the system prediction value, that is, the positioning data obtained by laser scanning is corrected by using the position information obtained by the IMU, data fusion is performed, and the current position information is updated by using the Kalman gain K k .

[0121] In some embodiments, the step of "receiving the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base station by the second laser positioning sensor arranged on the head-mounted device to obtain the first position information and the motion trajectory of the head-mounted device" specifically includes the following steps:

[0122] The second laser positioning sensor obtains the horizontal-axis angle and the vertical-axis angle relative to the laser positioning base station according to the time of measuring the horizontal-axis laser rays and the vertical-axis laser rays sent by the laser positioning base station;

[0123] The first position information and the motion trajectory of the head-mounted device are calculated according to the position difference between the first laser positioning sensor and the second laser positioning sensor.

[0124] After the LED light of the laser positioning base station flashes, the second laser positioning sensor distributed on the head-mounted device can measure the time when the horizontal-axis laser and the vertical-axis laser respectively arrive at the sensor, which is just the time when the horizontal-axis laser and the vertical-axis laser arrive at the angle of the specific second laser positioning sensor, so as to obtain the horizontal-axis angle and the vertical-axis angle of the second laser positioning sensor relative to the laser positioning base station; the position of the second laser positioning sensor distributed on the head-mounted device is also known. Then, the position information and the motion trajectory of the head-mounted device can be calculated through the position difference between the first laser positioning sensor and the second laser positioning sensor.

[0125] In some embodiments, the following steps are further included:

[0126] The local flight speed, the local flight height, the local six-dimensional pose, the task performance time, and the Beijing time are sent to the head-mounted device;

[0127] The head-mounted device generates a flight visualization interface according to data processing of data sent by the computer.

[0128] The computer acquires, in real time, world coordinates of the virtual target, flight speed of the aircraft, flight height of the aircraft, six-dimensional pose of the aircraft, task time, Beijing time, etc. by using an internal interface of air combat simulation software, and sends the data to the mixed reality head-mounted device through a UDP communication protocol; the virtual display and control software deployed in the head-mounted device processes the received data to generate a flight visualization interface, and provides the pilot with real-time battlefield situation information.

[0129] In some embodiments, the method further comprises the following steps:

[0130] The head-mounted device calculates the real-time distance between the virtual target and the cockpit according to the world coordinates of the virtual target and the position coordinates of the cockpit, and labels the calculated real-time distance above the virtual graphics corresponding to the virtual target.

[0131] The positioning and three-dimensional registration software deployed in the head-mounted device calculates the real-time distance between the virtual cockpit and the virtual target by means of the coordinates of the virtual target and the virtual cockpit, and displays the real-time distance above the virtual graphics labeling the position of the virtual target.

[0132] In some embodiments, a mixed reality display method for airborne use comprises the following steps:

[0133] In step one, the IMU (inertial measurement sensor) is connected to the computer through a USB-C data line; the Lighthouse positioning sensor (first and second laser positioning sensors) is positioned by receiving infrared signals from the Lighthouse base station (laser positioning base station), the base station needs to be connected to a power supply and placed at a height of about two meters with the help of a support, the computer receives signals from the base station and data from the Lighthouse positioning sensor (first and second laser positioning sensors) through a wireless receiver, and the Lighthouse positioning sensor, the base station and the wireless receiver should be placed in accordance with the principle that they should not be obstructed from each other as much as possible;

[0134] In step two, the IMU (inertial measurement sensor) and the Lighthouse positioning sensor (second laser positioning sensor) are fixed on the mixed reality head-mounted device (head-mounted device) through a metal skeleton, the IMU is placed in a plastic box on the skeleton, the Lighthouse positioning sensor (second laser positioning sensor) is fixed on the skeleton through a special screw, and the skeleton as a whole is adjusted in tightness by a special screw and clamped on the headband of the mixed reality head-mounted device.

[0135] Step three, after the LED flash of the laser positioning base station, the photosensitive sensor distributed on the head-mounted display (i.e. the second laser positioning sensor adopts a photosensitive sensor) can measure the time when the horizontal axis laser and the vertical axis laser respectively reach the sensor, which is exactly the time when the horizontal axis and the vertical axis laser turns to the angle of the sensor at this particular point, so as to obtain the horizontal axis and vertical axis angle of the sensor relative to the base station; the position of the photosensitive sensor distributed on the head-mounted device is also known. Then through the position difference of each sensor (the first laser positioning sensor and the second laser positioning sensor), the position and motion trajectory of the head-mounted device can be calculated;

[0136] Step four, the extended Kalman filtering algorithm is used for mixed reality head-mounted device positioning, first the initial position of the mixed reality head-mounted device is used to initialize the current position and covariance error; in the prediction part, the motion trajectory value of the mixed reality head-mounted device obtained by laser scanning is taken as the system input u k at the current time, the position of the next time is predicted with this value, and the predicted covariance error is obtained, then the position information obtained by the IMU is taken as the measurement value of the system, and the Kalman gain K k is calculated; then the system measurement value is used to modify the system prediction value, that is, the positioning data obtained by laser scanning is modified by the position information obtained by the IMU, and the data is fused, and the current position information is updated by the Kalman gain K k .

[0137] Step five, the world coordinates of the virtual target, the flight speed of the host machine, the flight height of the host machine, the six-dimensional pose of the host machine, the task time, the east eight time zone, etc. are obtained in real time by using the internal interface of the air combat simulation software, and the data is sent to the mixed reality head-mounted device through the UDP communication protocol;

[0138] Step six, the virtual display control software deployed in the mixed reality head-mounted device processes the data to generate a flight visualization interface, which provides real-time battlefield situation information to the pilot;

[0139] Step seven, the world coordinates of the virtual target are obtained in real time by using the internal interface of the air combat simulation software, then compared and calculated with the coordinates and pose of the virtual cockpit, and finally the coordinates are unified.

[0140] Step eight, the positioning and three-dimensional registration software for airborne mixed reality display deployed in the mixed reality head-mounted device unifies the pilot's visual system, the virtual cockpit and the virtual target in the same coordinate system with the help of the coordinates of the virtual cockpit, and then realizes the display of the virtual target in the virtual display control system and the pilot's visual perception system by using the principle of triangle similarity.

[0141] Step nine, the positioning and three-dimensional registration software oriented to the airborne mixed reality display deployed in the mixed reality head-mounted device calculates the real-time distance between the virtual cockpit and the virtual target by means of the coordinates of the virtual target and the virtual cockpit, and displays the real-time distance above the virtual graphics marking the position of the virtual target;

[0142] Step ten, the positioning and three-dimensional registration software oriented to the airborne mixed reality display deployed in the mixed reality head-mounted device carries out audio warning according to the received local enemy lock information.

[0143] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, the patent protection scope of the present application should not be limited. Any technical solution obtained by replacing or modifying the equivalent structure or equivalent process based on the essential concept of the present application, using the content described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, etc., are all included in the patent protection scope of the present application.

Claims

1. A mixed reality display system for airborne applications, characterized in that, include: Two laser positioning base stations are used to transmit horizontal and vertical laser beams. cockpit; A first laser positioning sensor is fixedly installed on the cockpit. The first laser positioning sensor is used to calculate the current cockpit pose based on the horizontal axis laser beam and the vertical axis laser beam sent by the laser positioning base station. Head-mounted display devices; The second laser positioning sensor is installed on the head-mounted display device. The second laser positioning sensor is used to obtain the first position information and motion trajectory of the head-mounted display device based on the horizontal axis laser beam and the vertical axis laser beam sent by the laser positioning base station. An inertial measurement sensor is disposed on the head-mounted display device, and the inertial measurement sensor is used to acquire second position information of the head-mounted display device; The computer is used to fuse the first position information and motion trajectory of the head-mounted display device obtained by the second laser positioning sensor with the second position information of the head-mounted display device obtained by the inertial measurement sensor through a Kalman filter algorithm to obtain the real-time position of the head-mounted display device, and to send the world coordinates of the virtual target, the current pose of the cockpit and the real-time position of the head-mounted display device to the head-mounted display device. The head-mounted display device is used to compare and calculate the world coordinates of the virtual target with the current pose of the cockpit, so as to unify the real-time position of the head-mounted display device, the current pose information of the cockpit, and the world coordinates of the virtual target under the same coordinate system.

2. The airborne mixed reality display system according to claim 1, characterized in that, The computer is also used to initialize the current position and covariance error of the head-mounted display device using its initial position, and to use the motion trajectory of the head-mounted display device acquired by the second laser positioning sensor as the system input u at the current moment. k , with u k The system predicts the position at the next moment, obtaining the system's predicted value. Simultaneously, it calculates the predicted covariance error. Using the second position information acquired by the inertial measurement sensor as the system's measured value, it calculates the Kalman gain K. k Then, the system measurements are used to correct the system predictions, and data fusion is performed, while using Kalman gain K. k The current location is updated to obtain the real-time location of the head-mounted display device.

3. The airborne mixed reality display system according to claim 1, characterized in that, The second laser positioning sensor is used to obtain the horizontal axis angle and vertical axis angle relative to the laser positioning base station based on the arrival time of the horizontal axis laser ray and the vertical axis laser ray sent by the laser positioning base station. The computer is used to calculate the first position information and motion trajectory of the head-mounted display device based on the position difference between the first laser positioning sensor and the second laser positioning sensor.

4. The airborne mixed reality display system according to claim 1, characterized in that, The computer is also used to send the aircraft's flight speed, flight altitude, six-dimensional pose, mission duration, and East 8 time zone to the head-mounted display device. The head-mounted display device is used to process data sent by a computer to generate a flight visualization interface.

5. The airborne mixed reality display system according to claim 1, characterized in that, The head-mounted display device is also used to calculate the real-time distance between the virtual target and the cockpit based on the world coordinates of the virtual target and the position coordinates of the cockpit, and to mark the calculated real-time distance above the virtual graphic corresponding to the virtual target.

6. A mixed reality display method for airborne applications, characterized in that, Includes the following steps: The current cockpit pose is calculated by receiving horizontal and vertical laser beams from the laser positioning base station through a first laser positioning sensor located above the cockpit. The first position information and motion trajectory of the head-mounted display device are obtained by receiving the horizontal and vertical axis laser beams sent by the laser positioning base station through the second laser positioning sensor installed on the head-mounted display device. The second position information of the head-mounted display device is obtained by using an inertial measurement sensor installed on the head-mounted display device; The first position information and motion trajectory of the head-mounted display device obtained by the second laser positioning sensor are fused with the second position information of the head-mounted display device obtained by the inertial measurement sensor using the Kalman filter algorithm to obtain the real-time position of the head-mounted display device. The world coordinates of the virtual target, the current pose of the cockpit, and the real-time position of the head-mounted display device are then sent to the head-mounted display device. The head-mounted display device compares and calculates the world coordinates of the virtual target with the current pose of the cockpit, unifying the real-time position of the head-mounted display device, the current pose information of the cockpit, and the world coordinates of the virtual target under the same coordinate system.

7. The airborne mixed reality display method according to claim 6, characterized in that, The step of fusing the first position information and motion trajectory of the head-mounted display device obtained by the second laser positioning sensor with the second position information of the head-mounted display device obtained by the inertial measurement sensor using the Kalman filter algorithm to obtain the real-time position of the head-mounted display device specifically includes the following steps: The current position and covariance error of the head-mounted display are initialized using the initial position of the head-mounted display; The motion trajectory of the head-mounted display device, collected by the second laser positioning sensor, is used as the system input u at the current moment. k , with u k The system predicts the position at the next moment by using the value; Simultaneously, the predicted covariance error is calculated, and the second position information acquired by the inertial measurement sensor is used as the system's measurement value to calculate the Kalman gain K. k ; Then, the system measurements are used to correct the system predictions, and data fusion is performed, while using Kalman gain K. k The current location is updated to obtain the real-time location of the head-mounted display device.

8. The airborne mixed reality display method according to claim 6, characterized in that, The step of obtaining the first position information and motion trajectory of the head-mounted display device by receiving the horizontal and vertical axis laser rays sent by the laser positioning base station through the second laser positioning sensor installed on the head-mounted display device specifically includes the following steps: The second laser positioning sensor obtains the horizontal axis angle and vertical axis angle relative to the laser positioning base station by measuring the arrival time of the horizontal axis laser ray and the vertical axis laser ray sent by the laser positioning base station. The first position information and motion trajectory of the head-mounted display device are calculated based on the position difference between the first laser positioning sensor and the second laser positioning sensor.

9. The airborne mixed reality display method according to claim 6, characterized in that, It also includes the following steps: The flight speed, altitude, six-dimensional pose, mission duration, and East 8 time zone of the aircraft are sent to the head-mounted display device. The head-mounted display device processes data sent from the computer to generate a flight visualization interface.

10. The airborne mixed reality display method according to claim 6, characterized in that, It also includes the following steps: The head-mounted display calculates the real-time distance between the virtual target and the cockpit based on the world coordinates of the virtual target and the position coordinates of the cockpit, and marks the calculated real-time distance above the virtual graphic corresponding to the virtual target.

Images