Multi-source fusion positioning and enhanced display calibration method for airborne augmented reality devices

Abstract

The present invention discloses a multi-source fusion positioning and augmented display calibration method for an airborne augmented reality device. This method is primarily implemented by an active infrared marker module, an inertial navigation sensor module, and an infrared image sensor module mounted on a head-mounted augmented reality display device; a binocular infrared visual positioning sensor mounted on an aircraft seat; a passive infrared target image module mounted in the aircraft cockpit; a display control processing computer and its associated positioning registration and display calibration, as well as a real-time posture fusion positioning workflow. The present invention implements multi-source fusion positioning and augmented reality display calibration functions based on inertial navigation and infrared vision sensors, providing a comprehensive positioning, display, and calibration solution for head-mounted augmented reality display devices with high precision, high reliability, and robustness.

Description

Technical Field

[0001] The present invention relates to an interactive device for an aircraft cockpit display and control system, and in particular to a multi-source fusion positioning and enhanced display calibration method for an airborne head-mounted augmented reality display device. Background Art

[0002] An augmented reality helmet with both augmented reality display and head-motion control functions is an ideal interactive display and control device for pilots to achieve omnidirectional situational awareness and ensure flight safety. To achieve augmented reality display in an airborne environment, the augmented reality display device must have accurate positioning and augmented display technologies. The accuracy and robustness of these technologies directly affect the accuracy of the augmented display and even affect flight safety. Compared with ground-based augmented reality display devices, the positioning and display registration schemes of airborne augmented reality display devices are more susceptible to the complex light, electromagnetic, and vibration environments of the airborne environment. Currently, there are many methods used for augmented reality positioning, such as inertial navigation positioning, ultrasonic positioning, magnetic positioning, infrared laser scanning positioning, and visual image positioning. However, a single positioning method cannot meet the positioning accuracy requirements of the airborne environment and is difficult to adapt to the complex airborne application environment. Therefore, it is impossible to accurately obtain high-precision augmented display effects. Summary of the Invention

[0003] Aiming at the complex airborne environment, the present invention provides a positioning and augmented display calibration method for airborne head-mounted augmented reality display devices, realizing multi-source fusion positioning and augmented reality display calibration functions based on inertial navigation and infrared vision sensors, and providing a comprehensive positioning, display and calibration solution with high precision, high reliability and robustness for head-mounted augmented reality display devices.

[0004] In order to achieve the above tasks, the present invention adopts the following technical solutions:

[0005] A multi-source fusion positioning and enhanced display calibration device for airborne augmented reality equipment includes a source infrared marker module, an inertial navigation sensor module, an infrared image acquisition array module, a binocular infrared vision positioning sensor, a passive infrared target image module, and a display control processing computer; wherein:

[0006] The binocular infrared visual positioning sensor is installed above and behind the pilot's seat and consists of a binocular infrared image sensor and an inertial navigation sensor. Based on the time synchronization signal uniformly sent by the display and control processing computer, the binocular infrared image sensor collects infrared marker point images of the active infrared marker point module installed on the head-mounted augmented reality display device, extracts the two-dimensional coordinates of the infrared marker points in the image, and performs real-time tracking and positioning calculations on the posture of the head-mounted augmented reality display device based on matching 2D-3D point matching information and combining the known three-dimensional coordinates of the infrared marker points. The inertial navigation sensor collects angular velocity and acceleration posture data of the aircraft cabin in real time.

[0007] The active infrared marker modules are regularly distributed on the outer surface of the head-mounted augmented reality display device; the infrared LED lights of the infrared marker modules are uniformly controlled by the time synchronization signal sent by the display control processing computer;

[0008] The inertial navigation sensor module is installed on the head-mounted augmented reality display device and is used to collect angular velocity and acceleration data of the head-mounted augmented reality display device in real time, and perform subtraction processing on the angular velocity and acceleration data of the aircraft cockpit collected by the inertial navigation sensor in the binocular infrared vision positioning sensor to obtain angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft cockpit;

[0009] The passive infrared target image module is installed on the front instrument panel directly in front of the pilot's seat;

[0010] The infrared image acquisition array module is installed on the front side of the head-mounted augmented reality device and can locate and identify the passive infrared target image module within the range of the head-mounted augmented reality device;

[0011] The display processing computer serves as the main control unit, performs synchronous alignment control on the time of the data from each sensor, and performs fusion calculation on the posture of the head-mounted augmented reality display device according to the primary and secondary relationship of each sensor in accordance with the real-time posture fusion positioning process.

[0012] Furthermore, the PnP algorithm is used to perform real-time tracking and positioning calculation on the posture of the head-mounted augmented reality display device.

[0013] A multi-source fusion positioning and enhanced display calibration method for airborne augmented reality devices, including positioning registration and display calibration processes:

[0014] Step 1: After entering the cockpit, the pilot adjusts the seat position and wears the head-mounted augmented reality display device, keeping it firmly and comfortably facing the passive infrared target image module.

[0015] Step 2: The infrared image acquisition array module on the head-mounted augmented reality display device acquires a passive infrared target image module on the front instrument panel of the aircraft and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module;

[0016] Step 3: The binocular infrared vision positioning sensor on the pilot's seat collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device, extracts and calculates the two-dimensional coordinates of all infrared markers in the image, matches the infrared markers on the image with the infrared markers on the helmet, and uses the PnP algorithm to calculate the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor based on the matching 2D-3D point matching information and the two-dimensional and three-dimensional coordinates of the infrared markers.

[0017] Step 4, calculating the pose of the binocular infrared vision positioning sensor relative to the passive infrared target image module based on the pose data obtained in steps 2 and 3;

[0018] Step 5: Taking the position of the passive infrared target image module as the origin of the world coordinate system, the position of the head-mounted augmented reality display device relative to the passive infrared target image module is calculated by the position of the binocular infrared vision positioning sensor relative to the head-mounted augmented reality display device in step 3 and the position of the binocular infrared vision positioning sensor relative to the infrared target in step 4, thereby completing the positioning calibration of the head-mounted augmented reality display device under the aircraft cockpit.

[0019] Furthermore, the method further comprises:

[0020] Step 6: The display control processing computer generates and places a virtual target object with the same physical shape and size as the passive infrared target image at the origin of the passive infrared target image module. The pilot observes the spatial consistency between the virtual target object and the real passive infrared target image through the head-mounted augmented reality display device.

[0021] Step 7: When the pilot observes a difference in spatial position between the virtual target object and the real passive infrared target image, the pose of the virtual target object is adjusted so that the spatial position of the virtual target object is consistent with that of the real passive infrared target image, and the pose difference of the adjusted virtual target object relative to the origin of the world coordinate system is calculated;

[0022] Step 8: Adjust the spatial positions of other virtual icons or objects displayed in the head-mounted augmented reality display device based on the pose difference of the virtual target object relative to the origin of the world coordinate system obtained in step 7, and complete the display calibration of the head-mounted augmented reality display device under the aircraft cockpit.

[0023] Furthermore, the method also includes a real-time posture fusion positioning process, which is as follows:

[0024] A. The display and control processing computer sends a time synchronization signal to the binocular infrared visual positioning sensor and the inertial navigation sensor module on the head-mounted augmented reality display device;

[0025] B. The inertial navigation sensor module of the head-mounted augmented reality display device acquires the raw sensor data of angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing;

[0026] C. The inertial navigation sensor in the binocular infrared vision positioning sensor acquires the raw sensor data of angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing;

[0027] D. The display control processing computer obtains the raw sensor data obtained in steps B and C and performs subtraction processing to obtain the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft;

[0028] E. The infrared image sensor in the binocular infrared vision positioning sensor collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device based on the time synchronization signal of the display control processing computer, and calculates the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor;

[0029] F. The display control processing computer obtains the position and posture of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor, combines the position and posture of the binocular infrared vision positioning sensor relative to the passive infrared target image module obtained in the positioning registration and display calibration process, and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module;

[0030] G. Using an asynchronous loosely coupled method in the display and control processing computer, the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft obtained in step D and the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module obtained in step F are fused through the extended Kalman filter algorithm. The fused position and posture information is calculated and output at a fixed frequency.

[0031] Furthermore, the method further comprises:

[0032] H. During the pose fusion calculation cycle, when the display and control processing computer obtains new pose data from the binocular infrared vision positioning sensor, it updates the current fusion status in real time;

[0033] I. During the pose fusion calculation cycle, if no new pose data is received from the binocular infrared vision positioning sensor, the future pose is predicted and denoised by the acceleration and angular velocity of the current pose to obtain the predicted pose data.

[0034] Furthermore, when the binocular infrared vision positioning sensor cannot output real-time posture data, the infrared image sensor on the head-mounted augmented reality display device collects the image of the passive infrared target image module, calculates the posture of the head-mounted augmented reality display device relative to the passive infrared target image module, and replaces the data of the binocular infrared vision positioning sensor to participate in the calculation of the fused positioning data in step GI.

[0035] Furthermore, the display and control processing computer calculates and generates an augmented reality display screen based on the fused posture information, the posture data of the aircraft's own inertial navigation system, and the augmented display calibration parameters, and outputs it to the head-mounted augmented reality device.

[0036] Compared with the prior art, the present invention has the following technical features:

[0037] First, the present invention eliminates the influence of the high-speed motion and vibration environment of the aircraft itself in the airborne environment through the subtraction processing algorithm of the inertial navigation sensor of the head-mounted device and the inertial navigation sensor in the external binocular infrared vision positioning sensor, and can obtain the relative motion parameters of the head-mounted augmented reality display device in the aircraft coordinate system.

[0038] Second, the present invention reduces the integration difficulty of configuring active positioning (electromagnetic positioning, infrared marker positioning) on ​​aircraft systems.

[0039] Third, the present invention realizes an augmented reality positioning method suitable for airborne environments through the visual positioning of infrared marker points from the outside to the inside and the visual positioning of infrared images from the inside to the outside, as well as a multi-source fusion positioning method with inertial navigation positioning. On the one hand, it improves the positioning accuracy, and on the other hand, it improves the redundancy of the system, providing a reliable positioning and augmented display calibration technical solution for airborne applications of head-mounted augmented reality display devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a block diagram of the architecture of a calibration device according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the overall layout of the equipment according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of the layout of a wearable augmented reality display device according to an embodiment of the present invention;

[0043] Figure 4 1 is a schematic diagram of the relevant components of a positioning sensor according to an embodiment of the present invention;

[0044] Figure 5 This is a workflow diagram of the present invention.

[0045] The purpose of the present invention is to solve the problem of accurate real-time positioning and display calibration of a head-mounted augmented reality display device in the aircraft body coordinate system in a complex airborne environment. Through three positioning algorithms, namely, real-time posture positioning of active infrared markers by a binocular infrared image sensor in an outside-in manner, real-time posture calculation of an inertial navigation sensor, and positioning and display calibration of passive infrared targets by an infrared image sensor in an inside-out manner, as well as a posture fusion algorithm that integrates the three positioning methods, a multi-source fusion positioning and enhanced display calibration scheme for airborne head-mounted augmented reality display devices is realized, which effectively solves the problems of easy positioning loss, low positioning calibration accuracy, inability to display and calibrate in time, and compatibility with the airborne environment.

[0046] The present invention first provides a multi-source fusion positioning and enhanced display calibration device for an airborne augmented reality device, comprising an active infrared marker module, an inertial navigation sensor module, and an infrared image acquisition array module installed on a head-mounted augmented reality display device; a binocular infrared visual positioning sensor installed on an aircraft seat, and a passive infrared target image module and a display control processing computer provided in the aircraft cockpit; wherein:

[0047] The binocular infrared visual positioning sensor is installed above and behind the pilot's seat and consists of a binocular infrared image sensor and an inertial navigation sensor. Based on the time synchronization signal uniformly sent by the display and control processing computer, the binocular infrared image sensor collects infrared marker point images of the active infrared marker point module installed on the head-mounted augmented reality display device, extracts the two-dimensional coordinates of the infrared marker points in the image, and uses the PnP algorithm to perform real-time tracking and positioning calculations on the head-mounted augmented reality display device based on the matching 2D-3D point matching information and the known three-dimensional coordinates of the infrared marker points. The inertial navigation sensor collects real-time posture data such as angular velocity and acceleration of the aircraft cockpit.

[0048] The active infrared marker modules are regularly distributed on the outer surface of the head-mounted augmented reality display device; the infrared LED lights of the infrared marker modules are uniformly controlled by the time synchronization signal sent by the display control processing computer, ensuring synchronization with the image acquisition and posture data calculation of the binocular infrared vision positioning sensor;

[0049] The inertial navigation sensor module is used to collect angular velocity and acceleration data of the head-mounted augmented reality display device in real time, and perform subtraction processing on the angular velocity and acceleration data of the aircraft cockpit collected by the inertial navigation sensor in the binocular infrared vision positioning sensor to obtain the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft cockpit;

[0050] The passive infrared target image module is installed on the front instrument panel directly in front of the pilot's seat or at other installable locations;

[0051] The infrared image acquisition array module is installed on the front side of the head-mounted augmented reality device and can locate and identify the passive infrared target image module within the range of the head-mounted augmented reality device for positioning calibration and display calibration;

[0052] The display processing computer serves as the main control unit, performs synchronous alignment control on the time of the data from each sensor, and performs fusion calculation on the posture of the head-mounted augmented reality display device according to the primary and secondary relationship of each sensor in accordance with the real-time posture fusion positioning process.

[0053] The present invention provides a multi-source fusion positioning and enhanced display calibration method for an airborne augmented reality device, including positioning registration and display calibration processes, as follows:

[0054] Step 1: After entering the cockpit, the pilot adjusts the seat position and wears the head-mounted augmented reality display device, keeping it firmly and comfortably facing the passive infrared target image module.

[0055] Step 2: The infrared image acquisition array module on the head-mounted augmented reality display device acquires a passive infrared target image module on the front instrument panel of the aircraft and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module;

[0056] Step 3: The binocular infrared vision positioning sensor on the pilot's seat collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device, extracts and calculates the two-dimensional coordinates of all infrared markers (infrared LED lights) in the image, matches the infrared markers on the image with the infrared markers on the helmet, and uses the PnP algorithm to calculate the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor based on the matching 2D-3D point matching information and the two-dimensional and three-dimensional coordinates of the infrared markers;

[0057] Step 4: Since the binocular infrared vision positioning sensor is installed on the pilot's seat, the position of the binocular infrared vision positioning sensor relative to the aircraft cockpit is not fixed due to the pilot's adjustment or vibration during flight, while the passive infrared target image module is hard-connected to the aircraft cockpit front panel and its position relative to the aircraft cockpit is fixed; therefore, the position data obtained in steps 2 and 3 can be used to calculate the position of the binocular infrared vision positioning sensor relative to the passive infrared target image module;

[0058] Step 5: Taking the position of the passive infrared target image module as the origin of the world coordinate system, the position of the head-mounted augmented reality display device relative to the passive infrared target image module can be calculated from the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor in step 3 and the position of the binocular infrared vision positioning sensor relative to the infrared target in step 4, completing the positioning calibration of the head-mounted augmented reality display device under the aircraft cockpit;

[0059] Step 6: The display control processing computer generates and places a virtual target object with the same physical shape and size as the passive infrared target image at the origin of the passive infrared target image module. The pilot observes the spatial consistency between the virtual target object and the real passive infrared target image through the head-mounted augmented reality display device.

[0060] Step 7: When the pilot observes a difference in spatial position between the virtual target object and the real passive infrared target image, the pose of the virtual target object is adjusted so that the spatial position of the virtual target object and the real passive infrared target image are consistent, and the pose difference of the adjusted virtual target object relative to the origin of the world coordinate system is calculated; when the pilot observes no difference in spatial position between the virtual target object and the real passive infrared target, the pose difference is zero;

[0061] Step 8: Adjust the spatial positions of other virtual icons or objects displayed in the head-mounted augmented reality display device based on the pose difference of the virtual target object relative to the origin of the world coordinate system obtained in step 7, and complete the display calibration of the head-mounted augmented reality display device under the aircraft cockpit.

[0062] The multi-source fusion positioning and enhanced display calibration method of an airborne augmented reality device of the present invention also includes a real-time posture fusion positioning process, which is specifically as follows:

[0063] A. The display and control processing computer sends time synchronization signals to devices such as the binocular infrared visual positioning sensor and the inertial navigation sensor module on the head-mounted augmented reality display device;

[0064] B. The inertial navigation sensor module of the head-mounted augmented reality display device acquires raw sensor data such as angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing;

[0065] C. The inertial navigation sensor in the binocular infrared vision positioning sensor acquires raw sensor data such as angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing;

[0066] D. The display and control processing computer obtains the raw sensor data obtained in steps B and C and performs subtraction processing to remove the effects of the aircraft's own angular velocity and acceleration, thereby obtaining the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft.

[0067] E. The infrared image sensor in the binocular infrared vision positioning sensor collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device based on the time synchronization signal of the display control processing computer, and calculates the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor;

[0068] F. The display control processing computer obtains the position and posture of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor, combines the position and posture of the binocular infrared vision positioning sensor relative to the passive infrared target image module obtained in step 4 of the positioning registration and display calibration process, and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module;

[0069] G. Using an asynchronous loosely coupled method in the display and control processing computer, the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft obtained in step D and the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module obtained in step F are fused using an extended Kalman filter algorithm. The fused position and posture information is calculated and output at a fixed frequency.

[0070] H. During the pose fusion calculation cycle, when the display and control processing computer obtains new pose data from the binocular infrared vision positioning sensor, it updates the current fusion status in real time;

[0071] I. During the pose fusion calculation cycle, if no new pose data is received from the binocular infrared vision positioning sensor, the future pose is predicted and denoised using the acceleration and angular velocity of the current pose to obtain the predicted pose data.

[0072] J. When the binocular infrared visual positioning sensor is affected by the configuration of the helmet and cannot identify and locate infrared markers at special head movement positions, or is unable to output real-time pose data due to other abnormal reasons, the infrared image sensor on the head-mounted augmented reality display device collects images from the passive infrared target image module, calculates the pose of the head-mounted augmented reality display device relative to the passive infrared target image module, and replaces the data from the binocular infrared visual positioning sensor in the calculation of the fused positioning data in step GI;

[0073] K. The display control processing computer calculates and generates the augmented reality display screen based on the fused posture information, the posture data of the aircraft's own inertial navigation system, and the augmented display calibration parameters, and outputs it to the head-mounted augmented reality device.

[0074] Example

[0075] In the following embodiments, Example 1 is the composition of each module of the system in the method and the specific implementation case configuration, and Example 2 is the specific workflow of the method in the usage scenario.

[0076] Example 1:

[0077] (1) The embodiment of the present invention is as follows Figure 1 As shown in the system block diagram, to achieve the positioning registration and enhanced display functions of a head-mounted augmented reality display device, the solution includes four components: a head-mounted augmented reality display device 201, a binocular infrared visual positioning sensor 202, a display control processing computer 203, and a passive infrared target image module 204. Among them, the head-mounted augmented reality display device needs to be equipped with an active infrared marker module 301, an inertial navigation sensor module 302, an infrared image acquisition array module 303, and other corresponding functional modules. The binocular infrared visual positioning sensor 202 mainly includes an inertial navigation sensor 304 and a binocular infrared image sensor 305.

[0078] (2) If Figure 2 As shown, the active infrared marker point module 301 is fixed on the shell surface of the head-mounted augmented reality display device 201 in the form of a preset dot matrix distribution, serving as an infrared image source marker based on marker point visual positioning, and is exposed and illuminated according to the time synchronization signal provided by the display control processing computer 203.

[0079] (3) If Figure 2 As shown, the inertial navigation sensor module 302 is installed in the control processing unit of the back shell of the head-mounted augmented reality display device 201 to obtain the original angular velocity, acceleration and other motion posture information of the head-mounted augmented reality display device 201.

[0080] (4) If Figure 2 and Figure 3As shown, the infrared vision sensor module 303 is a binocular infrared image sensor, which is installed on the front upper part of the head-mounted augmented reality display device 201. One of the infrared image sensors can be used to obtain the image of the passive infrared target image module 204 within the aircraft pilot's field of view, and based on the visual positioning algorithm, the relative position information between the head-mounted augmented reality display device and the passive infrared target image can be calculated.

[0081] (5) If Figure 2 and Figure 4 As shown, the binocular infrared vision sensor 202 is installed above the seat directly behind the aircraft pilot. The lens field of view of the infrared image sensor module 401 on the binocular infrared vision positioning sensor 202 covers the position of the head-mounted augmented reality display device 201 under extreme movement or extreme rotation in the aircraft cockpit, ensuring that the infrared marker point image can be collected within the aircraft pilot's range of movement.

[0082] (6) If Figure 2 and Figure 4 As shown, the binocular infrared vision sensor 202 is equipped with an inertial navigation sensor module 402, which can collect data such as the angular velocity and acceleration of the aircraft body in real time.

[0083] (7) Figure 2 As shown, the passive infrared target image module 204 is installed on the front instrument panel of the aircraft cockpit, providing an identification image source for the infrared image acquisition array module 303 on the head-mounted augmented reality display device 201.

[0084] (8) The display control processing computer 203 serves as the main control unit and is installed under the aircraft seat. It mainly performs time synchronization control on the sensors and performs fusion calculation on the real-time position of the head-mounted augmented reality display device according to the primary and secondary relationships of the sensors in accordance with the positioning fusion processing process.

[0085] (9) According to the hardware configuration plan, the display control processing computer 203 performs data modeling on the binocular infrared vision sensor 202, the passive infrared target image module 204, and the head-mounted augmented reality display device 201, and records the relative posture relationship of each hardware as a parameter for display calibration and posture fusion calculation.

[0086] Example 2

[0087] The specific implementation steps of the present invention are as follows Figure 5 As shown, it is divided into two parts: positioning registration and display calibration, and posture fusion positioning calculation. The specific steps are as follows:

[0088] Step 1: Configure according to the hardware configuration and system status of Example 1;

[0089] Step 2: The pilot activates the head-mounted augmented reality display device 201, the binocular infrared vision positioning sensor 202, the display control processing computer 203 and related systems;

[0090] Step 3: The pilot puts on the head-mounted augmented reality display device 201 and keeps it firmly and comfortably worn.

[0091] Step 4: The infrared image acquisition module 303 on the head-mounted augmented reality display device 201 acquires an image of the passive infrared target image module 204 on the aircraft's front instrument panel and calculates the position and posture of the head-mounted augmented reality display device relative to the infrared target.

[0092] Step 5: The binocular infrared vision positioning sensor 202 collects images of infrared marker points arranged on the head-mounted augmented reality display device 201 and calculates the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor;

[0093] Step 6: The position and posture data obtained in steps 4 and 5 can be used to calculate the position and posture of the binocular infrared vision positioning sensor 202 relative to the infrared target 204. Combined with the position and posture data in step 5, the position and posture of the head-mounted augmented reality display device relative to the passive infrared target are calculated to complete the positioning and calibration of the head-mounted augmented reality display device 201 under the aircraft cockpit.

[0094] Step 7: The display and control processing computer 203 generates and places a virtual target object with the same physical shape and size as the passive infrared target at the location of the passive infrared target 204. The pilot observes the spatial consistency between the virtual target object displayed on the augmented reality head-mounted display and the real passive infrared target through the head-mounted augmented reality display device. If there is a difference, the position of the virtual target object is adjusted to make the virtual target object and the real passive infrared target spatial position consistent, and the position difference of the adjusted virtual target object relative to the initial position, i.e., the origin of the world coordinate system, is calculated. If there is no difference, the position of the virtual target object does not need to be adjusted, and the position difference is zero.

[0095] Step 8: Based on the position difference of the virtual target object relative to the initial position obtained in Step 7, the spatial positions of other virtual icons or objects displayed in the head-mounted augmented reality display device 201 are adjusted to complete the display calibration of the head-mounted augmented reality display device 201 in the aircraft cockpit;

[0096] Step 9: The inertial navigation sensor modules 302 and 402 of the head-mounted augmented reality display device 201 and the binocular infrared visual positioning sensor 202 acquire raw data such as angular velocity and acceleration in real time based on the time synchronization signal of the display control processing computer, and perform denoising processing;

[0097] Step 10: The display control processing computer 203 obtains data from the inertial navigation sensor module 302 of the head-mounted augmented reality display device 201 and the inertial navigation sensor module 402 of the binocular infrared vision positioning sensor 202, performs subtraction processing to remove the effects of the aircraft's high-speed motion and vibration environment, and obtains the position change of the head-mounted augmented reality display device 201 relative to the aircraft.

[0098] Step 11: The binocular infrared vision positioning sensor 202 collects infrared marker images of the active infrared marker module 301 of the head-mounted augmented reality display device 201 according to the time synchronization signal, and performs real-time pose calculation of the augmented reality display device;

[0099] Step 12: The display control processing computer 203 obtains the position of the head-mounted augmented reality display device 201 relative to the binocular infrared vision positioning sensor 202 calculated by the binocular infrared vision positioning sensor 202, and combines the position of the binocular infrared vision positioning sensor relative to the infrared target obtained in step 5 to calculate the position of the head-mounted augmented reality display device relative to the infrared target;

[0100] Step 13: Using an asynchronous loosely coupled method, the display and control processing computer 203 fuses the angular velocity and acceleration data of the head-mounted augmented reality display device 201 relative to the aircraft obtained in step 10 with the position and posture of the head-mounted augmented reality display device 201 relative to the infrared target 204 obtained in step 12 using an extended Kalman filter algorithm. The fused position and posture information is calculated and output at a fixed frequency.

[0101] Step 14: When the display control processing computer 203 obtains new posture data of the binocular infrared vision positioning sensor 202 during the fusion posture data calculation cycle, the current fusion state and offset are updated in real time;

[0102] Step 15: If the pose data of the binocular infrared vision positioning sensor 202 is not obtained within the fusion pose data calculation cycle, the future pose is predicted and denoised using the acceleration and angular velocity obtained in step 10 to obtain predicted pose data.

[0103] Step 16: When the binocular infrared visual positioning sensor 202 is unable to identify and locate the infrared markers at a special head movement position due to the configuration of the helmet, or is unable to output real-time posture data due to other abnormal reasons, the infrared image acquisition array module 303 on the head-mounted augmented reality display device 202 collects images from the passive infrared target image module 204 at the front of the aircraft, calculates the posture of the head-mounted augmented reality display device 202 relative to the infrared target 204, and replaces the data of the binocular infrared visual positioning sensor in the calculation of the fused positioning data in steps 13-15;

[0104] In step 17, the display control processing computer 203 calculates and generates an augmented reality display image based on the fused real-time positioning posture, the posture data of the aircraft's own inertial navigation system, and the augmented display calibration parameters, and outputs it to the head-mounted augmented reality device.

[0105] In summary, the present invention provides a multi-source fusion positioning and enhanced display calibration method for an airborne head-mounted augmented reality display device, which effectively integrates the infrared marker point module, inertial navigation sensor module, infrared image sensor module installed on the head-mounted augmented reality display device, the binocular infrared visual positioning sensor installed on the aircraft seat, the passive infrared target image module installed in the aircraft cockpit, and the display control processing computer, effectively solving the problems of easy loss of positioning, low positioning calibration accuracy, inability to display and calibrate in time, and mismatch with the airborne environment, and provides a comprehensive positioning, display and calibration solution with high precision, high reliability and robustness for head-mounted augmented reality display devices.

[0106] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A multi-source fusion positioning and enhanced display calibration device for airborne augmented reality equipment, characterized in that: It includes a source infrared marker module, an inertial navigation sensor module, an infrared image acquisition array module, a binocular infrared visual positioning sensor, a passive infrared target image module, and a display and control processing computer; wherein: The binocular infrared visual positioning sensor is installed above and behind the pilot's seat and consists of a binocular infrared image sensor and an inertial navigation sensor. Based on the time synchronization signal uniformly sent by the display and control processing computer, the binocular infrared image sensor collects infrared marker point images of the active infrared marker point module installed on the head-mounted augmented reality display device, extracts the two-dimensional coordinates of the infrared marker points in the image, and performs real-time tracking and positioning calculations on the posture of the head-mounted augmented reality display device based on matching 2D-3D point matching information and combining the known three-dimensional coordinates of the infrared marker points. The inertial navigation sensor collects angular velocity and acceleration posture data of the aircraft cabin in real time. The active infrared marker modules are regularly distributed on the outer surface of the head-mounted augmented reality display device; the infrared LED lights of the infrared marker modules are uniformly controlled by the time synchronization signal sent by the display control processing computer; The inertial navigation sensor module is installed on the head-mounted augmented reality display device and is used to collect angular velocity and acceleration data of the head-mounted augmented reality display device in real time, and perform subtraction processing on the angular velocity and acceleration data of the aircraft cockpit collected by the inertial navigation sensor in the binocular infrared vision positioning sensor to obtain angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft cockpit; The passive infrared target image module is installed on the front instrument panel directly in front of the pilot's seat; The infrared image acquisition array module is installed on the front side of the head-mounted augmented reality device and can locate and identify the passive infrared target image module within the range of the head-mounted augmented reality device; The display control processing computer serves as the main control unit, performs synchronous alignment control on the time of the data of each sensor, and performs fusion calculation of the posture of the head-mounted augmented reality display device according to the primary and secondary relationship of each sensor in accordance with the real-time posture fusion positioning process.

2. The multi-source fusion positioning and enhanced display calibration device for airborne augmented reality equipment according to claim 1, characterized in that: The PnP algorithm is used to perform real-time tracking and positioning calculation of the posture of the head-mounted augmented reality display device.

3. A multi-source fusion positioning and enhanced display calibration method for airborne augmented reality equipment, characterized in that: Including positioning registration and display calibration process: Step 1: After entering the cockpit, the pilot adjusts the seat position and wears the head-mounted augmented reality display device, keeping it firmly and comfortably facing the passive infrared target image module. Step 2: The infrared image acquisition array module on the head-mounted augmented reality display device acquires a passive infrared target image module on the front instrument panel of the aircraft and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module; Step 3: The binocular infrared vision positioning sensor on the pilot's seat collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device, extracts and calculates the two-dimensional coordinates of all infrared markers in the image, matches the infrared markers on the image with the infrared markers on the helmet, and uses the PnP algorithm to calculate the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor based on the matching 2D-3D point matching information and the two-dimensional and three-dimensional coordinates of the infrared markers. Step 4, calculating the pose of the binocular infrared vision positioning sensor relative to the passive infrared target image module based on the pose data obtained in steps 2 and 3; Step 5: Taking the position of the passive infrared target image module as the origin of the world coordinate system, the position of the head-mounted augmented reality display device relative to the passive infrared target image module is calculated through the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor in step 3 and the position of the binocular infrared vision positioning sensor relative to the infrared target in step 4, thereby completing the positioning calibration of the head-mounted augmented reality display device under the aircraft cockpit.

4. The multi-source fusion positioning and enhanced display calibration method for an airborne augmented reality device according to claim 3, characterized in that: The method further comprises: Step 6: The display control processing computer generates and places a virtual target object with the same physical shape and size as the passive infrared target image at the origin of the passive infrared target image module. The pilot observes the spatial consistency between the virtual target object and the real passive infrared target image through the head-mounted augmented reality display device. Step 7: When the pilot observes a difference in spatial position between the virtual target object and the real passive infrared target image, the pose of the virtual target object is adjusted so that the spatial position of the virtual target object is consistent with that of the real passive infrared target image, and the pose difference of the adjusted virtual target object relative to the origin of the world coordinate system is calculated; Step 8: Adjust the spatial positions of other virtual icons or objects displayed in the head-mounted augmented reality display device based on the pose difference of the virtual target object relative to the origin of the world coordinate system obtained in step 7, and complete the display calibration of the head-mounted augmented reality display device under the aircraft cockpit.

5. The multi-source fusion positioning and enhanced display calibration method for airborne augmented reality equipment according to claim 3, characterized in that: The method also includes a real-time posture fusion positioning process, which is as follows: A. The display and control processing computer sends a time synchronization signal to the binocular infrared visual positioning sensor and the inertial navigation sensor module on the head-mounted augmented reality display device; B. The inertial navigation sensor module of the head-mounted augmented reality display device acquires the raw sensor data of angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing; C. The inertial navigation sensor in the binocular infrared vision positioning sensor acquires the raw sensor data of angular velocity and acceleration in real time based on the time synchronization signal and performs denoising processing; D. The display control processing computer obtains the raw sensor data obtained in steps B and C and performs subtraction processing to obtain the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft; E. The infrared image sensor in the binocular infrared vision positioning sensor collects images of the active infrared marker modules arranged on the head-mounted augmented reality display device based on the time synchronization signal of the display control processing computer, and calculates the position of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor; F. The display control processing computer obtains the position and posture of the head-mounted augmented reality display device relative to the binocular infrared vision positioning sensor, combines the position and posture of the binocular infrared vision positioning sensor relative to the passive infrared target image module obtained in the positioning registration and display calibration process, and calculates the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module; G. Using an asynchronous loosely coupled method in the display and control processing computer, the angular velocity and acceleration data of the head-mounted augmented reality display device relative to the aircraft obtained in step D and the position and posture of the head-mounted augmented reality display device relative to the passive infrared target image module obtained in step F are fused through the extended Kalman filter algorithm. The fused position and posture information is calculated and output at a fixed frequency.

6. The multi-source fusion positioning and enhanced display calibration method for an airborne augmented reality device according to claim 5, characterized in that: Also includes: H. During the pose fusion calculation cycle, when the display and control processing computer obtains new pose data from the binocular infrared vision positioning sensor, it updates the current fusion status in real time; I. During the pose fusion calculation cycle, if no new pose data is received from the binocular infrared vision positioning sensor, the future pose is predicted and denoised using the acceleration and angular velocity of the current pose to obtain the predicted pose data.

7. The multi-source fusion positioning and enhanced display calibration method for an airborne augmented reality device according to claim 5, characterized in that: When the binocular infrared vision positioning sensor cannot output real-time posture data, the infrared image sensor on the head-mounted augmented reality display device collects the image of the passive infrared target image module, calculates the posture of the head-mounted augmented reality display device relative to the passive infrared target image module, and replaces the data of the binocular infrared vision positioning sensor to participate in the calculation of the fusion positioning data in step GI.

8. The multi-source fusion positioning and enhanced display calibration method for an airborne augmented reality device according to claim 5, characterized in that: The display and control processing computer calculates and generates the augmented reality display screen based on the fused posture information, the posture data of the aircraft's own inertial navigation system, and the augmented display calibration parameters, and outputs it to the head-mounted augmented reality device.

Images