A head-mounted HUD perspective display and control system based on multi-source fusion

Abstract

The application relates to the field of aviation display and augmented reality technology, in particular to a head-mounted HUD perspective display and control system based on multi-source fusion, which is suitable for aircrafts and comprises a multi-source data acquisition module, a unified space-time calibration module, a positioning and attitude sensing module, an image registration and perspective view generation module, a display imaging module and a mode triggering and management module. The system supports multiple modes such as extravehicular perspective mode, cargo cabin perspective mode and passenger cabin perspective mode, can fuse visible light data and thermal imaging data, is suitable for rapid situation awareness and safety decision in flight operation and ground operation, and through combination of multi-source camera acquisition, AI position and attitude registration, space registration and perspective mapping, multi-mode intelligent switching and delay fault tolerance mechanism, the perspective display of extravehicular, cargo cabin and passenger cabin in the flight operation scene and the ground operation scene is well realized, and the situation awareness and emergency disposal capability of pilots and ground staff is significantly improved.

Description

Technical Field

[0001] Embodiments of this application relate to the field of aviation display and augmented reality technology, and in particular to a head-mounted HUD perspective display and control system based on multi-source fusion. Background Technology

[0002] During flight and ground operations, pilots and ground personnel need to maintain real-time awareness of the situation outside the cabin, cargo hold, and passenger cabin to promptly detect obstacles, fires, or safety threats. Currently used fixed HUD (Head-Up Display) systems project flight information into the field of view using optical reflection, but their fixed installation location and viewing angle are tied to the device's position, making it impossible to flexibly adjust based on the user's location. While some research teams have used EVS (Enhanced Vision System) and SVS (Synthetic Vision System) to overlay airborne camera footage onto the HUD, these are primarily used for flight navigation and external view enhancement, offering limited functionality and failing to provide perspective-taking of the cargo hold or passenger cabin. In the event of a cargo hold fire, a passenger cabin safety incident, or ground loading / unloading operations, pilots and ground personnel still rely on independent monitoring equipment or voice notifications for awareness, resulting in information transmission delays and limited observation range.

[0003] In addition, traditional HUD systems lack multi-mode intelligent switching and AI scene recognition capabilities, and cannot automatically select information sources and match perspectives for different mission scenarios. Therefore, we urgently need a head-mounted HUD system that combines multi-source image fusion, AI position and attitude registration, and multi-scene perspective display to improve situational awareness and safety handling efficiency in flight and ground operations. Summary of the Invention

[0004] In view of this, embodiments of this application propose a head-mounted HUD perspective display and control system based on multi-source fusion. It aims to achieve perspective display of the cabin exterior, cargo hold, and passenger cabin in flight operation scenarios and ground operation scenarios by combining multi-source camera acquisition, AI position and attitude registration, spatial registration and perspective mapping, multi-mode intelligent switching, and delay fault tolerance mechanism. This significantly improves the situational awareness and emergency response capabilities of pilots and ground staff.

[0005] To achieve the above objectives, embodiments of this application propose a multi-source fusion-based head-mounted display and control system for aircraft. The system includes: a multi-source data acquisition module for acquiring image data from the head-mounted HUD's own camera and an airborne camera; wherein the airborne camera includes at least an external airborne camera, a cargo hold camera, a passenger cabin camera, and a thermal imaging camera; a unified spatiotemporal calibration module for calibrating external parameters and synchronizing time on the acquired multi-source image data, ensuring that all image data are fused under a unified spatial coordinate system and thus a unified time reference; a positioning and attitude perception module for calculating the wearer's real-time position and attitude in the aircraft coordinate system based on the image data acquired by the head-mounted HUD's camera and the attitude data measured by the inertial measurement unit, combined with AI recognition algorithms and synchronous positioning and mapping algorithms; and an image registration and perspective generation module for performing spatial registration and perspective transformation on the image data acquired by the airborne camera according to the wearer's real-time position and attitude in the aircraft coordinate system, and combining depth information and occlusion detection results to execute... The system integrates and generates a virtual perspective view that matches the wearer's perspective; the display imaging module projects the virtual perspective view onto the optical lenses of the head-mounted HUD to achieve perspective display of obscured areas; the mode triggering and management module switches between cabin exterior perspective mode, cargo hold perspective mode, and passenger cabin perspective mode based on task priority, alarm signals, or user operations.

[0006] To achieve the above objectives, embodiments of this application also propose a multi-source fusion-based head-mounted HUD perspective display and control method, applicable to aircraft, implemented based on a multi-source fusion-based head-mounted HUD perspective display and control system as described above. The method includes the following steps: activating the head-mounted HUD, sequentially activating the multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module, and display imaging module; establishing a wired or wireless data channel between the head-mounted HUD and the airborne camera; and performing unified external parameter calibration and time synchronization to ensure all video streams are in a unified spatial coordinate system and a unified time reference. The airborne camera includes at least an external airborne camera, a cargo hold camera, a passenger cabin camera, and a thermal imaging camera. The system utilizes video streams captured by the head-mounted display's (HUD) main camera, attitude data measured by the inertial measurement unit (IMU), and AI recognition results, combined with airborne positioning information, to calculate the wearer's position and attitude in the aircraft coordinate system in real time using a synchronous positioning and mapping algorithm. Following the current perspective mode, it calls upon the HUD's main camera and corresponding airborne cameras to capture video streams, performing preprocessing including distortion correction, noise reduction, and brightness equalization, and then adding a unified timestamp and spatial pose label. Based on the wearer's position and attitude in the aircraft coordinate system, it maps the video streams captured by each airborne camera to a unified spatial coordinate system, performing perspective transformation and viewpoint correction to ensure the virtual perspective image matches the wearer's actual viewing direction and parallax. Finally, it performs depth estimation and occlusion detection on the registered video streams, based on... The fusion algorithm prioritizes different video sources according to depth order, ensuring that obstructed areas are filled in by visible video sources, generating a perspective view that matches the real scene, thus obtaining a virtual perspective image. AI scene recognition and analysis are then applied to the virtual perspective image to detect and label the positions of personnel, obstacles, cargo, and areas with abnormal temperatures. This labeled information is then overlaid on the virtual perspective image for display, resulting in the final perspective image to assist pilots and ground staff in decision-making. The final perspective image is projected onto the optical lenses of the head-mounted HUD through an optical projection component, achieving an infinity-far imaging effect and enabling perspective display of obstructed areas. Throughout the head-mounted HUD perspective display and control process, rapid switching between external perspective mode, cargo hold perspective mode, and passenger cabin perspective mode is achieved based on task priority, alarm signals, or user operation trigger modes. Transition buffers maintain image stability and continuity. Upon detecting a malfunction in the head-mounted HUD perspective display and control system, an immediate warning is issued and a switch to a safe mode is initiated, ensuring the basic availability of the head-mounted HUD's field of view.

[0007] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a head-mounted HUD perspective display and control method based on multi-source fusion as described above.

[0008] Optionally, after the head-mounted HUD is activated, the head-mounted HUD perspective display and control system enters the initialization phase. The multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module, and display imaging module are activated sequentially. After the wired or wireless data channel between the head-mounted HUD and the airborne external camera, cargo cabin camera, passenger cabin camera, and thermal imaging camera is established, the unified spatiotemporal calibration module performs unified extrinsic parameter calibration and time synchronization. During the operation of the head-mounted HUD perspective display and control system, it continuously performs unified extrinsic parameter calibration and time synchronization to ensure that all image data are under a unified spatial coordinate system and a unified time reference. Specifically, the time synchronization is PTP protocol time synchronization with a delay of no more than 5ms.

[0009] Optionally, the positioning and attitude perception module calculates the wearer's real-time position and attitude in the aircraft coordinate system based on image data collected by the camera of the head-mounted HUD and attitude data measured by the inertial measurement unit, combined with AI recognition algorithms. It also optimizes the calculation by combining simultaneous localization and mapping algorithms, ensuring that the misalignment between the perspective view and the actual visual scene does not exceed [the specified value]. Perspective; In the localization and attitude perception module, the AI ​​recognition algorithm is used to extract semantic feature information of the environment, including scene boundaries, obstacle edges, landmarks and personnel positions, and uses the semantic feature information as a constraint input to the synchronous localization and mapping unit for loop closure detection and feature matching optimization; through joint optimization of semantic features and geometric features, the robustness of position and attitude estimation is improved, so that the misalignment between the virtual perspective image and the real view does not exceed 20 pixels.

[0010] Optionally, the image registration and perspective generation module incorporates AI scene recognition and analysis functions to achieve spatial registration and perspective transformation. This is used to detect and label the locations of people, obstacles, cargo, and areas with abnormal temperatures, and then overlay the labeled information onto the virtual perspective view for display, assisting pilots and ground staff in decision-making. Spatial registration and perspective transformation are achieved through a spatial mapping algorithm based on feature point matching and attitude estimation. By utilizing the extrinsic parameter relationship between the HUD's body camera and the airborne camera from the wearer's perspective, the homography matrix or projection matrix is ​​calculated to map the airborne image to a unified visual coordinate system. Combined with depth map information, a weighted perspective fusion strategy is adopted, using depth weights to determine the overlay order of different image sources in the virtual perspective view, thereby generating a geometrically consistent and depth-correct fused perspective display.

[0011] Optionally, the head-mounted HUD perspective display and control system supports multiple interaction methods, including at least gesture recognition, voice commands, and HUD button input. The wearer can control the virtual perspective image through various interaction methods to achieve image zooming, panning, freezing, and recording operations.

[0012] Optionally, in the cargo hold perspective mode, the image registration and perspective generation module merges the image data collected by the cargo hold camera and the image data collected by the thermal imaging camera to generate a virtual perspective image with temperature distribution overlay, in order to help the wearer determine the location and extent of the fire.

[0013] Optionally, when the signal from the airborne camera is delayed or interrupted, the image registration and perspective generation module calls the cached image data, uses a prediction algorithm to generate a temporary predicted virtual perspective image for the display imaging module to display, and adds a prediction label to the head-mounted HUD to indicate to the wearer that the current virtual perspective image is the predicted content. The prediction algorithm performs time interpolation prediction based on optical flow estimation and inter-frame difference to maintain image stability during the delay period.

[0014] Optionally, after projecting the virtual perspective image onto the optical lenses of the head-mounted HUD, the display imaging module automatically adjusts the viewing area according to the wearer's interpupillary distance and eye position to achieve perspective display of the obscured area, thereby reducing the wearer's visual fatigue.

[0015] This application proposes a head-mounted HUD perspective display and control system based on multi-source fusion. Through the collaboration of a multi-source data acquisition module, a unified spatiotemporal calibration module, a positioning and attitude perception module, an image registration and perspective view generation module, a display imaging module, and a mode triggering and management module, it can achieve perspective display in multiple scenarios such as the cabin exterior, cargo hold, and passenger cabin, breaking through the limitations of traditional head-mounted HUDs in terms of field of view and function. AI position and attitude registration and SLAM optimization ensure a high degree of consistency between the virtual view and the user's perspective. Multi-source fusion and thermal imaging overlay of image data from the HUD's own camera and airborne camera effectively improves the detection capabilities for anomalies such as fires and live objects. The design of the mode triggering and management module and the configuration of AI algorithms enable the head-mounted HUD perspective display and control system to have AI-automatic recognition and labeling functions and intelligent multi-mode switching capabilities, effectively improving emergency response efficiency. The system supports multiple interaction methods such as gestures, voice, and buttons, and is designed with a latency-tolerant mechanism to ensure the continuity and reliability of critical tasks. This system is widely adaptable to both flight and ground operations, improving the safety and efficiency of aviation operations and significantly enhancing the situational awareness and emergency response capabilities of pilots and ground staff. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0017] Figure 1 This is a schematic diagram of a head-mounted HUD perspective display and control system based on multi-source fusion provided in one embodiment of this application; Figure 2 This is a flowchart of a head-mounted HUD perspective display and control method based on multi-source fusion, provided in another embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0019] One embodiment of this application proposes a head-mounted HUD perspective display and control system based on multi-source fusion, applicable to aircraft. The implementation details of the head-mounted HUD perspective display and control system based on multi-source fusion proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding only and are not necessary for implementing this solution.

[0020] The specific structure of the head-mounted HUD perspective display and control system proposed in this embodiment can be as follows: Figure 1 As shown, it includes: a multi-source data acquisition module 11, a unified spatiotemporal calibration module 12, a positioning and attitude perception module 13, an image registration and perspective generation module 14, a display imaging module 15, and a mode triggering and management module 16.

[0021] The multi-source data acquisition module 11 is used to acquire image data from the main camera of the head-mounted HUD and the airborne camera, respectively. The airborne camera includes at least an external airborne camera, a cargo hold camera, a passenger cabin camera, and a thermal imaging camera.

[0022] The unified spatiotemporal calibration module 12 is used to perform extrinsic parameter calibration and time synchronization on the acquired multi-source image data, ensuring that all image data are fused under a unified spatial coordinate system and thus a unified time reference.

[0023] The positioning and attitude perception module 13 is used to calculate the wearer's real-time position and attitude in the aircraft coordinate system by combining image data collected by the camera of the head-mounted HUD and attitude data measured by the inertial measurement unit, and AI recognition algorithm and synchronous positioning and mapping algorithm.

[0024] The image registration and perspective generation module 14 is used to perform spatial registration and perspective transformation on the image data acquired by the airborne camera based on the wearer's real-time position and attitude in the aircraft coordinate system, and to perform [further processing] by combining depth information and occlusion detection results. It merges and generates a virtual perspective view that matches the wearer's perspective.

[0025] The display imaging module 15 is used to project virtual perspective images onto the optical lenses of the head-mounted HUD to achieve perspective display of the obscured area.

[0026] The mode triggering and management module 16 is used to switch between the cabin exterior perspective mode, cargo hold perspective mode, and passenger cabin perspective mode based on task priority, alarm signals, or user operations.

[0027] The functions of each component module of the head-mounted HUD perspective display and control system based on multi-source fusion proposed in this embodiment will be described in detail below.

[0028] The multi-source data acquisition module is used to acquire image data from the head-mounted HUD's main camera and airborne cameras, with the specific data selection depending on the current perspective mode. There are three perspective modes: external perspective mode, cargo hold perspective mode, and passenger cabin perspective mode. The airborne external camera is deployed outside the cabin, responsible for capturing and collecting real-time images of the outside environment. The cargo hold camera and thermal imaging camera are deployed inside the cargo hold, responsible for capturing and collecting real-time images of the cargo hold environment. The passenger cabin camera is deployed inside the passenger cabin, responsible for capturing and collecting real-time images of the passenger cabin environment. In external perspective mode, only image data (or video streams) from the airborne external camera are needed. In cargo hold perspective mode, image data (or video streams) from the cargo hold camera and thermal imaging camera are needed. In passenger cabin perspective mode, only image data (or video streams) from the passenger cabin camera are needed.

[0029] Among them, the airborne external cameras, cargo hold cameras, and passenger cabin cameras are all visible light cameras, and the image data (or video streams) they collect are all visible light image data (or visible light video streams), while the image data (or video streams) collected by the thermal imaging cameras are infrared image data (or infrared video streams).

[0030] After the head-mounted HUD is activated, the head-mounted HUD perspective display and control system enters the initialization phase, and the multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module, and display imaging module are activated in sequence.

[0031] The unified spatiotemporal calibration module requires unified extrinsic parameter calibration and time synchronization during system startup and operation. Specifically, after establishing wired or wireless data channels between the head-mounted HUD and the airborne external cameras, cargo hold cameras, cabin cameras, and thermal imaging cameras, the module performs unified extrinsic parameter calibration and time synchronization. This process continues throughout the operation of the head-mounted HUD's perspective display and control system, ensuring all image data is within a unified spatial coordinate system and time reference. The time synchronization specifically uses the PTP protocol, with a delay not exceeding 5ms. It's important to note that the unified spatiotemporal calibration module operates continuously and automatically; however, it can be manually controlled if the pilot or ground personnel deem it necessary to re-calibrate and re-synchronize.

[0032] The positioning and attitude perception module uses image data captured by the camera of the head-mounted HUD and attitude data measured by the inertial measurement unit (IMU). It combines AI recognition algorithms to calculate the wearer's real-time position and attitude in the aircraft coordinate system, and uses Simultaneous Localization and Mapping (SLAM) algorithms for optimization to ensure that the misalignment between the perspective view and the actual view does not exceed [a certain value]. Viewpoint (or 20 pixels).

[0033] The image registration and perspective generation module incorporates AI scene recognition and analysis capabilities to detect and label the locations of personnel, obstacles, cargo, and areas of abnormal temperature. This labeled information is then overlaid onto the virtual perspective view for display, assisting pilots and ground personnel in decision-making. This design is crucial for fire detection. In cargo hold perspective mode, the module merges image data from cargo hold cameras and thermal imaging cameras to generate a virtual perspective view with temperature distribution overlay, helping the wearer determine the location and extent of a fire.

[0034] It should be noted that, Image fusion is a commonly used image fusion algorithm, which is essentially a process of overlaying the foreground onto the background using transparency.

[0035] The head-mounted HUD (Head-Up Display) and control system supports multiple interaction methods, including at least gesture recognition, voice commands, and HUD button input. Wearers can control the virtual view through these methods, enabling zooming, panning, freezing, and recording. This design is commonly used in cabin view modes, allowing pilots to zoom in, pan, and control areas via gestures, voice commands, or controllers for a better real-time view of the cabin.

[0036] The head-mounted HUD perspective display and control system is equipped with a delay-tolerance mechanism. When the signal from the airborne camera is delayed or interrupted (delay exceeding 100ms, visual misalignment exceeding...), the system will compensate for the delay. When a visual misalignment exceeds 20 pixels, a delay-tolerance mechanism is automatically activated. The image registration and perspective generation module calls the cached image data (cached video frames) and uses a prediction algorithm to generate a temporary predicted virtual perspective image based on the cached and predicted video frames for the display imaging module to display. A prediction indicator is added to the head-mounted HUD to indicate to the wearer that the current virtual perspective image is the predicted content (prediction status), ensuring that key information remains visible. The prediction algorithm, based on the optical flow estimation and inter-frame difference results of the cached video frame sequence, predicts the motion trend of future frames through temporal interpolation, generating a short-term predicted perspective image to fill in blank displays caused by network or computational latency. The prediction duration does not exceed 500ms, and it automatically and smoothly transitions to the real-time image after the real image is restored.

[0037] After projecting the virtual see-through image onto the optical lenses of the head-mounted HUD, the display imaging module automatically adjusts the viewing area according to the wearer's interpupillary distance and eye position to achieve see-through display of obstructed areas, reducing visual fatigue and better conforming to the wearer's visual habits. The end-to-end display latency is less than 100ms.

[0038] During the switching of perspective modes based on task priority, alarm signals, or user operations, the mode triggering and management module ensures a switching delay of no more than 1 second, maintaining image continuity through buffering and gradual transitions. During mode switching, the system maintains image continuity through frame buffering and dual-channel rendering mechanisms, performing parallel preprocessing on the video streams of the old and new modes, and employing fade-in / fade-out or gradual transitions for smooth switching to prevent image flicker and parallax abrupt changes. Furthermore, in the event of switching failure or communication interruption, the system automatically reverts to a safe display mode, ensuring continuous output of basic visual information.

[0039] This embodiment proposes a head-mounted HUD perspective display and control system based on multi-source fusion. Through the collaboration of a multi-source data acquisition module, a unified spatiotemporal calibration module, a positioning and attitude perception module, an image registration and perspective view generation module, a display imaging module, and a mode triggering and management module, it can achieve perspective display in multiple scenarios such as the cabin exterior, cargo hold, and passenger cabin, breaking through the limitations of traditional head-mounted HUDs in terms of field of view and function. AI position and attitude registration and SLAM optimization ensure a high degree of consistency between the virtual view and the user's perspective. Multi-source fusion and thermal imaging overlay of image data from the HUD's own camera and airborne camera effectively improves the detection capabilities for anomalies such as fires and live objects. The design of the mode triggering and management module and the configuration of AI algorithms enable the head-mounted HUD perspective display and control system to have AI-automatic recognition and labeling functions and intelligent multi-mode switching capabilities, effectively improving emergency response efficiency. The system supports multiple interaction methods such as gestures, voice, and buttons, and is designed with a latency-tolerant mechanism to ensure the continuity and reliability of critical tasks. This system is widely adaptable to both flight and ground operations, improving the safety and efficiency of aviation operations and significantly enhancing the situational awareness and emergency response capabilities of pilots and ground staff.

[0040] It is worth noting that all modules involved in this embodiment are logical modules. In practical applications, a logical module can be a physical module, a part of a physical module, or an organic combination of multiple physical modules. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce modules that are not closely related to solving the technical problems proposed in this application. However, this does not mean that other modules are absent from this embodiment.

[0041] Another embodiment of this application proposes a head-mounted HUD perspective display and control method based on multi-source fusion, applicable to aircraft, and implemented based on a head-mounted HUD perspective display and control system based on multi-source fusion as described in the above system embodiment. The implementation details of the head-mounted HUD perspective display and control method based on multi-source fusion proposed in this embodiment are described below. The following implementation details are provided for ease of understanding only and are not essential for implementing this solution.

[0042] The specific process of the head-mounted HUD perspective display and control method based on multi-source fusion proposed in this embodiment can be described as follows: Figure 2 As shown, it includes: Step 21: Activate the head-mounted HUD, and sequentially activate the multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module, and display imaging module. Establish a wired or wireless data channel between the head-mounted HUD and the airborne camera, and perform unified external parameter calibration and time synchronization to ensure that all video streams are under a unified spatial coordinate system and a unified time reference.

[0043] The airborne cameras include at least external airborne cameras, cargo hold cameras, passenger cabin cameras, and thermal imaging cameras. The external airborne cameras, cargo hold cameras, and passenger cabin cameras are all visible light cameras. Time synchronization is standardized to PTP level, with a delay of no more than 5ms.

[0044] Step 22: Using the video stream collected by the head-mounted HUD's main camera, the attitude data measured by the inertial measurement unit, and the AI ​​recognition results, combined with the airborne positioning information, the wearer's position and attitude in the aircraft coordinate system are calculated in real time through a synchronous positioning and mapping algorithm.

[0045] Step 23: According to the current perspective mode, call the main camera of the head-mounted HUD and the corresponding airborne camera to collect video streams, perform preprocessing including distortion correction, noise reduction, and brightness equalization, and then attach a unified timestamp and spatial pose label.

[0046] Step 24: Based on the wearer's position and attitude in the aircraft coordinate system, map the video streams captured by each airborne camera to a unified spatial coordinate system, perform perspective transformation and viewpoint correction, so that the virtual perspective image is consistent with the wearer's actual observation direction and parallax.

[0047] Step 25: Perform depth estimation and occlusion detection on the registered video stream. Based on the α fusion algorithm, adjust the priority of different video sources according to the depth order to ensure that the occluded area is filled by the visible video source, generate a perspective view that matches the real scene, and obtain a virtual perspective image.

[0048] Step 26: Perform AI scene recognition and analysis on the virtual perspective image, detect and mark the positions of personnel, obstacles, cargo and abnormal temperature areas, and overlay the marked information on the virtual perspective image to obtain the final perspective image to assist pilots and ground staff in decision-making.

[0049] Step 27: The final transparent image is projected onto the optical lens of the head-mounted HUD through the optical projection component to achieve an infinity imaging effect and realize the transparent display of the obscured area.

[0050] Throughout the entire process of head-mounted HUD perspective display and control, it is necessary to switch between cabin exterior perspective mode, cargo hold perspective mode and passenger cabin perspective mode quickly according to task priority, alarm signal or user operation trigger mode, and maintain image stability and continuity through transition buffer. When a fault is detected in the head-mounted HUD perspective display and control system, an alarm is immediately issued and the system switches to safety mode to ensure the basic availability of the head-mounted HUD's field of view.

[0051] During the operation of the head-mounted HUD perspective display and control system, the wearer can perform interactive operations such as zooming in, zooming out, panning, freezing, and recording via gestures, voice, or buttons. When a video delay exceeding 100ms or a visual misalignment exceeding [a certain threshold] is detected, [further action will be taken]. When the resolution is 20 pixels, the delay tolerance mechanism is automatically enabled, the cached frame and the prediction frame are called to generate a temporary image, and the current image is displayed as a prediction image.

[0052] During the detection of the operation of the head-mounted HUD perspective display and control system, the fault level is determined according to the degree of signal interruption or quality degradation, and corresponding safety degradation strategies are implemented, including but not limited to switching to redundant video sources, reducing resolution, increasing refresh rate, or retaining only key annotation information.

[0053] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0054] It is not difficult to see that this embodiment is a method embodiment corresponding to the above system embodiment, and this embodiment can be implemented in conjunction with the above system embodiment. The relevant technical details and technical effects mentioned in the above system embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above system embodiment.

[0055] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement a head-mounted HUD perspective display and control method based on multi-source fusion as described in the above method embodiments.

[0056] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0057] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A head-mounted HUD perspective display and control system based on multi-source fusion, suitable for aircraft, characterized in that, The system includes: The multi-source data acquisition module is used to acquire image data from the head-mounted HUD's main camera and airborne cameras respectively; wherein, the airborne cameras include at least an airborne external camera, a cargo hold camera, a passenger cabin camera, and a thermal imaging camera; The unified spatiotemporal calibration module is used to perform extrinsic parameter calibration and time synchronization on the acquired multi-source image data, ensuring that all image data are fused under a unified spatial coordinate system and thus a unified time reference. The positioning and attitude perception module is used to calculate the wearer's real-time position and attitude in the aircraft coordinate system by combining image data collected by the camera of the head-mounted HUD and attitude data measured by the inertial measurement unit, along with AI recognition algorithms and simultaneous positioning and mapping algorithms. The image registration and perspective generation module is used to perform spatial registration and perspective transformation on image data acquired by the airborne camera based on the wearer's real-time position and attitude in the aircraft coordinate system, and to combine depth information and occlusion detection results to execute... The system merges and generates a virtual perspective view that matches the wearer's viewpoint. The display imaging module is used to project virtual see-through images onto the optical lenses of the head-mounted HUD to achieve see-through display of the obscured area; The mode triggering and management module is used to switch between the exterior view mode, cargo hold view mode, and passenger cabin view mode based on task priority, alarm signals, or user operations.

2. The head-mounted HUD perspective display and control system based on multi-source fusion according to claim 1, characterized in that, After the head-mounted HUD is activated, the head-mounted HUD perspective display and control system enters the initialization phase, and the multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module and display imaging module are activated in sequence. After the wired or wireless data channel between the head-mounted HUD and the airborne external cameras, cargo cabin cameras, passenger cabin cameras and thermal imaging cameras is established, the unified spatiotemporal calibration module performs unified external parameter calibration and time synchronization. During the operation of the head-mounted HUD perspective display and control system, it continuously performs unified external parameter calibration and time synchronization to ensure that all image data are under a unified spatial coordinate system and a unified time reference. Specifically, time synchronization refers to PTP protocol time synchronization, with a delay of no more than 5ms.

3. The head-mounted HUD perspective display and control system based on multi-source fusion according to claim 1, characterized in that, The positioning and attitude perception module uses image data captured by the camera of the head-mounted HUD and attitude data measured by the inertial measurement unit. It combines AI recognition algorithms to calculate the wearer's real-time position and attitude in the aircraft coordinate system, and optimizes the calculation by combining simultaneous localization and mapping algorithms to ensure that the misalignment between the perspective view and the actual view does not exceed [a certain value]. Perspective; In the localization and attitude perception module, the AI ​​recognition algorithm is used to extract semantic feature information of the environment, including scene boundaries, obstacle edges, landmarks and personnel positions, and uses the semantic feature information as a constraint input to the synchronous localization and mapping unit for loop closure detection and feature matching optimization; through joint optimization of semantic features and geometric features, the robustness of position and attitude estimation is improved, so that the misalignment between the virtual perspective image and the real view does not exceed 20 pixels.

4. The head-mounted HUD perspective display and control system based on multi-source fusion according to claim 1, characterized in that, The image registration and perspective generation module incorporates AI scene recognition and analysis capabilities to achieve spatial registration and perspective transformation. This is used to detect and label the locations of people, obstacles, cargo, and areas with abnormal temperatures, and then overlay the labeled information onto the virtual perspective view for display, assisting pilots and ground staff in decision-making. Spatial registration and perspective transformation are achieved through a spatial mapping algorithm based on feature point matching and attitude estimation. By utilizing the extrinsic parameter relationship between the HUD's body camera and the airborne camera from the wearer's perspective, the homography matrix or projection matrix is ​​calculated to map the airborne image to a unified visual coordinate system. Combined with depth map information, a weighted perspective fusion strategy is adopted, using depth weights to determine the overlay order of different image sources in the virtual perspective view, thereby generating a geometrically consistent and depth-correct fused perspective display.

5. A head-mounted HUD perspective display and control system based on multi-source fusion according to claim 1, characterized in that, The head-mounted HUD perspective display and control system supports multiple interaction methods, including at least gesture recognition, voice commands, and HUD button input. Wearers can control the virtual perspective screen through various interaction methods to achieve screen zooming, panning, freezing, and recording operations.

6. The head-mounted HUD perspective display and control system based on multi-source fusion according to claim 1, characterized in that, In cargo hold perspective mode, the image registration and perspective generation module merges image data collected by the cargo hold camera and image data collected by the thermal imaging camera to generate a virtual perspective image with temperature distribution overlay, in order to help the wearer determine the location and extent of the fire.

7. A head-mounted HUD perspective display and control system based on multi-source fusion according to any one of claims 1 to 6, characterized in that, When the signal from the airborne camera is delayed or interrupted, the image registration and perspective generation module calls the cached image data and uses a prediction algorithm to generate a temporary predicted virtual perspective image for the display imaging module to display. A prediction indicator is added to the head-mounted HUD to indicate to the wearer that the current virtual perspective image is the predicted content. The prediction algorithm is based on optical flow estimation and inter-frame difference to perform time interpolation prediction in order to maintain image stability during the delay period.

8. A head-mounted HUD perspective display and control system based on multi-source fusion according to any one of claims 1 to 6, characterized in that, After projecting the virtual perspective image onto the optical lenses of the head-mounted HUD, the display imaging module automatically adjusts the viewing area according to the wearer's interpupillary distance and eye position to achieve perspective display of the obscured area, thereby reducing the wearer's visual fatigue.

9. A method for perspective display and control of a head-mounted HUD based on multi-source fusion, characterized in that, Based on the head-mounted HUD perspective display and control system according to any one of claims 1 to 8, the method includes: The head-mounted HUD is activated, sequentially activating the multi-source data acquisition module, unified spatiotemporal calibration module, positioning and attitude perception module, image registration and perspective generation module, and display imaging module. A wired or wireless data channel is established between the head-mounted HUD and the airborne camera, and unified external parameter calibration and time synchronization are performed to ensure that all video streams are under a unified spatial coordinate system and a unified time reference. The airborne camera includes at least an airborne external camera, a cargo hold camera, a passenger cabin camera, and a thermal imaging camera. By using the video stream captured by the camera on the head-mounted HUD, the attitude data measured by the inertial measurement unit, and the AI ​​recognition results, combined with the airborne positioning information, the wearer's position and attitude in the aircraft coordinate system are calculated in real time through synchronous positioning and mapping algorithms. According to the current perspective mode, the video stream is captured by the main camera of the head-mounted HUD and the corresponding airborne camera. After preprocessing including distortion correction, noise reduction and brightness equalization, a unified timestamp and spatial pose label are added. Based on the wearer's position and attitude in the aircraft coordinate system, the video streams captured by each airborne camera are mapped to a unified spatial coordinate system, and perspective transformation and viewpoint correction are performed to make the virtual perspective image consistent with the wearer's actual observation direction and parallax. Depth estimation and occlusion detection are performed on the registered video stream, based on The fusion algorithm adjusts the priority of different video sources according to depth order to ensure that the occluded area is filled by the visible video source, generating a perspective view that matches the real scene and obtaining a virtual perspective image; AI scene recognition and analysis are performed on the virtual perspective image to detect and mark the positions of people, obstacles, cargo and abnormal temperature areas. The marked information is then overlaid on the virtual perspective image for display to obtain the final perspective image, which assists pilots and ground staff in decision-making. The final transparent image is projected onto the optical lenses of the head-mounted HUD through an optical projection component, achieving an infinity imaging effect and enabling transparent display of obscured areas. Throughout the entire process of head-mounted HUD perspective display and control, the system rapidly switches between cabin exterior perspective mode, cargo hold perspective mode, and passenger cabin perspective mode based on task priority, alarm signals, or user operation trigger mode. It also maintains stable and continuous image through transition buffers. When a malfunction is detected in the head-mounted HUD perspective display and control system, it immediately issues a warning and switches to a safe mode to ensure the basic availability of the head-mounted HUD's field of view.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can realize the head-mounted HUD perspective display and control method based on multi-source fusion as described in claim 9.

Images