Image processing method, image display device, computer program product and aircraft

Abstract

An image processing method, an image display apparatus, a computer program product, and an aircraft are disclosed. The image processing method includes: generating a two-dimensional output image based on input information; guiding a light beam corresponding to the output image to a display unit; capturing a three-dimensional image displayed on the display unit; obtaining a photometric error and a geometric error between the captured three-dimensional image and the output image; constructing a loss function for the photometric error and the geometric error; solving the loss function through an algorithm for solving three-dimensional to two-dimensional point pair motion to obtain a rotation matrix and a translation matrix; and correcting the output image according to the rotation matrix and the translation matrix to display the corrected output image on the display unit.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more specifically to improving the imaging quality of image display devices. Background Art

[0002] In recent years, head-up displays (HUDs) have experienced rapid development and widespread application. Also known as head-up display systems, HUDs are technologies that project important information such as vehicle information, turn signals, lane departure warnings, navigation, road conditions, and telephone information directly into the driver's field of vision. They combine information with the driver's real-world view, allowing the driver to obtain the necessary information without looking down or turning their head, thus improving driving safety and avoiding distractions caused by looking down at the instrument panel.

[0003] Traditional HUD optical paths typically employ two-stage specular reflection technology, which presents two drawbacks. First, the need for a larger second-stage aspherical mirror to achieve a wider field of view (FOV) results in a larger overall HUD size. Second, the stringent tolerance requirements for concave lenses in order to generate a distortion-free image lead to higher manufacturing and assembly costs. Summary of the Invention

[0004] A brief overview of this disclosure is provided below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the disclosure. It is not intended to identify key or essential parts of the disclosure, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] According to one aspect of this disclosure, an image processing method is provided, comprising: generating a two-dimensional output image based on input information; guiding a beam of light corresponding to the output image to a display unit; capturing a three-dimensional image displayed on the display unit; calculating an error between the captured three-dimensional image and the output image; and correcting the output image based on the error to display the corrected output image on the display unit.

[0006] According to a preferred embodiment, the calculation error also includes photometric and geometric errors between the captured 3D image and the output image.

[0007] According to a preferred embodiment, the calculation error further includes constructing a loss function for photometric and geometric errors, and solving the loss function using an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

[0008] According to a preferred embodiment, the algorithm for solving the motion of three-dimensional to two-dimensional point pairs is the perspective n-point problem (PNP) algorithm.

[0009] According to a preferred embodiment, the image processing method further includes correcting the output image based on the translation and rotation matrices obtained by solving the loss function through the perspective n-point problem (PNP) algorithm.

[0010] According to a preferred embodiment, the photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto a two-dimensional plane and the pixels of the output image.

[0011] According to a preferred embodiment, the geometric error is obtained by performing triangulation on the captured 3D image and the output image.

[0012] According to a preferred embodiment, a beam of light corresponding to the output image is guided to the display unit via a waveguide.

[0013] According to a preferred embodiment, the waveguide is equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit and a corner protector for removing distortion generated when the output image is projected onto the display unit.

[0014] According to a preferred embodiment, the image processing method further includes using a stitching algorithm to stitch together images of each of the multiple waveguides with smooth edges.

[0015] According to a preferred embodiment, a three-dimensional image displayed on a display unit is captured by a sensor arranged in the viewer's eye box area.

[0016] According to another aspect of this disclosure, an image display device is provided, comprising: an image generation unit configured to generate a two-dimensional output image based on input information; a display unit configured to display the output image; an optical path transmission unit configured to guide a light beam corresponding to the output image to the display unit; a three-dimensional sensor arranged to capture an image displayed on the display unit; and an error calculation unit configured to calculate an error between the three-dimensional image captured by the three-dimensional sensor and the output image, wherein the image generation unit is further configured to correct the output image based on the error to display the corrected output image on the display unit.

[0017] According to a preferred embodiment, the three-dimensional sensor is arranged within the viewer's eyebox area.

[0018] According to a preferred embodiment, the error calculation unit is further configured to obtain the photometric error and geometric error between the captured 3D image and the output image.

[0019] According to a preferred embodiment, the error calculation unit is also configured to construct a loss function for photometric and geometric errors, and to solve the loss function using an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

[0020] According to a preferred embodiment, the algorithm for solving the motion of three-dimensional to two-dimensional point pairs is the perspective n-point problem (PNP) algorithm.

[0021] According to a preferred embodiment, the image generation unit is further configured to correct the output image based on the translation and rotation matrices obtained by solving the loss function using the PNP algorithm.

[0022] According to a preferred embodiment, the photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto a two-dimensional plane and the pixels of the output image.

[0023] According to a preferred embodiment, the geometric error is obtained by performing triangulation on the captured 3D image and the output image.

[0024] According to a preferred embodiment, the optical transmission unit is a waveguide.

[0025] According to a preferred embodiment, the waveguide is equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit and a corner protector for removing distortion generated when the output image is projected onto the display unit.

[0026] According to a preferred embodiment, the image display device is a holographic waveguide head-up display device.

[0027] According to a preferred embodiment, the holographic waveguide head-up display device includes a plurality of waveguides, wherein the image generation unit is further configured to stitch together images for each waveguide with smooth edges using a stitching algorithm.

[0028] According to a preferred embodiment, when the image display device is used in a vehicle, the display unit is the windshield of the vehicle.

[0029] According to a preferred embodiment, the input information includes text information and image information.

[0030] According to another aspect of this disclosure, corresponding computer program code and computer program products are also provided.

[0031] According to another aspect of this disclosure, an aircraft including the aforementioned image display device is also provided.

[0032] According to other aspects of this disclosure, a computer-readable storage medium having a program stored thereon is also provided, which, when executed by a processor, causes the computer to perform the steps of the image processing method described above.

[0033] The image display device and image processing method disclosed herein improve the imaging quality of the HUD.

[0034] These and other advantages of this disclosure will become more apparent from the following detailed description of preferred embodiments of this disclosure in conjunction with the accompanying drawings. Attached Figure Description

[0035] To further illustrate the above and other advantages and features of this disclosure, the specific embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. These drawings, together with the following detailed description, are included in and form a part of this specification. Elements having the same function and structure are indicated by the same reference numerals. It should be understood that these drawings only depict typical examples of this disclosure and should not be considered as limiting the scope of this disclosure. In the drawings:

[0036] Figure 1 A schematic view of a holographic waveguide HUD in the prior art is shown.

[0037] Figure 2 This is a flowchart of an image processing method 200 for improving the imaging quality of a holographic waveguide HUD according to an embodiment of the present disclosure.

[0038] Figure 3 A flowchart is shown of a method for obtaining photometric and geometric errors according to an embodiment of the present disclosure.

[0039] Figure 4 The arrangement of sensor 6 and HUD in the aircraft cockpit according to an embodiment of the present disclosure is schematically shown.

[0040] Figure 5 The illustration schematically shows a comparison between the virtual images produced by setting an edge absorber and not setting an edge absorber.

[0041] Figure 6 The illustration schematically shows a comparison of the virtual images produced by having corner protectors on and not having corner protectors on.

[0042] Figure 7 An example of an image stitching algorithm is shown.

[0043] Figure 8 This is a block diagram of an image display device 800 according to an embodiment of the present disclosure.

[0044] Figure 9This is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or apparatuses according to embodiments of the present invention can be implemented. Detailed Implementation

[0045] Exemplary embodiments of this disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer's specific goals, such as complying with constraints related to the system and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from this disclosure.

[0046] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0047] As mentioned earlier, traditional HUDs suffer from drawbacks such as large size and high cost. Furthermore, traditional HUDs also face the sunlight backflow issue, which requires the imaging plane to be positioned far from the focusing plane of the optical system, resulting in a smaller collimation distance. In addition, increasing the field of view (FOV) of a traditional HUD would significantly increase its size.

[0048] To address the aforementioned issues, HUDs based on optical waveguides have been developed. Optical waveguides based on thin-plate optics offer the following advantages: small size, low cost, support for larger collimation distances (due to the absence of sunlight backflow), and the ability to increase the field of view (FOV) simply by increasing the area of ​​the thin plate without increasing the size of the HUD.

[0049] Optical waveguides can be broadly classified into two types: geometric waveguides and diffractive waveguides. Geometric waveguides achieve image output through the stacking of arrayed mirrors. Based on traditional geometric optics design principles, geometric waveguides do not involve any micro- or nano-scale structures. Therefore, they can achieve very high levels of color and contrast in images. The drawback is the complex manufacturing process, requiring high yield rates at each step. This makes the overall yield of all process steps a challenge.

[0050] Diffractive waveguides mainly include surface relief grating waveguides fabricated using photolithography and holographic volume grating waveguides (VHG, holographic grating) or HOE (holographic optical element) fabricated using holographic interferometry. The design threshold for surface relief grating waveguides is higher than that of traditional optics, primarily because the structure of the diffraction grating enters the micro-nano scale, requiring simulation tools in physical optics; furthermore, the ray tracing after light enters the waveguide needs to be combined with traditional geometric optics simulation tools. The disadvantage of this technology is its higher manufacturing cost. Holographic volume grating waveguides use holographic volume grating elements instead of relief gratings. The volume of holographic waveguides can be reduced to one-fifth to one-tenth of the volume of traditional HUDs, significantly reducing the space occupied by HUDs in vehicles such as motor vehicles and aircraft. Moreover, the manufacturing cost of HUDs based on holographic waveguides is also greatly reduced.

[0051] Figure 1 An overall view of a holographic waveguide HUD (e.g., AR HUD) in the prior art is schematically shown.

[0052] As the name suggests, a holographic waveguide is composed of an optical waveguide and a volume holographic grating. The optical waveguide can be implemented using various glass plates, while the volume holographic grating refers to a diffraction grating manufactured using holographic exposure interference. For example... Figure 1 As shown, the holographic waveguide HUD includes an optical combiner 10, a waveguide 11, and an image generation unit 14. The waveguide 11 also includes an input grating 12 and an output grating 13. The principle of the optical waveguide is that after the image generation unit 14 images the image, the light beam passes through the input grating 12, enters the waveguide, and undergoes total internal reflection. The beam is then projected onto the optical combiner 10 via the output grating 13. Finally, the light beam projected onto the optical combiner 10 is reflected into the driver's eyes.

[0053] The input grating 12 and the output grating 13 are, for example, holographic grating elements.

[0054] Optical combiner 10 is typically positioned between the windshield of a vehicle or aircraft and the human eye. Due to its proximity to the eye, prolonged viewing can cause dizziness, reducing driving safety. Furthermore, the optical combiner 10 has a limited field of view (FOV). To address this issue, the windshield of a vehicle or aircraft can be used as a display screen. However, since both the inner and outer parallel surfaces of the windshield reflect light, the same incident light ray will produce two reflected rays. When the human eye receives both reflected rays, a double image is created, resulting in two distinct patterns.

[0055] To eliminate ghosting on the windshield, the current common solution is to use wedge-shaped PVB laminated glass. By adjusting the angle of the wedge-shaped film and the thickness of the windshield, the reflected light from the inner and outer surfaces of the windshield can be brought closer together, thus eliminating ghosting.

[0056] For example, P-pol optical combiner films on windshields improve image quality by reducing relative ghost luminance (RGL). This method works well within a small range of HUD virtual image distance (VID) and driver position, but its effectiveness is less than ideal when the driver's position is above or below the optimal viewing position. For instance, if drivers have significantly different heights, their optimal viewing positions will also differ considerably.

[0057] Alternatively, a method combining a transparent nanofilm with a Brewster-angle-based P-polarization optical engine can be used. However, this method affects image brightness and is difficult to meet windshield design requirements due to the strict incident angle.

[0058] A parallel vector technique was also developed, which uses optical and algorithmic techniques to achieve ghosting-free projection on windshields from 2 meters to optical infinity without the need for a wedge-shaped PVB film. However, this method is not universal because it is only applicable to free-form reflection systems based on off-axis three mirrors, and the larger the FOV, the larger the volume.

[0059] To address the problems in the prior art, this disclosure proposes an image processing method and an image display device for improving image quality. High-quality virtual images can be measured, for example, by aspects such as: brightness, chromaticity, uniformity, modulation transfer function (MTF), ghosting, distortion, warping, and eyebox (eye movement space) limitation.

[0060] The following is combined Figures 2 to 4 This describes an image processing method 200 for improving imaging quality according to embodiments of the present disclosure.

[0061] First, in step 201, a two-dimensional output image is generated based on the input information. It should be understood that the input information can be externally input and can include both textual and image information. Specifically, taking an aircraft application scenario as an example, a two-dimensional output image to be projected onto the aircraft's windshield can be generated based on information such as runway, speed, wind speed, and ground conditions provided by avionics equipment, to provide the pilot with the necessary information.

[0062] Next, in step 202, the light beam corresponding to the output image is guided to the display unit. Specifically, the light beam corresponding to the output image is guided, for example, through a waveguide and projected onto a display unit such as a windshield.

[0063] Next, in step 203, a three-dimensional image displayed on the display unit is captured using sensors arranged within the eye box area. Specifically, the image displayed on a display unit, such as the windshield, is captured by sensors arranged within the pilot's eye box area. It should be noted that the image captured by the sensors is three-dimensional. For example, an RGB-D camera can be used to capture an image on the windshield to obtain a three-dimensional image containing depth information.

[0064] Figure 4 An example of sensor placement inside an aircraft cockpit is shown. Figure 4 As shown, sensor 6 is positioned within the pilot's eye box area, and the virtual image it captures corresponds to the image seen by the pilot when looking straight ahead. The HUD is positioned near the control panel in front of the pilot.

[0065] It should be noted that if only factory settings are required, sensor 6 can be removed after image correction is completed. However, this disclosure is not limited to this, and sensor 6 can also be retained for real-time correction.

[0066] Next, in step 204, the photometric error and geometric error between the output image and the captured 3D image are obtained.

[0067] The following is combined Figure 3 The illustrated implementation describes how to obtain photometric and geometric errors.

[0068] Specifically, in step 2041, feature points of the output image and the captured 3D image are extracted respectively.

[0069] Next, in step 2042, the feature points of the output image are matched with the feature points of the captured 3D image.

[0070] Next, in step 2043, based on the matched feature points, triangulation is used to obtain the geometric error between the output image and the captured 3D image.

[0071] Triangulation, as is known, refers to determining the distance to a pixel by observing the angle between two points where the same pixel is located. The basic principle of triangulation is based on the properties of triangles; by measuring two angles and one side of a triangle, the lengths of the other sides can be calculated. In practical applications, a baseline is typically chosen, and the position of the target point can be calculated by measuring the angles from the endpoints of the baseline to the target point.

[0072] To avoid obscuring the contents of this disclosure, the specific steps for obtaining the geometric error between the output image and the captured 3D image using triangulation will not be described in detail here.

[0073] It should be understood that although the embodiments of this disclosure are based on triangulation to obtain the geometric error between the output image and the captured 3D image, this disclosure is not limited thereto, and the geometric error can be calculated based on any other suitable algorithm.

[0074] Next, in step 2044, the photometric error is obtained by calculating the mean square error between the pixels of the captured 3D image projected onto the 2D plane and the pixels of the 2D output image.

[0075] It should be understood that although the embodiments of this disclosure are based on mean square error to obtain the photometric error between the pixels of the captured three-dimensional image projected onto the two-dimensional plane and the pixels of the two-dimensional output image, this disclosure is not limited to this, and the photometric error can be obtained based on any other suitable algorithm.

[0076] It should also be understood that correction of geometric errors can improve image distortion, while correction of photometric errors can eliminate ghosting in images.

[0077] Turn back Figure 2 Next, in step 205, a loss function is constructed for photometric and geometric errors. For example, it can be based on... Figure 3 The implementation method constructs a loss function as shown in the following formula (1):

[0078]

[0079] in:

[0080] m represents the number of corresponding pixels between the output image and the captured 3D image;

[0081] P(pt i ) represents the projection of the pixels of the captured 3D image onto a 2D plane;

[0082] c i (∧, T, λ, θ) represents the corresponding pixel in the two-dimensional output image. The position of this pixel is related to the waveguide parameters ∧, T, λ, θ, where ∧ represents the grating period, T represents the grating thickness, λ represents the wavelength, and θ represents the incident angle of the light beam; and

[0083] Δ(d(pt i ) represents the geometric error of feature points in the captured 3D image calculated based on triangulation, where d represents the projection of all 3D feature points onto the 2D image plane to perform additional 2D deformation operations.

[0084] Next, in step 206, the loss function is solved using a PNP parser to obtain the translation and rotation matrices. Specifically, the loss function as shown in Equation (1) is iteratively solved using a PNP (Perspective-n-Point) parser, and the translation and rotation matrices used to correct the output image can be obtained by minimizing the loss function.

[0085] As is known, PnP (Progressive Point Numerical Transformation) is a method for solving the motion of 3D to 2D point pairs. It estimates the camera pose given the coordinates (relative to the world coordinate system) of n 3D points and their pixel coordinates, essentially solving for the rotation and translation matrices from the world coordinate system to the camera coordinate system. Methods for solving the PnP problem include direct linear transformation, P3P (Progressive Point Numerical Transformation), bundle adjustment, and others.

[0086] In the case of an RGB-D camera, the matching pixel coordinates and the corresponding depth information can be obtained. When matching a set of 3D points obtained from RGB-D with 2D points, the PNP algorithm can be used for camera pose estimation.

[0087] To avoid obscuring the content of this disclosure, the method of using a PNP parser to solve the loss function F(pt) will not be elaborated upon here. i c i The specific steps.

[0088] It should be understood that although the embodiments of this disclosure utilize a PNP parser to solve the loss function, this disclosure is not limited thereto, and any other suitable algorithm for solving the motion of three-dimensional to two-dimensional point pairs can be used to solve the loss function.

[0089] Finally, in step 207, the output image is corrected based on the translation and rotation matrices, and the corrected output image is displayed on the display unit. Specifically, the two-dimensional output image generated by the HUD's image generation unit is corrected based on the rotation and translation matrices output by the PNP resolver, and the corrected two-dimensional output image is projected onto a display unit such as a windshield via a waveguide.

[0090] As is known, image translation refers to shifting all pixels to a uniform position. Image rotation refers to rotating all pixel positions around the origin or any specified anchor point. Based on the translation and rotation matrices obtained by iterating the loss function through the PNP parser, all pixels in the output image initially generated by the image generation unit are translated and rotated, minimizing the distortion of the virtual image projected onto the windshield by the waveguide after translation and rotation, and eliminating ghosting.

[0091] The image processing method 200 according to the embodiments of the present disclosure enables the generation of high-quality virtual images, improves the pilot's viewing experience, and enhances operational safety.

[0092] The image processing method 200 for improving image quality according to this disclosure is particularly capable of enhancing the situational awareness of motion guidance for airport ground vehicles and improving ground operation safety, especially improving the pilot's operating experience during the taxiing phase. However, it should be understood that this disclosure is not limited to airport applications, but can also be applied to motion guidance in scenarios involving other modes of transportation such as motor vehicles and trains.

[0093] According to a preferred embodiment of this disclosure, in order to further improve the imaging quality of the holographic waveguide HUD, the following settings may also be considered.

[0094] For example, a tilted surface relief grating can be used to enhance the diffraction effect of the input grating. A folded grating and an output grating use a holographic diffraction grating. The combined use of a tilted surface relief grating and a holographic diffraction grating can increase the brightness of a virtual image to, for example, 6800 nits.

[0095] For example, edge absorbers can be placed on the waveguide to remove ghosting caused by edge reflections. Figure 5 The upper view of view (a) schematically shows edge reflection, and the lower view shows ghosting in the image due to edge reflection. Figure 5 The upper view of view (b) shows the arrangement of an edge absorber in the waveguide, and the lower view shows that the ghosting in the image disappears when the edge absorber is arranged.

[0096] For example, corner protectors can be placed on the waveguide to remove ghosting caused by stray light on the image generation unit. Figure 6 The upper view of view (a) schematically shows stray light on the image generation unit, and the lower view shows ghosting in the image due to stray light. Figure 6 The upper view of view (b) shows the arrangement of a corner protector in the waveguide, and the lower view shows that the ghosting in the image disappears when the corner protector is arranged.

[0097] Furthermore, a stitching algorithm can be used in the image generation unit to stitch together holographic images of multiple waveguides, resulting in a smooth edge for the stitched image, which is then coupled into the waveguide. This increases the field of view (FOV) to further improve the user experience. Figure 7 Examples of known image stitching algorithms are shown. For example... Figure 7 As shown, the stitching algorithm includes packing, twisting, and merging images in the image stream, and finally stitching them together.

[0098] It should be understood that the content of this disclosure is not limited to... Figure 7 The example shown is not a direct representation of an image stitching algorithm, but can be applied to any image stitching algorithm.

[0099] The methods discussed above can be implemented entirely by a computer-executable program, or partially or entirely using hardware and / or firmware. When implemented with hardware and / or firmware, or when a computer-executable program is loaded into a hardware device capable of running the program, an image display device, as described below, is realized. Hereinafter, an overview of these devices is given without repeating some details already discussed above; however, it should be noted that while these devices can perform the methods described above, the methods may not necessarily employ or be performed by those components of the described device.

[0100] Figure 8 An image display device 800 is shown. The image display device 800 includes an image generation unit 801, a display unit 802, an optical path transmission unit 803, a three-dimensional sensor 804, and an error calculation unit 805. The image generation unit 801 is configured to generate a two-dimensional output image based on input information. The display unit 802 is configured to display the output image. The optical path transmission unit 803 is configured to guide a light beam corresponding to the output image to the display unit. The three-dimensional sensor 804 is arranged to capture the image displayed on the display unit. The error calculation unit 805 is configured to calculate the error between the three-dimensional image captured by the three-dimensional sensor and the output image. The image generation unit 801 is also configured to correct the output image based on the error, so as to display the corrected output image on the display unit 802.

[0101] According to a preferred embodiment, a three-dimensional sensor 804 is arranged within the viewer's eye box area.

[0102] According to a preferred embodiment, the error calculation unit 805 is further configured to obtain the photometric error and geometric error between the captured 3D image and the output image, construct a loss function based on the photometric error and geometric error, and solve the loss function using an algorithm for solving the motion of 3D-to-2D point pairs. The photometric error is obtained by calculating the mean square error between the pixels of the captured 3D image projected onto the 2D plane and the pixels of the output image. The geometric error is obtained by performing triangulation on the captured 3D image and the output image.

[0103] According to a preferred embodiment, the algorithm for solving the motion of three-dimensional to two-dimensional point pairs is the PNP algorithm, and the image generation unit 801 is further configured to correct the output image based on the translation matrix and rotation matrix obtained by solving the loss function through the PNP algorithm.

[0104] According to a preferred embodiment, the optical path transmission unit 803 is a waveguide equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit 802 and a corner protector for removing distortion generated when the output image is projected onto the display unit 802.

[0105] According to a preferred embodiment, the image display device 800 is a holographic waveguide HUD, which includes multiple waveguides, and the image generation unit 801 is also configured to stitch together the images for each waveguide with smooth edges using a stitching algorithm.

[0106] According to a preferred embodiment, the display unit 802 is the windshield of a vehicle.

[0107] According to a preferred embodiment, the three-dimensional sensor 804 can be removed from the image display device 800 after image correction is completed once.

[0108] Figure 8 The image display device 800 shown corresponds to Figure 2 and Figure 3 The image processing method 200 is shown. Therefore, the relevant details of each unit or device in the image display device 800 have already been described in the previous section. Figure 2 and Figure 3 The image processing method 200 is described in detail here and will not be repeated here.

[0109] The image display device 800 according to embodiments of this disclosure has a reduced size, lower cost, and produces high-quality virtual images, i.e., the distortion of the virtual image projected onto the windshield is minimized and ghosting is eliminated, which improves the pilot's viewing experience and enhances operational safety.

[0110] Each component module and unit in the above-described device can be configured via software, firmware, hardware, or a combination thereof. Specific means or methods of configuration are well known to those skilled in the art and will not be elaborated upon here. When implemented via software or firmware, data can be transferred from a storage medium or network to a computer with a dedicated hardware architecture (e.g., ...). Figure 9 The general-purpose computer 900 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc.

[0111] Figure 9 This is a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or devices according to embodiments of this disclosure can be implemented. For example... Figure 9As shown, the Central Processing Unit (CPU) 901 performs various processes based on programs stored in the Read-Only Memory (ROM) 902 or programs loaded into the Random Access Memory (RAM) 903 from the storage device 908. The RAM 903 also stores data required as needed when the CPU 901 performs various processes, etc. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output interface 905 is also connected to the bus 904.

[0112] The following components are connected to input / output interface 905: input device 906 (including keyboard, mouse, etc.), output device 907 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage device 908 (including hard disk, etc.), and communication device 909 (including network interface card, such as LAN card, modem, etc.). Communication device 909 performs communication processing via a network such as the Internet. Drive 910 may also be connected to input / output interface 905 as needed. Removable media 911, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed, so that computer programs read from them can be installed into storage device 908 as needed.

[0113] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as a removable medium 911.

[0114] Those skilled in the art will understand that such storage media are not limited to Figure 9 The illustration shows a removable medium 911 containing a program, distributed separately from the device to provide the program to the user. Examples of removable media 911 include disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-disk (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 902, a hard disk included in storage device 908, etc., containing programs and distributed to the user along with the device containing them.

[0115] This disclosure also provides corresponding computer program code and a computer program product storing machine-readable instruction code. When the instruction code is read and executed by a machine, the image processing method described above according to the embodiments of this disclosure can be performed.

[0116] Accordingly, the storage medium configured to carry the aforementioned program product storing machine-readable instruction code is also included in this disclosure. The storage medium includes, but is not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0117] The computing device including the aforementioned storage medium also includes the image display device disclosed herein, for example, an image display device including a processor and a storage medium storing program code, which, when executed by the processor, implements the image processing method described above according to the embodiments of this disclosure.

[0118] Based on the above description, the embodiments of this disclosure provide the following technical solutions, but are not limited thereto.

[0119] Solution 1. An image processing method, comprising:

[0120] Generate a two-dimensional output image based on the input information;

[0121] Guide the light beam corresponding to the output image to the display unit;

[0122] Capture the 3D image displayed on the display unit;

[0123] Calculate the error between the captured 3D image and the output image; and

[0124] The output image is corrected based on the error so that the corrected output image can be displayed on the display unit.

[0125] Scheme 2. Based on the image processing method of Scheme 1, the calculation error also includes the photometric error and geometric error between the captured 3D image and the output image.

[0126] Option 3. Based on the image processing method of Option 2, the calculation of error further includes constructing a loss function for photometric error and geometric error, and solving the loss function through an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

[0127] Option 4. Based on the image processing method of Option 3, the algorithm used to solve the motion of three-dimensional to two-dimensional point pairs is the perspective n-point problem (PNP) algorithm.

[0128] Scheme 5. The image processing method according to Scheme 4 also includes correcting the output image based on the translation and rotation matrices obtained by solving the loss function through the perspective n-point problem (PNP) algorithm.

[0129] Scheme 6. The image processing method according to any one of Schemes 1 to 5, wherein the photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto the two-dimensional plane and the pixels of the output image.

[0130] Scheme 7. An image processing method according to any one of Schemes 1 to 5, wherein the geometric error is obtained by performing triangulation on the captured 3D image and the output image.

[0131] Scheme 8. The image processing method according to Scheme 1, wherein a beam corresponding to the output image is guided to the display unit via a waveguide.

[0132] Scheme 9. The image processing method according to Scheme 8, wherein the waveguide is equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit and a corner protector for removing distortion generated when the output image is projected onto the display unit.

[0133] Scheme 10. The image processing method according to Scheme 8 further includes using a stitching algorithm to stitch together the images of each waveguide in multiple waveguides with smooth edges.

[0134] Solution 11. The image processing method according to Solution 1, wherein a three-dimensional image displayed on a display unit is captured by a sensor arranged in the viewer's eye box area.

[0135] Scheme 12. The image processing method according to Scheme 1, wherein the input information includes text information and image information.

[0136] Solution 13. An image display device, comprising:

[0137] An image generation unit is configured to generate a two-dimensional output image based on input information;

[0138] The display unit is configured to display the output image;

[0139] An optical path transmission unit is configured to guide a light beam corresponding to the output image to a display unit;

[0140] A three-dimensional sensor is arranged to capture an image displayed on a display unit; and

[0141] An error calculation unit is configured to calculate the error between the 3D image captured by the 3D sensor and the output image.

[0142] The image generation unit is also configured to correct the output image based on the error so that the corrected output image can be displayed on the display unit.

[0143] Option 14. The image display device according to Option 13, wherein a three-dimensional sensor is arranged within the viewer's eye box area.

[0144] Option 15. The image display device according to Option 14, wherein the error calculation unit is further configured to obtain the photometric error and geometric error between the captured three-dimensional image and the output image.

[0145] Option 16. The image display device according to Option 15, wherein the error calculation unit is further configured to construct a loss function for photometric error and geometric error, and to solve the loss function by means of an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

[0146] Scheme 17. The image display device according to Scheme 16, wherein the algorithm for solving the motion of three-dimensional to two-dimensional point pairs is the perspective n-point problem (PNP) algorithm.

[0147] Scheme 18. The image display device according to Scheme 17, wherein the image generation unit is further configured to correct the output image based on the translation matrix and rotation matrix obtained by solving the loss function through the perspective n-point problem (PNP) algorithm.

[0148] Option 19. The image display device according to Option 15, wherein the photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto the two-dimensional plane and the pixels of the output image.

[0149] Option 20. The image display device according to Option 15, wherein the geometric error is obtained by performing triangulation on the captured three-dimensional image and the output image.

[0150] Option 21. The image display device according to Option 13, wherein the optical path transmission unit is a waveguide.

[0151] Option 22. The image display device according to Option 21, wherein the waveguide is equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit and a corner protector for removing distortion generated when the output image is projected onto the display unit.

[0152] Option 23. An image display device according to Option 13, wherein the image display device is a holographic waveguide head-up display device.

[0153] Option 24. The image display device according to Option 23, wherein the holographic waveguide head-up display device includes multiple waveguides, and wherein the image generation unit is further configured to stitch together images for each waveguide with smooth edges using a stitching algorithm.

[0154] Option 25. The image display device according to Option 13, wherein, when the image display device is used in a vehicle, the display unit is the windshield of the vehicle.

[0155] Option 26. The image display device according to Option 13, wherein the input information includes text information and image information.

[0156] Scheme 27. A computer program product comprising a computer program that, when executed by a processor, implements the steps of an image processing method according to any one of Schemes 1 to 12.

[0157] Scheme 28. A computer-readable storage medium having a program stored thereon, the program causing the computer to perform an image processing method according to any one of Schemes 1 to 12 when executed by a processor.

[0158] Scheme 29. An aircraft comprising an image display device according to any one of Schemes 13 to 26.

[0159] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Furthermore, unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0160] While the embodiments of this disclosure have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative of this disclosure and do not constitute a limitation thereof. Those skilled in the art can make various modifications and alterations to the above embodiments without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure is defined only by the appended claims and their equivalents.

Claims

1. An image processing method, comprising: Generate a two-dimensional output image based on the input information; The beam of light corresponding to the output image is guided to the display unit; Capture the three-dimensional image displayed on the display unit; Calculate the error between the captured 3D image and the output image; as well as The output image is corrected based on the error so that the corrected output image is displayed on the display unit.

2. The image processing method according to claim 1, wherein, The calculation of the error also includes obtaining the photometric error and geometric error between the captured 3D image and the output image.

3. The image processing method according to claim 2, wherein, The calculation of the error also includes constructing a loss function for the photometric error and the geometric error, and solving the loss function using an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

4. The image processing method according to claim 3, wherein, The algorithm described for solving the motion of point pairs from three dimensions to two dimensions is the perspective n-point problem algorithm.

5. The image processing method according to claim 4 further includes correcting the output image based on the translation matrix and rotation matrix obtained by solving the loss function through the perspective n-point problem algorithm.

6. The image processing method according to claim 2, wherein, The photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto the two-dimensional plane and the pixels of the output image.

7. The image processing method according to claim 2, wherein, The geometric error is obtained by performing triangulation on the captured 3D image and the output image.

8. The image processing method according to claim 1 further includes using a stitching algorithm to stitch together the generated multiple output images with smooth edges, and guiding the light beam corresponding to the stitched output image to the display unit.

9. The image processing method according to claim 1, wherein, The input information includes text information and image information.

10. An image display device, comprising: An image generation unit is configured to generate a two-dimensional output image based on input information; A display unit configured to display the output image; An optical path transmission unit is configured to guide a light beam corresponding to the output image to the display unit; A three-dimensional sensor is arranged to capture images displayed on the display unit; and An error calculation unit is configured to calculate the error between the three-dimensional image captured by the three-dimensional sensor and the output image. The image generation unit is further configured to correct the output image based on the error, so as to display the corrected output image on the display unit.

11. The image display device according to claim 10, wherein, The three-dimensional sensor is positioned within the viewer's eye box area.

12. The image display device according to claim 11, wherein, The error calculation unit is also configured to obtain the photometric error and geometric error between the captured 3D image and the output image.

13. The image display device according to claim 12, wherein, The error calculation unit is further configured to construct a loss function for the photometric error and the geometric error, and to solve the loss function using an algorithm for solving the motion of three-dimensional to two-dimensional point pairs.

14. The image display device according to claim 13, wherein, The algorithm for solving the motion of three-dimensional to two-dimensional point pairs is a perspective n-point problem algorithm, and the image generation unit is further configured to correct the output image based on the translation and rotation matrices obtained by solving the loss function through the perspective n-point problem algorithm.

15. The image display device according to claim 12, wherein, The photometric error is obtained by calculating the mean square error between the pixels of the captured three-dimensional image projected onto the two-dimensional plane and the pixels of the output image.

16. The image display device according to claim 12, wherein, The geometric error is obtained by performing triangulation on the captured 3D image and the output image.

17. The image display device according to claim 10, wherein, The optical transmission unit is a waveguide.

18. The image display device according to claim 17, wherein, The waveguide is equipped with at least one of an edge absorber for removing ghosting generated when the output image is projected onto the display unit and a corner protector for removing distortion generated when the output image is projected onto the display unit.

19. The image display device according to claim 10, wherein, The optical path transmission unit includes multiple waveguides, and the image generation unit is further configured to stitch together images for each waveguide with smooth edges using a stitching algorithm.

20. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the image processing method according to any one of claims 1 to 9.

21. An aircraft comprising an image display device according to any one of claims 10 to 19.

Images