Head-mounted display device and display method

Abstract

The invention provides a head-mounted display device and a display method. The head-mounted display device includes a display, an optical system, and a processor. The display is configured to display the pre-deformed image. The optical system is configured to receive the pre-deformed image and output an undistorted image. The processor is configured to perform a tolerance enhancement operation on the original image to produce an enhanced image. Further, the processor is configured to apply software distortion to the enhanced image to produce a pre-deformed image. The software distortion is configured to compensate for optical distortion of the optical system.

Description

Technical Field

[0001] The present disclosure relates to a head-mounted display device; specifically, the present disclosure relates to a head-mounted display device and a display method. Background Art

[0002] In order to bring an immersive experience to users, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR) and mixed reality (MR), are being developed. AR technology enables users to bring virtual elements into the real world. VR technology enables users to enter a new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to bring a complete immersive experience to users, visual content, audio content or other sensory content can be provided through one or more devices. Summary of the invention

[0003] The present disclosure relates to a head mounted display device and a display method to improve the quality of an image displayed to a user.

[0004] In the present disclosure, a head-mounted display device is provided. The head-mounted display device includes a display, an optical system, and a processor. The display is configured to display a pre-deformed image. The optical system is configured to receive the pre-deformed image and output an undistorted image. The processor is configured to perform a tolerance enhancement operation on the original image to generate an enhanced image. In addition, the processor is configured to apply software distortion to the enhanced image to generate the pre-deformed image. The software distortion is configured to compensate for the optical distortion of the optical system.

[0005] In the present disclosure, a display method of a head mounted display device is provided. The display method includes: performing a tolerance enhancement operation on an original image to generate an enhanced image; applying software distortion to the enhanced image to generate a pre-deformed image, wherein the software distortion is configured to compensate for an optical distortion of an optical system; displaying the pre-deformed image; receiving the pre-deformed image through an optical system; and outputting a distortion-free image through the optical system.

[0006] Based on the above content, according to the head-mounted display device and the display method, a distortion-free image can be displayed to the user without significantly reducing the quality of the image, thereby improving the user experience.

[0007] In order to make the foregoing content easier to understand, several embodiments attached with the drawings will be described in detail below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure includes accompanying drawings to provide a further understanding of the present disclosure, and the accompanying drawings are incorporated in and constitute a part of this specification. The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure.

[0009] Figure 1 Schematic diagram of the hardware structure of a head mounted display device according to an embodiment of the present disclosure.

[0010] Figure 2 Schematic diagram of the software structure of the head mounted display device according to an embodiment of the present disclosure.

[0011] Figure 3A It is a schematic diagram of a display scene of a head-mounted display device according to an embodiment of the present disclosure.

[0012] Figure 3B It is a schematic diagram of a display scene of a head-mounted display device according to an embodiment of the present disclosure.

[0013] Figure 4A Schematic diagram of the hardware structure of a head mounted display device according to an embodiment of the present disclosure.

[0014] Figure 4B is a schematic diagram of a training scenario of a tolerance enhancement model according to an embodiment of the present disclosure.

[0015] Figure 5 is a schematic flowchart of a display method of a head mounted display device according to an embodiment of the present disclosure.

[0016] [Explanation of Symbols]

[0017] 100, 400: Head-mounted display (HMD) device

[0018] 110: Processor

[0019] 120: Display

[0020] 130: Optical system

[0021] 140: Memory

[0022] 200: Software Structure

[0023] 300A, 300B: Display scene

[0024] 301, 303, 305A, 305B: Image

[0025] 400B: Training scenario

[0026] 410: First Simulation

[0027] 420: Second Simulation

[0028] 500: Display method

[0029] C n :Difference

[0030] IMG_E: enhanced image

[0031] IMG_O: Original image

[0032] IMG_P: Pre-deformed image

[0033] IMG_U: Undistorted image

[0034] MD: Tolerance Enhanced Model

[0035] NN: Neural Network

[0036] S510, S520, S530, S540, S550: Steps

[0037] SDO: Software Distortion Operation

[0038] TEO: Tolerance Enhanced Operation

[0039] x n ,y n , Intrinsic Value

[0040] θ: training parameters / parameters DETAILED DESCRIPTION

[0041] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the description to refer to the same or like components.

[0042] Certain terms are used throughout the specification and claims of this disclosure to refer to specific components. It will be understood by those skilled in the art that electronic device manufacturers may refer to the same components by different names. It is not intended herein to distinguish between components that have the same function but different names. In the following description and claims, words such as "comprise" and "include" are open-ended terms and should be interpreted as "including but not limited to...".

[0043] The term "coupling (coupling) (or connection (connection))" used throughout the entire specification of this application (including the attached claims) may refer to any direct or indirect connection mode. For example, if the text states that a first device is coupled (or connected) to a second device, it should be interpreted as that the first device can be directly connected to the second device, or that the first device can be indirectly connected to the second device through other devices or certain connection modes to be connected. The terms "first (first)", "second (second)" and similar terms mentioned throughout the entire specification of this application (including the attached claims) are only used to name discrete elements or to distinguish different embodiments or scopes. Therefore, the terms should not be regarded as limiting the upper or lower limits of the number of elements and should not be used to limit the order of arrangement of elements. In addition, as much as possible, the elements / components / steps using the same reference numbers in the drawings and embodiments represent the same or similar parts. In different embodiments, the same reference numbers or the same terms can be used to refer to the relevant descriptions of the elements / components / steps.

[0044] It should be noted that in the following embodiments, the technical features of several different embodiments may be replaced, recombined and mixed to complete other embodiments without departing from the spirit of the present disclosure. As long as the features of each embodiment do not violate the spirit of the present disclosure or conflict with each other, the embodiments may be mixed and used together at will.

[0045] In order to bring an immersive experience to users, technologies related to extended reality (XR), such as augmented reality (AR), virtual reality (VR) and mixed reality (MR), are being developed. AR technology enables users to bring virtual elements into the real world. VR technology enables users to enter a new virtual world to experience a different life. MR technology merges the real world with the virtual world. In addition, in order to bring a complete immersive experience to users, visual content, audio content or other sensory content can be provided through one or more devices.

[0046] Lenses can be used to provide a comfortable display experience to users. However, lenses may cause distortion in the image displayed to the user. Even though the distortion can be reduced by applying an inverse of the distortion to the image, the quality of the image may be reduced during the distortion and / or the inverse of the distortion. Therefore, those skilled in the art seek to display a distortion-free image to the user without significantly reducing the quality of the image.

[0047] Figure 1 Schematic diagram of the hardware structure of a head-mounted display (HMD) device according to an embodiment of the present disclosure. Figure 1, the hardware structure of the HMD device 100 may include a processor 110, a display 120, and an optical system 130. The processor 110 may be coupled to the display 120. The display 120 may be further coupled to the optical system 130.

[0048] In one embodiment, the display 120 may be configured to display a pre-warp image. The optical system 130 may be configured to receive the pre-warp image and output an undistorted image. The processor 110 may be configured to perform a tolerance enhancement operation on the original image to generate an enhanced image. In addition, the processor 110 may be configured to apply software distortion to the enhanced image IMG_E to generate the pre-warp image. In addition, the software distortion may be configured to compensate for the optical distortion of the optical system 130.

[0049] In this way, since the original image is enhanced by the tolerance enhancement operation, the quality of the image displayed to the user (i.e., the distortion-free image) can be almost the same as the quality of the original image. In other words, the distortion-free image can be displayed to the user without significantly reducing the quality of the image, thereby improving the user experience.

[0050] In one embodiment, the HMD device 100 may be configured to display augmented reality (AR), virtual reality (VR), mixed reality (MR), or extended reality (XR) content. The HMD device 100 may include, for example, a headset, wearable glasses (e.g., AR / VR goggles), other similar devices suitable for AR, VR, MR, XR, or other reality-related technologies, or a combination of these devices. However, the present disclosure is not limited thereto.

[0051] In one embodiment, the processor 110 may include, for example, a microcontroller unit (MCU), a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a programmable controller, a programmable logic device (PLD), other similar devices or a combination of these devices. The present disclosure is not limited to this. In addition, in an embodiment, each of the functions of the processor 110 can be implemented as a plurality of program codes. The program code is stored in a memory and executed by the processor 110. Alternatively, in an embodiment, each of the functions of the processor 110 can be implemented as one or more circuits. That is, the present disclosure does not limit whether software or hardware is used to implement the functions of the processor 110.

[0052] In one embodiment, the display 120 may include, for example, an organic light-emitting diode (OLED) display device, a small light-emitting diode (LED) display device, a micro LED display device, a quantum dot (QD) LED display device, a liquid crystal display (LCD) display device, a tiled display device, a foldable display device, or an electronic paper display (EPD). However, the present disclosure is not limited thereto.

[0053] In one embodiment, the optical system 130 may include a lens, and the optical distortion is at least partially caused by the lens. However, the present disclosure is not limited thereto.

[0054] Figure 2 Schematic diagram of the software structure of the head mounted display device according to an embodiment of the present disclosure. Figure 1 and Figure 2 , Figure 1 and Figure 2 A software structure 200 of the HMD device 100 is depicted. The software structure 200 may include a tolerance enhancement operation TEO, a software distortion operation SDO, a display 120, and an optical system 130. The data involved in the software structure 200 may include an original image IMG_O, an enhanced image IMG_E, a pre-deformed image IMG_P, and an undistorted image IMG_U.

[0055] It should be noted that the optical system 130 may include a lens, and the lens may cause optical distortion of the original image IMG_O, which is the image to be displayed to the user through the panel of the display 120 and the optical system 130. Therefore, in order to compensate for the optical distortion caused by the lens of the optical system 130, a software distortion operation SDO may be used to apply software distortion. Software distortion may be the inverse of optical distortion. In other words, when both software distortion and optical distortion are applied, the software distortion and optical distortion may compensate for each other. However, the quality of the image may also be reduced due to the simultaneous application of two distortions (i.e., software distortion and optical distortion).

[0056] In order to compensate for the quality loss caused by the two distortions, a tolerance enhancement operation TEO may be performed on the original image IMG_O. The tolerance enhancement operation TEO may be configured to increase the tolerance for the distortion of the original image IMG_O. For example, it is assumed that the quality score of the original image IMG_O may be 100 and the quality score of the image after the two distortions (e.g., the undistorted image IMG_U) may be 80. That is, the quality score may be reduced by 20 points. By performing the tolerance enhancement operation TEO on the original image IMG_O, the quality score of the enhanced image IMG_E may be 120. Therefore, even if the two distortions are applied to the enhanced image IMG_E (i.e., the quality score may be reduced by 20 points), the quality score of the undistorted image IMG_U may still be 100.

[0057] In this way, since the original image IMG_O is enhanced by the tolerance enhancement operation TEO, the quality of the image displayed to the user (i.e., the distortion-free image IMG_U) can be almost the same as the quality of the original image IMG_O. In other words, the distortion-free image IMG_U can be displayed to the user without significantly reducing the quality of the image, thereby improving the user experience.

[0058] Figure 3A is a schematic diagram of a display scene of a head mounted display device according to an embodiment of the present disclosure. Figure 3A , display scene 300A illustrates the image distortion caused by the lens.

[0059] In order to reduce the size and weight of a VR device (e.g., HMD device 100), a display device with a relatively small area can be used and placed at a relatively close distance from the user's eyes. In such a small area and close distance display hardware setting, the usability and comfort of the VR device must still be considered. Therefore, it is necessary to use an optical lens. The function of the optical lens is to project the image of the near display device into an enlarged virtual image at a long distance. The optical lens is used to shift the direction of light propagation so that the image of the near display device is focused on the retina and forms a virtual image at a distance. In this way, when the crystalline lens of the eye remains relaxed, the user can clearly see the image from the near display device, as if the displayed image is presented at a distance.

[0060] However, when a virtual image is formed at a distance through an optical lens, the virtual image must have a magnification effect, thus generating pincushion distortion. For example, after image 301 passes through the optical lens, image 301 may be distorted and become image 305A.

[0061] Figure 3B is a schematic diagram of a display scene of a head mounted display device according to an embodiment of the present disclosure. Figure 3B , display scene 300B illustrates image distortion caused by software and image distortion caused by a lens.

[0062] In theory, it is possible to design an optical lens that can perfectly produce an undistorted virtual image. However, limited by the cost and volume of VR devices, the VR imaging system of VR devices generally uses an optical lens with distortion and is equipped with lens distortion correction software for correcting the distorted virtual image. The lens distortion correction software can be configured to utilize the complementary characteristics of barrel distortion and pincushion distortion to correct the distorted virtual image generated by the optical lens. First, the lens distortion correction software can be configured to analyze the pincushion distortion image generated by the lens. In addition, the lens distortion correction software can be configured to calculate a barrel distortion model that is complementary to the pincushion distortion image. The calculated barrel distortion model can then be applied to the undistorted image originally to be presented on the display device, thereby converting the undistorted image into a barrel distortion image, and then outputting the barrel distortion image on the display device. That is, the image after applying the barrel distortion model can be displayed on the panel of the display device and then passed through the lens. In other words, referring to Figure 2, the software distortion of the software distortion operation SDO may be barrel distortion, and the optical distortion of the optical system 130 may be pincushion distortion. In this way, the undistorted image IMG_U as the original image IMG_O may be displayed to the user.

[0063] Reference Figure 3B , the image 301 to be displayed to the user may be barrel distorted and become the image 303. Then, the image 303 may be pincushion distorted and become the image 305B. It should be noted that the pincushion distortion and the barrel distortion may generate disproportionate distortions in the periphery of the image 301 and the center of the image 301. For example, the pincushion distortion may enlarge the peripheral portion of the image 301 more than the central portion of the image 301. On the other hand, the barrel distortion may shrink the peripheral portion of the image 301 more than the central portion of the image 301. That is, if the barrel distortion is applied to the image 301 and the barrel distortion of the image 301 is restored using the pincushion distortion of the lens, the peripheral portion of the image 301 will undergo a drastic process of first shrinking and then enlarging. Such a process may greatly degrade the details of the peripheral portion of the image 301, thereby causing the user to clearly perceive that the peripheral portion of the image 305B is blurry.

[0064] In order to retain the details of the peripheral part of the image 301 after the distortion, the tolerance enhancement operation TEO may be performed on the image 301. It should be noted that when the user tries to identify the object in the image 305B, the shape of the object plays the most critical role. For example, if the shape details of the object in the peripheral part of the image 301 are lost during the distortion, the user may not be able to identify the object in the peripheral part of the image 305B. Therefore, in order to improve the recognizability of the object in the peripheral part of the image 305B, the shape of the object in the peripheral part of the image 301 may be enhanced by sharpening the edge of the object in the peripheral part of the image 301. After the edge of the object in the peripheral part of the image 301 is sharpened, the object may have a greater tolerance to the distortion. That is, after the edge of the object in the peripheral part of the image 301 is sharpened, the user may be able to easily identify the object in the image 305B. In other words, since the key information is not lost during the distortion, the user can still use the key information to identify the object. That is, the tolerance enhancement operation TEO may include a sharpening operation performed on the edge of the peripheral part of the original image IMG_O. However, the present disclosure is not limited thereto.

[0065] On the other hand, it may also be considered to discard unnecessary information and / or unimportant information instead of enhancing the key information in the original image IMG_O. That is, when the user tries to identify the object in the image 305B, the unnecessary information and / or unimportant information may not be helpful. What is more, when the user tries to identify the object in the image 305B, the unnecessary information and / or unimportant information may confuse the user. For example, when the object is a cloth or a pillar, the texture of the cloth or the pillar may not be necessary information or important information for the user to identify the object compared to the sharpening of the cloth or the pillar. Therefore, in order to improve the recognizability of the object in the peripheral part of the image 305B, the shape of the object in the peripheral part of the image 301 may be enhanced by blurring the texture of the object in the peripheral part of the image 301. After the texture of the object in the peripheral part of the image 301 is blurred, the object may have a greater tolerance to distortion. That is, after the texture of the object in the peripheral part of the image 301 is blurred, the user may be able to identify the object in the image 305B based only on the key information. In other words, since unnecessary information and / or unimportant information does not interfere with the user's judgment, the user can only use key information to identify the object. That is, the tolerance enhancement operation TEO may include a blurring operation on the texture of the peripheral portion of the original image IMG_O. However, the present disclosure is not limited thereto.

[0066] In one embodiment, both the sharpening operation of the edge of the object and the blurring operation of the texture of the object can be performed, thereby significantly improving the recognizability of the object. In this way, the user experience can be improved.

[0067] Figure 4A Schematic diagram of the hardware structure of the head mounted display device according to an embodiment of the present disclosure. Figure 1 and Figure 4A , the HMD device 400 may include a hardware structure similar to that of the HMD device 100, and the HMD device 400 may further include a memory 140. The memory 140 may be configured to store the tolerance enhancement model MD.

[0068] In one embodiment, the processor 110 may be configured to perform a tolerance enhancement operation TEO based on a tolerance enhancement model MD. That is, the tolerance enhancement model MD may be used to enhance the tolerance of the distortion of the original image IMG_O. In addition, the tolerance enhancement model MD may be pre-trained using a machine learning algorithm. However, the present disclosure is not limited thereto.

[0069] In one embodiment, the memory 140 may include, for example, a NAND flash memory core, a NOR flash memory core, a static random access memory (SRAM) core, a dynamic random access memory (DRAM) core, a magnetoresistive random access memory (MRAM) core, a phase change memory (PCM) core, a resistive random access memory (ReRAM) core, a three-dimensional (3D) XPoint memory core, a ferroelectric random access memory (FeRAM) core, and other types of memory cores suitable for storing data. However, the present disclosure is not limited thereto.

[0070] Figure 4B is a schematic diagram of a training scenario of a tolerance enhancement model according to an embodiment of the present disclosure. Figure 4A and Figure 4B , training scenario 400B illustrates how the tolerance enhancement model can be trained.

[0071] Reference Figure 4B , x n The image eigenvalues ​​x can represent the original image IMG_O. The image eigenvalues ​​can be quantitative values ​​extracted by image analysis, which can be used to describe or distinguish the properties of the image. For example, the simplest method is to use the pixel value of the image as the eigenvalue of the image. That is, x n Can represent n pixel values ​​of an image. It can also represent the intrinsic value of an image. The corresponding image may be an ideal image for simulating the visual experience received by the user through the VR optical imaging system (e.g., the display 120 and the optical system 130). In one embodiment, the method for simulating the ideal image received through the VR optical imaging system may be to perform a first simulation 410 on the original image IMG_O. For example, the first simulation 410 may use a low-pass filter to moderately blur the original image IMG_O, but the present disclosure is not limited thereto. Next, the intrinsic value may be extracted from the simulated ideal image

[0072] To train the tolerance enhancement model MD, the intrinsic value xn As input, and the output can be the intrinsic value y n In one embodiment, the neural network NN in the tolerance enhancement model MD can be used to adjust the input (e.g., the intrinsic value x n ) to process and output the result (for example, the intrinsic value y n ) and generate training parameters θ based on the input and the output. The training parameters θ can be applied to the neural network NN to transform the intrinsic value x n Convert to intrinsic value y n Then, a second simulation 420 may be used to simulate the process of the image passing through the optical lens. n can be the input of the second simulation 420, and the inherent value may be the output of the second simulation 420. The corresponding image may be a simulated image of the undistorted image IMG_U.

[0073] By iteratively training the neural network NN, the simulated image can be made closer to the ideal image. As close to the intrinsic value as possible The training parameter θ can be found by using mathematical optimization methods. The process of finding the training parameter θ can use recursive optimization. That is, the intrinsic value With intrinsic value The difference between n Continuously feed back to the neural network NN as the basis for gradually modifying the parameters θ until the difference C n The expected standard is met. In other words, the tolerance enhancement model MD can be pre-trained using a machine learning algorithm based on a comparison of an ideal image of the original image IMG_O and a simulated image of the enhanced image IMG_E. In addition, the ideal image can be the output of a low-pass filter, and the original image IMG_O can be the input of the low-pass filter. In addition, the simulated image can be a simulation result of the process of the enhanced image IMG_E passing through the optical system 130. However, the present disclosure is not limited to this.

[0074] After pre-training the tolerance enhancement model MD, the tolerance enhancement model MD may be stored in the memory 140 of the HMD device 400. Therefore, the processor 110 may be configured to perform a tolerance enhancement operation TEO based on the tolerance enhancement model MD.

[0075] In this way, since the original image is enhanced by the tolerance enhancement operation based on the tolerance enhancement model MD, the quality of the image displayed to the user (i.e., the distortion-free image) can be almost the same as the quality of the original image. In other words, the distortion-free image can be displayed to the user without significantly reducing the quality of the image, thereby improving the user experience.

[0076] Figure 5 is a schematic flow chart of a display method of a head mounted display device according to an embodiment of the present disclosure. Figures 1 to 5 , the display method 500 of the HMD device 100 may include step S510, step S520, step S530, step S540 and step S550.

[0077] In step S510, a tolerance enhancement operation TEO may be performed by the processor 110 on the original image IMG_O to generate an enhanced image IMG_E. In step S520, a software distortion may be applied by the processor 110 to the enhanced image IMG_E to generate a pre-deformed image IMG_P. In step S530, the display 120 may display the pre-deformed image IMG_P. In step S540, the pre-deformed image IMG_P may be received by the optical system 130. In step S550, an undistorted image may be output by the optical system 130.

[0078] In this way, a distortion-free image can be displayed to the user without significantly reducing the image quality, thereby improving the user experience.

[0079] In addition, the implementation details of the display method 500 can be referred to Figure 1 Sufficient teachings, suggestions and implementation examples can be obtained from the description of FIG. 4 , and the details will not be repeated here one by one.

[0080] In summary, according to the HMD device 100 and the display method 500, since the original image IMG_O is enhanced by the tolerance enhancement operation TEO, the quality of the image displayed to the user (i.e., the distortion-free image IMG_U) can be almost the same as the quality of the original image IMG_O. That is, the distortion-free image IMG_U can be displayed to the user without significantly reducing the quality of the image, thereby improving the user experience.

[0081] It is obvious to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In summary, the present disclosure is intended to cover modifications and variations within the scope of the following claims and their equivalents.

Claims

1. A head mounted display device, comprising: a display configured to display the pre-deformed image; an optical system configured to receive the pre-deformed image and output an undistorted image; as well as The processor is configured to: Performing a tolerance enhancement operation on the original image to generate an enhanced image; as well as A software distortion is applied to the enhanced image to produce the pre-deformed image, wherein the software distortion is configured to compensate for optical distortion of the optical system.

2. The head mounted display device of claim 1, wherein the optical system comprises a lens, and the optical distortion is at least partially caused by the lens. 3 . The head mounted display device of claim 1 , wherein the software distortion is barrel distortion and the optical distortion is pincushion distortion. 4 . The head mounted display device according to claim 1 , wherein the tolerance enhancement operation comprises a sharpening operation performed on an edge of a peripheral portion of the original image. 5 . The head mounted display device according to claim 1 , wherein the tolerance enhancement operation comprises a blurring operation performed on a texture of a peripheral portion of the original image. 6 . The head mounted display device according to claim 1 , wherein the tolerance enhancement operation comprises a sharpening operation performed on an edge of a peripheral portion of the original image and a blurring operation performed on a texture of a peripheral portion of the original image.

7. The head mounted display device according to claim 1, further comprising: A memory configured to store a tolerance enhancement model, wherein The processor is further configured to perform the tolerance enhancement operation based on the tolerance enhancement model.

8. The head mounted display device of claim 7, wherein the tolerance enhancement model is pre-trained using a machine learning algorithm based on a comparison of an ideal image of the original image and a simulated image of the enhanced image.

9. The head mounted display device of claim 8, wherein the ideal image is an output of a low pass filter and the original image is an input of the low pass filter.

10. The head mounted display device of claim 8, wherein the simulated image is a simulated result of a process of the enhanced image passing through the optical system.

11. A display method for a head mounted display device, comprising: Performing a tolerance enhancement operation on the original image to generate an enhanced image; applying a software distortion to the enhanced image to produce a pre-deformed image, wherein the software distortion is configured to compensate for optical distortion of an optical system; displaying the pre-deformed image; receiving the pre-deformed image through the optical system; as well as A distortion-free image is output through the optical system.

Images