Apparatus and method for improving visualization quality of XR device

Abstract

An apparatus and a method for improving the visualization quality of an XR device are provided. In the method for improving visualization quality according to an embodiment of the present invention, a rendered image is input into a first neural network to generate a modified image, the modified image is transferred through an optical system, and the transferred image is input into a second neural network to generate a modified image. Accordingly, when the rendered image is provided to a user of the XR device, image distortion caused by the optical system is minimized and, as a result, a high-quality image is provided, so that strangeness felt by the user in a state where real content and virtual content are mixed can be minimized.

Description

Device and method for improving the visualization quality of XR devices

[0001] The present invention relates to improving image quality, and more particularly, to a method for providing high-quality content to a user by minimizing distortion caused by a lens of an optical system when providing content rendered on an XR device to a user by scanning the optical system.

[0002] In order to minimize image distortion that occurs when content images pass through the lenses of an optical system in existing XR devices, a method is used to compensate for the distortion that occurs when passing through the lenses by adding an opposite distortion to the content to compensate for this distortion and scanning it through the optical system.

[0003] However, even for XR devices of the same model, the distortion aberration that occurs is different, so there are limitations in providing users with optimal image quality in the above manner.

[0004] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide an XR device visualization quality improvement device and method capable of providing high-quality images by minimizing image distortion caused by an optical system when providing rendered images to an XR device user.

[0005] A method for improving visualization quality according to one embodiment of the present invention for achieving the above object includes a first generation step of inputting a rendered image into a first neural network to generate a deformed image; a step of transmitting the deformed image by passing it through an optical system; and a second generation step of inputting the transmitted image into a second neural network to generate a deformed image.

[0006] In the transmission stage, the image passing through the optical system may be transformed into a distorted image by the lens of the optical system.

[0007] A distorted image may be an image with barrel distortion added by the lens of the optical system.

[0008] The first neural network and the second neural network can be trained simultaneously through end-to-end learning.

[0009] The first neural network and the second neural network can be trained in a direction that reduces the loss between the rendered image input to the first neural network and the transformed image output from the second neural network.

[0010] The loss function can be a loss function generated by weighting PSNR (Peak Signal-to-noise ratio), SSIM (Structural Similarity Index Measure), or PSNR and SSIM.

[0011] The rendered image may be a chess board image or a structured light pattern image.

[0012] A method for improving visualization quality according to the present invention may include a step of displaying a transformed image generated from a second neural network.

[0013] The rendered image may be an image to be displayed through an XR device.

[0014] According to another aspect of the present invention, a visualization quality improvement device is provided, characterized by including: a first neural network that receives a rendered image as input and generates a transformed image; and a second neural network that receives an image transformed by the first neural network and then transmitted through an optical system and generates a transformed image.

[0015] According to another aspect of the present invention, an image display method is provided, comprising: a step of rendering an image; a first generation step of inputting the rendered image into a first neural network to generate a deformed image; a step of transmitting the deformed image by passing it through an optical system; a second generation step of inputting the transmitted image into a second neural network to generate a deformed image; and a step of displaying the deformed image generated in the second generation step.

[0016] According to another aspect of the present invention, an image display device is provided, comprising: a rendering unit that renders an image; a first neural network that receives the rendered image as input and generates a transformed image; a second neural network that receives an image transformed by the first neural network and then transmitted through an optical system and generates a transformed image; and a display unit that displays the transformed image generated by the second neural network.

[0017] As described above, according to embodiments of the present invention, when providing a rendered image to an XR device user, image distortion caused by an optical system is minimized to provide a high-quality image, thereby minimizing the sense of incongruity felt by the user in a state where real content and virtual content are mixed.

[0018] Figure 1 is a drawing showing barrel distortion.

[0019] Figure 2 is a drawing showing pincushion distortion.

[0020] Figure 3 is a device for improving the visualization quality of an XR device according to one embodiment of the present invention.

[0021] FIG. 4 is a diagram illustrating the configuration of an XR device according to another embodiment of the present invention.

[0022] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0023] XR devices suffer from image distortion during the process of scanning content screens through the lenses of the optical system to render content and output it on the XR screen. Figure 1 illustrates distortion in an XR device.

[0024] As shown, when an input image passes through the optical lens of an XR device, barrel distortion occurs, as shown in the output image. This causes some pixels to be mapped to a single pixel or a single pixel to be mapped to multiple locations due to grid shifts in the output image, resulting in deterioration of the image quality of the content.

[0025] To address these issues, it is conceivable to provide rendered content to users through a process similar to that shown in Figure 2. This involves artificially inducing distortion before scanning the rendered content through an optical system, thereby compensating for the distortion caused by the lens.

[0026] Specifically, pincushion distortion is applied to the rendered image to create a distorted input image, which is then passed through the lens of the optical system of the XR device to provide the output image to the user.

[0027] However, even for XR devices of the same model, the barrel distortion distortion aberration usually varies depending on the mass-produced lens, so the optimal distortion aberration must be found after the XR device is produced.

[0028] Accordingly, an embodiment of the present invention proposes a device and method for improving the visualization quality of an XR device. This technology provides users with high-quality, distortion-free images by using a method for correcting distortion through a quality comparison between a rendered image and an image provided to the user, rather than a method for mathematically correcting distortion caused by a lens constituting the optical system of the XR device.

[0029] FIG. 3 is a diagram illustrating a configuration of an XR device visualization quality improvement device according to one embodiment of the present invention. As illustrated, the XR device visualization quality improvement device according to one embodiment of the present invention is configured to include neural network-1 (110) and neural network-2 (120).

[0030] Neural network-1 (110) refers to a neural network that receives a rendered image (I) to be provided through an XR device and generates a transformed image (I') or a processor for executing the same.

[0031] The image (I') transformed by the neural network-1 (110) is transformed into a distorted image (I'') with barrel distortion added by the lens (OL) of the optical system in the process of being transmitted through the lens (OL) of the optical system of the XR device.

[0032] Neural network-2 (120) refers to a neural network that receives a distorted image (I'') transmitted through a lens (OL) of an optical system, transforms it, and generates a distorted image (I'''), or a processor for executing the same. The distorted image (I''') output from neural network-2 (120) is displayed to the user, i.e., the image the user sees.

[0033] Meanwhile, neural network-1 (110) and neural network-2 (120) are trained together simultaneously through end-to-end learning. Specifically, neural network-1 (110) and neural network-2 (120) are trained in a direction that reduces the loss (difference) between the transformed image (I'''), which is the output image of neural network-2 (120), and the rendered image (I), which is the input image of neural network-1 (110).

[0034] The loss function can use PSNR (Peak Signal-to-noise ratio) or SSIM (Structural Similarity Index Measure), or a loss function generated by weighting PSNR and SSIM can be used.

[0035] Additionally, general content can be used as an input image for the neural network-1 (110), but pattern images such as chess board images or structured light pattern images can also be used.

[0036] Meanwhile, an image display means must be provided between the neural network-1 (110) and the optical system, and an image sensor must be provided between the optical system and the neural network-2 (120). However, in order to enhance understanding of the concept of the present invention, the illustration of the corresponding configuration is omitted in FIG. 3.

[0037] FIG. 4 is a diagram illustrating the configuration of an XR device according to another embodiment of the present invention. As illustrated, the XR device according to the embodiment of the present invention is configured to include a rendering unit (210), a visualization quality enhancement unit (220), a display unit (230), a communication unit (240), a control unit (250), and an operation unit (260).

[0038] The rendering unit (210) renders an XR image to be provided to the user. The visualization quality improvement unit (220) is a processor and memory for executing the neural networks (110, 120) that constitute the visualization quality improvement device presented through the aforementioned FIG. 3.

[0039] The display unit (230) is configured to display an image rendered by the rendering unit (210) and having barrel distortion removed by the visualization quality improvement unit (220) and provide it to the user.

[0040] The communication unit (240) is a means for communicating with external devices and external networks, and receives XR images or external control commands. The control unit (250) controls the overall operation of the XR device according to external commands input through the communication unit (240) or user commands input through the operation unit (260).

[0041] So far, a device and method for improving the visualization quality of an XR device and an XR device applying the same have been described in detail with preferred embodiments.

[0042] In the above embodiment, when providing a rendered image to an XR device user, the image distortion caused by the optical system is minimized to provide a high-quality image, thereby minimizing the sense of incongruity felt by the user in a state where real content and virtual content are mixed.

[0043] Meanwhile, it goes without saying that the technical idea of ​​the present invention can also be applied to a computer-readable recording medium containing a computer program that performs the functions of the device and method according to the present embodiment. In addition, the technical idea according to various embodiments of the present invention can be implemented in the form of computer-readable code recorded on a computer-readable recording medium. The computer-readable recording medium can be any data storage device that can be read by a computer and store data. For example, the computer-readable recording medium can be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, etc. In addition, the computer-readable code or program stored on the computer-readable recording medium can be transmitted through a network connected between computers.

[0044] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. A first generation step of inputting the rendered image into the first neural network to generate a transformed image; A step of transmitting a transformed image through an optical system; A method for improving visualization quality, characterized by including a second generation step of generating a transformed image by inputting the transmitted image into a second neural network.

2. In claim 1, The image passing through the optical system in the transmission stage is A method for improving visualization quality, characterized in that the image is transformed into a distorted image by a lens of an optical system.

3. In claim 2, The distorted image is, A method for improving visualization quality, characterized in that the image has barrel distortion added by a lens of an optical system.

4. In claim 1, The first neural network and the second neural network, A method for improving visualization quality, characterized by simultaneously learning through end-to-end learning.

5. In claim 4, The first neural network and the second neural network, A method for improving visualization quality, characterized in that learning is performed in a direction in which loss between a rendered image input to a first neural network and a transformed image output from a second neural network is reduced.

6. In claim 5, The loss function is, A method for improving visualization quality, characterized by a loss function generated by weighting PSNR (Peak Signal-to-noise ratio), SSIM (Structural Similarity Index Measure), or PSNR and SSIM.

7. In claim 1, The rendered image is, A method for improving visualization quality, characterized in that the image is a chess board image or a structured light pattern image.

8. In claim 1, A method for improving visualization quality, characterized by comprising: a step of displaying a transformed image generated from a second neural network.

9. In claim 1, The rendered image is, A method for improving visualization quality, characterized in that the image is to be displayed through an XR device.

10. A first neural network that receives a rendered image as input and generates a transformed image; A visualization quality improvement device, characterized by including a second neural network that receives an image transformed by a first neural network and passes it through an optical system to generate a transformed image.

11. Step of rendering the image; A first generation step for generating a transformed image by inputting the rendered image into the first neural network; A step of transmitting a transformed image through an optical system; A second generation step for generating a transformed image by inputting the transmitted image into a second neural network; An image display method, characterized by including a step of displaying a transformed image generated in a second generation step.

12. Rendering unit that renders images; A first neural network that takes a rendered image as input and generates a transformed image; A second neural network that receives an image transformed by the first neural network and passes it through an optical system and generates a transformed image; An image display device characterized by including a display unit that displays a transformed image generated by a second neural network.

Images