IMAGE GENERATION DEVICE, PROGRAM, IMAGE GENERATION METHOD, AND IMAGE DISPLAY SYSTEM
The new mirroring system for head-mounted displays corrects distortion within the rendering device to maintain image quality, providing high-quality mirrored images on both head-mounted and flat displays.
Patent Information
- Application Number
- JP2021037685
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-09
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2041-03-09
AI Technical Summary
Existing mirroring technologies for head-mounted displays result in deteriorated mirrored images due to the need for inverse distortion correction, which degrades image quality.
A new mirroring system that generates undistorted source images and applies distortion correction within the rendering device to ensure high-quality mirrored images are displayed on both head-mounted and flat displays.
The system provides high-quality mirrored images by correcting distortion before transmission, ensuring clear and immersive viewing experiences across both display types.
Smart Images

Figure 0007594945000001 
Figure 0007594945000002 
Figure 0007594945000003
Abstract
Description
[Technical field]
[0001] The present invention relates to image generation technology. [Background technology]
[0002] A head-mounted display connected to a game console is worn on the head, and a game is played by operating a controller while looking at the screen displayed on the head-mounted display. When a head-mounted display is worn, the user sees nothing but the image displayed on the head-mounted display, which increases the sense of immersion in the image world and further enhances the entertainment value of the game. In addition, if a virtual reality image is displayed on the head-mounted display and a 360-degree virtual space is displayed all around the user when the user wearing the head-mounted display rotates his or her head, the sense of immersion in the image is further increased and the operability of applications such as games is also improved.
[0003] Meanwhile, in recent years, mirroring has been proposed for sharing the screen displayed by a head mounted display with another display device. Mirroring is a technology for sharing the displayed screen between a transmitting device and a receiving device by transmitting image data as information of the screen displayed by the transmitting device to the receiving device via a network. Summary of the Invention [Problem to be solved by the invention]
[0004] A new system for providing images using a head mounted display is being developed, and therefore a mirroring technique for providing a high quality mirrored image in this new system is required.
[0005] In view of the above, an object of the present invention is to provide a mirroring technique for providing a high-quality mirrored image in a new system for providing an image using a head-mounted display. [Means for solving the problem]
[0006] In order to solve the above problem, an image generating device of one embodiment of the present invention is characterized in that it comprises a source image generating unit that generates an undistorted source image, an HMD image generating unit that generates an HMD image to be displayed on a head-mounted display based on the source image, and a mirroring image generating unit that generates a mirroring image for mirroring the HMD image on a flat-panel display based on the source image.
[0007] A program according to one embodiment of the present invention is characterized in that it causes a processor to function as a source image generation unit that generates an undistorted source image, an HMD image generation unit that generates an HMD image to be displayed on a head-mounted display based on the source image, and a mirroring image generation unit that generates a mirroring image for mirroring the HMD image on a flat-panel display based on the source image.
[0008] An image generation method according to one embodiment of the present invention is characterized in that it includes the steps of generating an undistorted source image, and generating, based on the source image, an HMD image to be displayed on a head-mounted display and a mirroring image for mirroring the HMD image on a flat-panel display.
[0009] An image display system of one embodiment of the present invention is characterized in that it comprises an image generating device including a source image generation unit that generates an undistorted source image, an HMD image generation unit that generates an HMD image to be displayed on a head-mounted display based on the source image, a mirroring image generation unit that generates a mirroring image for mirroring the HMD image on a flat-panel display based on the source image, an HMD image supply unit that supplies the HMD image to the head-mounted display, and a mirroring image supply unit that supplies the mirroring image to the flat-panel display, the head-mounted display that displays the supplied HMD image, and the flat-panel display that displays the supplied mirroring image.
[0010] In addition, any combination of the above, or mutual substitution of the components or expressions of the present invention among methods, devices, programs, temporary or non-temporary storage media recording programs, systems, etc., are also valid aspects of the present invention. Effect of the Invention
[0011] According to the present invention, it is possible to provide a mirroring technique for providing a high-quality mirrored image in a new system for providing an image using a head-mounted display. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is an external view of a head mounted display. [Diagram 2] FIG. 1 is a configuration diagram of an image generating system. [Diagram 3] FIG. 1 is a diagram for explaining a mirroring system according to an embodiment of the present invention. [Figure 4] FIG. 3 is a configuration diagram of the rendering device in FIG. 2. [Diagram 5] 5(a) and 5(b) are examples of a head-mounted display image and a mirrored image in 3D display, respectively. [Figure 6] 6(a) and (b) are examples of a head-mounted display image and a mirrored image in two-dimensional display, respectively. [Figure 7] 4 is a flowchart showing a process flow in the rendering device. [Figure 8] FIG. 1 is a diagram for explaining a mirroring system of a comparative example. [Figure 9] FIG. 13 is a diagram for explaining a mirroring system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] (Embodiment) 1 is an external view of a head mounted display 100. The head mounted display 100 is a display device that is worn on a user's head to view still images and videos displayed on the display and to listen to audio and music outputted from headphones.
[0014] A gyro sensor or an acceleration sensor built into or attached externally to the head mounted display 100 can measure position information of the head of a user wearing the head mounted display 100 and orientation information such as the rotation angle and inclination of the head.
[0015] A camera for photographing the user's eyes may be further provided in the head mounted display 100. The camera mounted in the head mounted display 100 can detect the user's gaze direction, pupil movement, blinking, and the like.
[0016] The head mounted display 100 is an example of a "wearable display." Here, a method for generating an image displayed on the head mounted display 100 will be described. Note that the image generating method of this embodiment is not limited to the head mounted display 100 in the narrow sense, but can also be applied to cases where glasses, glasses-type displays, glasses-type cameras, headphones, headsets (headphones with microphones), earphones, earrings, ear-hanging cameras, hats, hats with cameras, hair bands, and the like are worn.
[0017] 2 is a configuration diagram of an image transfer system according to this embodiment. The head mounted display 100 is connected to a rendering device 200 via an interface such as HDMI (registered trademark) (High-Definition Multimedia Interface), which is a standard for a communication interface that transmits video and audio as digital signals, or DisplayPort, which is a standard for a video output interface. A flat display 302 is connected to the rendering device 200. The rendering device 200 of this embodiment is an example of an image generating device.
[0018] In this embodiment, the data transmission path 300 between the head mounted display 100 and the rendering device 200 is an HDMI transmission path or a DisplayPort transmission path. In the HDMI standard or the DisplayPort standard, a secondary data packet can be transmitted in association with an image frame, and metadata related to the frame can be included in the secondary data packet. The HDMI2.1 standard has a function called dynamic HDR (High Dynamic Range), which can generate an image in which the luminance and color depth are optimally adjusted for each frame according to the scene by referring to dynamic metadata of the image. In the HDMI2.1 standard, the dynamic metadata can transmit information required for dynamic HDR, such as the maximum luminance, average luminance, and minimum luminance of the scene, in synchronization with the image. The communication interface between the head mounted display 100 and the rendering device 200 is not limited to HDMI or DisplayPort, as long as it can transmit metadata in synchronization with the image.
[0019] The rendering device 200 of this embodiment is a game machine. The rendering device 200 may be further connected to a server via a network. In this case, the server may provide the rendering device 200 with an online application such as a game in which multiple users can participate via the network.
[0020] The rendering device 200 basically processes the program of the content, generates a display image, and transmits it to the head mounted display 100 or the flat display 302. The program and data of the content are read by a media drive (not shown) from a ROM medium (not shown) on which application software of the content, such as a game, and license information are recorded. This ROM medium is a read-only recording medium such as an optical disk, a magneto-optical disk, or a Blu-ray disk. In one aspect, the rendering device 200 identifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head mounted display 100, and generates a display image of the content at a predetermined rate so as to provide a field of view corresponding to the identified position and direction.
[0021] The head mounted display 100 receives the data of the display image and displays it as an image of the content. The image displayed on the head mounted display 100 may be an image captured in advance by a camera, an image based on computer graphics such as a game image, or a live image of a remote location distributed via a network. The image displayed on the head mounted display 100 may be a VR image, an AR (augmented reality) image, an MR image, or the like.
[0022] The flat display 302 may be a television having a display for outputting images and a speaker for outputting audio, and specifically may be any of a liquid crystal television, an organic electroluminescence television, a plasma television, a PC display, etc. Alternatively, the flat display 302 may be the display and speaker of a tablet terminal or a mobile terminal.
[0023] Here, an assumed method of mirroring technology for sharing an image displayed on the head mounted display 100 by displaying it on the flat display 302 will be described using a comparative example of FIG. 9. The mirroring system of FIG. 9 includes a rendering device 400, a head mounted display 100, a flat display 302, and an output control device 500. The output control device 500 is provided separately from the rendering device 400 and connected to the rendering device 400. The output control device 500 processes data output by the rendering device 400. The output control device 500 relays the processed data to the head mounted display 100 and the flat display 302 via an output port 500a for the head mounted display and an output port 500b for the mirrored image, respectively.
[0024] In the mirroring system of FIG. 9, the rendering device 400 first generates a left-eye image L and a right-eye image R as source images related to the content. The rendering device 400 generates a binocular image C for the head mounted display 100 by performing distortion correction on the left-eye image L and the right-eye image R in accordance with the distortion caused by the eyepiece lens of the head mounted display 100. The binocular image C is intended for stereoscopic viewing of an image, and has a configuration in which a pair of parallax images consisting of a left-eye image L and a right-eye image R are arranged in the left and right regions obtained by dividing an image plane corresponding to a display panel into two. Here, the head mounted display 100 employs an optical lens with a high curvature in order to display an image with a wide viewing angle in front of and around the user's eyes, and is configured so that the user looks into the display panel through the lens. If a lens with a high curvature is used, the image will be distorted due to the distortion aberration of the lens. Therefore, in order to make the image appear correctly when viewed through a lens with a high curvature, the rendered image is subjected to a distortion process in advance, and the image after the distortion process is transmitted to the head mounted display and displayed on the display panel so that the image appears correctly when the user views the image through a lens with a high curvature. The rendering device 400 supplies this binocular image C to the output control device 500.
[0025] The output control device 500 supplies the binocular image C as an image for head mounted display (hereinafter, HMD image) C to the head mounted display 100 as it is via the output port for HMD image 500a. The output control device 500 also applies the inverse correction to the distortion correction to either the left eye portion or the right eye portion of the binocular image C, and generates a mirrored image M for the flat display 302 by cutting out the center portion of the image that has been subjected to the correction. The output control device 500 supplies the mirrored image M to the flat display 302 via the output port for mirrored image 500b. In this way, the HMD image C is displayed on the head mounted display 100, and the mirrored image M is displayed on the flat display 302, and mirroring is realized in the mirroring system of FIG. 9. However, the mirroring system of FIG. 9 has a problem in that the mirrored image M is deteriorated because it is necessary to apply the inverse correction to the distortion correction.
[0026] Meanwhile, a new mirroring system shown in Fig. 3 has been developed. The mirroring system in Fig. 3 includes a rendering device 200, a head mounted display 100, and a flat display 302. In the system in Fig. 3, an output control device 500 is not provided between the rendering device 400 and the head mounted display 100 and flat display 302 as in the system in Fig. 9. In the system in Fig. 3, the functions of generating a source image, performing necessary processing on the source image to generate an HMD image and a mirrored image, and supplying them to the head mounted display 100 and the flat display 302, respectively, are realized within the same device.
[0027] An overview of the mirroring technique in the system of Fig. 3 will be described. The rendering device 200 generates a left-eye image L and a right-eye image R as source images related to the content. Based on the source images, the rendering device 200 performs distortion correction in accordance with distortion caused by the eyepiece lens of the head mounted display 100, reprojection processing based on the latest position information and attitude information of the head mounted display 100, and the like on the left-eye image L and the right-eye image R to generate a binocular image C. The rendering device 200 supplies the generated binocular image C to the head mounted display 100 as an HMD image via an HMD image output port 200a.
[0028] Furthermore, the rendering device 200 generates a mirrored image M by performing a process described below on one of the left eye image L and the right eye image R (the right eye image R in this embodiment) based on the source image. The rendering device 200 supplies the generated mirrored image M to the flat display 302 via the mirrored image output port 200b. In this manner, the HMD image C is displayed on the head mounted display 100, and the mirrored image M is displayed on the flat display 302, thereby realizing mirroring in the mirroring system of FIG. 3.
[0029] According to this embodiment, the mirrored image M is generated by performing distortion correction on the source image, and since the inverse correction to the distortion correction is not performed as in the mirroring system of FIG. 9, it is possible to suppress deterioration of the mirrored image M. As a result, it is possible to provide a high-quality mirrored image. The mirroring method of this embodiment will be described in detail below.
[0030] Fig. 4 is a configuration diagram of the rendering device 200 in Fig. 2. The rendering device 200 includes an input data acquisition unit 201, an information processing unit 202, a source image generation unit 203, a source image processing unit 204, first and second communication units 205a and 205b, and a data storage unit 206 that stores data required for generating an image. Fig. 4 is a block diagram focusing on functions, and these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof.
[0031] The first and second communication units 205a and 205b establish communication by a predetermined protocol such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus). The first communication unit 205a is configured to be able to communicate with the head mounted display 100 via the HMD image output port 200a. The first communication unit 205a receives position information and attitude information of the head mounted display 100 from the head mounted display 100. The first communication unit 205a is configured to be able to supply an HMD image C to the head mounted display 100. The first communication unit 205a of this embodiment is an example of an HMD image supply unit. The second communication unit 205b is configured to be able to communicate with the flat display 302 via the mirroring image output port 500b. The second communication unit 205b is configured to be able to supply a mirroring image M to the flat display 302. The second communication unit 205b of this embodiment is an example of a mirroring image supply unit.
[0032] The input data acquisition unit 201 acquires input data. The input data includes operation information indicating the contents of a user operation inputted via an input device (not shown), position information and posture information of the user's head acquired via the first communication unit 205a, and the like. Here, the user operation may be an operation performed in general information processing, such as selection of an application to be executed, start / end of processing, command input, and the like. The input data acquisition unit 201 supplies the acquired input data to the information processing unit 202.
[0033] The information processing unit 202 performs information processing according to an application to be executed, such as a game. For example, the information processing unit 202 progresses the game or performs corresponding processing based on input data. The information processing unit 202 supplies the result of the information processing to the source image generation unit 203.
[0034] Based on the result of the information processing, the source image generation unit 203 reads necessary data from the data storage unit 206 to generate a source image such as a game image to be output. For example, the source image generation unit 203 generates, as source images, a left eye image L and a right eye image R related to the virtual world seen from a viewpoint corresponding to the position and posture of the user's head. In this way, the source image generation unit 203 generates a source image without distortion. The source image generation unit 203 supplies the source image to the image processing unit 204.
[0035] The image processing unit 204 processes the source image as necessary to generate an HMD image C and a mirroring image M. The source image processing unit 204 includes an HMD image generation unit 204a that generates the HMD image C, and a mirroring image generation unit 204b that generates the mirroring image M.
[0036] The HMD image generating unit 204a renders the left eye image L and the right eye image R as source images as left and right parallax images, and then performs necessary processing to generate a binocular image C arranged on the left and right. This necessary processing includes distortion correction taking the lens into consideration, reprojection processing based on the latest position information and attitude information of the head mounted display 100, superimposition processing of a menu screen, etc., and each of these can be performed by a general method. The HMD image generating unit 204a supplies the generated binocular image C to the head mounted display 100 as an HMD image C via the first communication unit 205a.
[0037] The mirroring image generating unit 204b generates a mirroring image M by using a portion (e.g., a central portion) of one of the left eye image L and the right eye image R (the right eye image R in this embodiment) as a source image. The mirroring image generating unit 204b supplies the mirroring image M to the flat display 302 via the second communication unit 205b. A method of generating a mirroring image in the mirroring image generating unit 204b will be described below.
[0038] First, a case where a mirrored image M in a three-dimensional display mode is generated will be described. The mirrored image generating unit 204b obtains a viewing angle (UV coordinate value) of the head mounted display 100 with respect to the HMD image C in which the reprojection processing in the HMD image generating unit 204a is reflected. The UV coordinate value is a parameter coordinate for referring to a texture attribute. The mirrored image generating unit 204b generates a plurality of texture images corresponding to a part of the right-eye image R by referring to a texture corresponding to the viewing angle using the obtained UV coordinate value. The mirrored image generating unit 204b generates a mirrored image M in a three-dimensional display mode by superimposing the plurality of texture images. In this way, by using the same viewing angle as the viewing angle in the binocular image C after processing in the HMD image generating unit 204a, it is possible to generate a mirrored image M so as to match the viewing angle of the HMD image C after the reprojection processing. Therefore, it is possible to provide a high-quality mirrored image M in a three-dimensional display mode in which the influence of tracking of the head mounted display 100 is suppressed. In addition, the texture can be selected by adjusting the display width of the flat display 302 using the viewing angle of the head mounted display 100 in the processed binocular image C. Furthermore, it is possible to make the image quality dependent on the texture of the source image.
[0039] Next, a case where a mirrored image M in a two-dimensional display mode is generated will be described. The mirrored image generating unit 204b generates a mirrored image M in a two-dimensional display mode by attaching a part of the right-eye image R as a texture to a 16:9 panel (virtual screen) corresponding to the screen size of the flat display 302. This allows a part of the right-eye image R to be displayed as is, regardless of the tracking (head movement) of the head mounted display 100. Therefore, it is possible to provide a high-quality mirrored image M in a two-dimensional display mode in which the influence of the tracking of the head mounted display 100 is suppressed. Here, the mirrored image M in a two-dimensional display mode is used when projecting a two-dimensional plane image in a virtual space, and for example, the mirrored image M in a two-dimensional display mode is used when displaying a virtual movie screen in a virtual space.
[0040] In this way, the HMD image and the mirroring image are supplied to the head mounted display 100 and the flat display 302, respectively, and these images are displayed on the head mounted display 100 and the flat display 302. Figures 5(a) and (b) are examples of an HMD image C and a mirroring image M in a three-dimensional display. Figures 6(a) and (b) are examples of an HMD image C and a mirroring image M in a two-dimensional display.
[0041] Hereinafter, with reference to Fig. 7, a process S10 for displaying the HMD image C and the mirroring image M of this embodiment on the head mounted display 100 and the flat display 302, respectively, will be described. In step S11, the source image generating unit 203 generates an undistorted source image. In step S12, the image processing unit 204 generates the HMD image C and the mirroring image M based on the source image. In step S13, the first communication unit 205a supplies the HMD image C to the head mounted display 100, and the second communication unit 205b supplies the mirroring image M to the flat display 302. After S13, the process S10 ends.
[0042] (Modification) Modifications of the embodiment will be described below.
[0043] In the embodiment, the mirroring image M is generated using a part of the right-eye image R, but is not limited thereto. For example, the mirroring image M may be equivalent to the binocular image C generated by the HMD image generating unit 204a (see FIG. 8). In this case, the mirroring image generating unit 204b may generate the binocular image C by performing the same processing as the HMD image generating unit 204a, and supply this to the flat display 302 as the mirroring image M. With this configuration, the HMD image C (binocular image C) supplied to the head mounted display 100 can be confirmed on the flat display 302, making it easier for developers to debug.
[0044] In this embodiment, the rendering device 200 is a game machine, but is not limited to this. The rendering device 200 may be a server or the like further connected to the head mounted display 100 and the flat display 302 via a network.
[0045] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing step, and that such modifications are also within the scope of the present invention. Such modifications will be described. [Explanation of symbols]
[0046] 100 Head mounted display, 200 Rendering device, 200a Output port for HMD image, 200b Output port for mirroring image, 200b, 201 Input data acquisition unit, 202 Information processing unit, 203 Source image generation unit, 204 Source image processing unit, 204a HMD image generation unit, 204b Mirroring image generation unit, 205 Communication unit, 206 Data storage unit, 300 Data transmission path, 302 Flat panel display.
Claims
1. a source image generator for generating an undistorted source image; an HMD image generating unit that generates an HMD image to be displayed on a head mounted display based on the source image; a mirroring image generating unit that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image; Equipped with the source image includes a right-eye image and a left-eye image of the head-mounted display; the mirroring image generation unit generates the mirroring image using a portion of one of the right-eye image and the left-eye image; the HMD image generating unit generates the HMD image by performing a reprojection process on a portion of the image, The image generating device is characterized in that the mirroring image generation unit generates the mirroring image in a three-dimensional display mode by referring to a texture corresponding to the viewing angle of the head mounted display for the HMD image in which the reprojection processing is reflected.
2. The image generating device according to claim 1 , wherein the mirrored image generating unit generates the mirrored image in a two-dimensional display mode by attaching the portion of one of the images to a panel corresponding to a screen size of the flat display.
3. The processor, a source image generator for generating an undistorted source image; an HMD image generating unit that generates an HMD image to be displayed on a head mounted display based on the source image; a mirroring image generating unit that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image; Function as a the source image includes a right-eye image and a left-eye image of the head-mounted display; the mirroring image generation unit generates the mirroring image using a portion of one of the right-eye image and the left-eye image; the HMD image generating unit generates the HMD image by performing a reprojection process on a portion of the image, The program is characterized in that the mirroring image generation unit generates the mirroring image in a three-dimensional display mode by referring to a texture corresponding to the viewing angle of the head mounted display for the HMD image in which the reprojection processing is reflected.
4. generating an undistorted source image; generating an HMD image to be displayed on a head mounted display and a mirroring image for mirroring the HMD image on a flat panel display based on the source image; Including, the source image includes a right-eye image and a left-eye image of the head-mounted display; the generating step includes a step of generating the mirrored image using a portion of one of the right-eye image and the left-eye image; The generating step includes a step of generating the HMD image by performing a reprojection process on a portion of the image, The image generating method is characterized in that the generating step includes a step of generating the mirrored image in a three-dimensional display manner by referring to a texture corresponding to the viewing angle of the head mounted display for the HMD image in which the reprojection processing is reflected.
5. a source image generator for generating an undistorted source image; an HMD image generating unit that generates an HMD image to be displayed on a head mounted display based on the source image; a mirroring image generating unit that generates a mirroring image for mirroring the HMD image on a flat panel display based on the source image; an HMD image supply unit that supplies the HMD image to the head mounted display; a mirroring image supply unit that supplies the mirroring image to the flat panel display; An image generating device comprising: the head mounted display for displaying the supplied HMD image; the flat panel display for displaying the supplied mirrored image; Equipped with the source image includes a right-eye image and a left-eye image of the head-mounted display; the mirroring image generation unit generates the mirroring image using a portion of one of the right-eye image and the left-eye image; the HMD image generating unit generates the HMD image by performing a reprojection process on a portion of the image, The image display system is characterized in that the mirroring image generation unit generates the mirroring image in a three-dimensional display mode by referring to a texture corresponding to the viewing angle of the head-mounted display for the HMD image in which the reprojection processing is reflected.
Citation Information
Patent Citations
Display apparatus, video display system, and video display method
JP2010057069A
Image correction device, image correction method, and program
JP2019106628A
Image display apparatus and image display method
JP2020052143A
Image display device, image display system, and image display method
JP2021015156A
Video signal processing device and video signal processing method
WO2011045872A1
