Image processing apparatus, image processing method, and program

Abstract

It is not possible to correct the difference in sharpness between the left and right images caused by the deviation of the HMD mounting position.SOLUTION: According to an aspect of the present invention, there is provided an image processing apparatus including a position information acquisition unit configured to acquire position information of a left eye and a right eye of a wearer wearing a head-mounted display, and a correction unit configured to correct sharpness of a left-side image projected to the left eye of the wearer and a right-side image projected to the right eye of the wearer in the head-mounted display based on the position information of the left eye and the right eye of the wearer.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for correcting the sharpness of left and right images in a head-mounted display. [Background technology]

[0002] Head-mounted displays (HMDs) are commercially available, allowing viewers to enjoy immersive video by wearing them on their heads. There are several types of HMDs, but video see-through HMDs are equipped with a display and two eyepieces, one for each eye, and viewers view the video displayed on the display through the eyepieces. When using an HMD, viewers adjust the length of the HMD's head strap and the distance between the left and right eyepieces so that their eyes are positioned appropriately in relation to the eyepieces.

[0003] However, because these adjustments are performed manually by the viewer, some viewers may not be able to adjust them to the appropriate position. If the viewer's eyes are not positioned properly relative to the eyepieces, the viewer will be unable to view the images properly, for example, the images will appear blurred. In other words, a misalignment between the eyes and the appropriate position of the eyepieces when wearing the HMD may result in different image sharpness between the left and right eyes. It is also known that viewing different images with the left and right eyes can cause "visual sickness," which is similar to motion sickness and can cause headaches, discomfort, nausea, and other symptoms.

[0004] Patent Document 1 discloses a technology for correcting the sharpness of a left-eye image projected to the left eye and a right-eye image projected to the right eye, respectively, according to the visual characteristics of the left and right eyes. The technology in Patent Document 1 makes it possible to correct the sharpness of both the left-eye image and the right-eye image so that they each have appropriate sharpness. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-253471 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 does not take into consideration changes in sharpness depending on the position of the viewer's eyes when wearing the HMD, and therefore cannot correct differences in sharpness between left and right images caused by misalignment of the HMD wearing position. [Means for solving the problem]

[0007] The image processing device according to the present invention is characterized by having a position information acquisition means for acquiring position information of the left eye and right eye of a wearer wearing a head-mounted display, and a correction means for correcting the sharpness of a left-side image to be projected onto the left eye of the wearer and a right-side image to be projected onto the right eye of the wearer in the head-mounted display based on the position information of the left eye and right eye of the wearer, respectively. [Effects of the Invention]

[0008] The present invention makes it possible to correct the difference in sharpness between left and right images caused by misalignment of the HMD wearing position. [Brief explanation of the drawings]

[0009] [Figure 1] Schematic diagram showing an overview of the HMD. [Figure 2] FIG. 1 is a schematic diagram showing a hardware configuration of an image processing apparatus. [Figure 3] FIG. 1 is a block diagram showing the arrangement of an image processing apparatus according to a first embodiment. [Figure 4] 5A and 5B are diagrams for explaining acquisition of left and right eye position information in the first embodiment. [Figure 5] 5A to 5C are diagrams for explaining the amount of eye positional deviation when determining the amount of correction according to the first embodiment. [Figure 6]5A to 5C are diagrams for explaining sharpness correction processing in determining the correction amount according to the first embodiment. [Figure 7] 1 is a flowchart of processing according to the first embodiment. [Figure 8] FIG. 10 is a block diagram showing the arrangement of an image processing apparatus according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a user interface for acquiring visual information according to the second embodiment. [Figure 10] 10 is a flowchart showing processing in the second embodiment. [Figure 11] FIG. 10 is a block diagram showing the configuration of an image processing apparatus according to a third embodiment. [Figure 12] 11A and 11B are diagrams for explaining a change in the amount of parallax in the third embodiment. [Figure 13] 10 is a flowchart showing processing in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. Note that the same components will be described with the same reference numerals.

[0011] <Embodiment 1> In this embodiment, a process will be described in which the positions of the left and right eyes are detected when a head-mounted display (HMD) is worn, and the sharpness of the left and right images is corrected based on the detection results.

[0012] First, an overview of the HMD used in this embodiment will be described with reference to Fig. 1. The HMD 101 has a strap 102 for attaching the HMD to the viewer's head and a strap length adjustment unit 103. It also has a left eyepiece 104, a right eyepiece 105, an interpupillary distance adjustment dial 106, a left eye display 107, a right eye display 108, a motion sensor 109, and an eye camera 110. When using the HMD 101, the viewer attaches the HMD 101 to their head by adjusting the length of the strap 102 using the strap length adjustment unit 103.

[0013] The video input to the HMD 101 consists of a left-side video image projected onto a left-eye display 107 and a right-side video image projected onto a right-eye display 108. The viewer views the left-side video image with their left eye through a left-eye eyepiece 104, and the right-side video image with their right eye through a right-eye eyepiece 105. The left-eye display 107 and the right-eye display 108 may be independent left and right displays, or may be displayed separately on a single display.

[0014] To match the distance between the viewer's left and right eyes with the distance between the left eye eyepiece 104 and the right eye eyepiece 105 of the HMD 101, the distance between the left eye eyepiece 104 and the right eye eyepiece 105 is adjusted using an interpupillary distance adjustment dial 106. Furthermore, the viewer's posture is detected by a motion sensor 109, and information about the viewer's left and right eyes is detected in real time by an eye camera 110, which projects light onto each of the left and right corneas and captures the reflected light.

[0015] Next, an example of the configuration of the image processing device 201 in this embodiment is shown in Figure 2. The image processing device 201 is configured to be included in the HMD 101, detects the positions of the left and right eyes when the HMD 101 is worn, and changes the sharpness of the image based on the detected positions. Note that the image processing device 201 may not be included in the HMD 101 but may be in a separate housing, in which case the image processing device 201 and the HMD 101 are connected by wire or wirelessly.

[0016] 2, the image processing device 201 includes a central control unit (CPU) 202, a random access memory (RAM) 203, and a read-only memory (ROM) 204. It also includes a video card (VC) 205, a serial ATA (SATA) interface (I / F) 206, a general-purpose I / F 207, a network interface card (NIC) 208, and a system bus 209. The CPU 202 uses the RAM 203 as a work memory to execute an operating system (OS) and various programs stored in the ROM 204, a hard disk drive (HDD) 211, etc. The CPU 202 also controls each component via the system bus 209. Note that, in the processing of the flowcharts described below, program code stored in the ROM 204, the hard disk drive (HDD) 211, etc. is loaded into the RAM 203 and executed by the CPU 202. A display device 210, such as a display, is connected to the VC 205. An input device 213 such as a mouse or keyboard and an imaging device 214 are connected to the general-purpose I / F 207 via a system bus 209. A general-purpose drive 212 that reads from and writes to an HDD 211 and various recording media is connected to the SATA I / F 206 via the system bus 209. The NIC 208 inputs and outputs information to and from external devices. The CPU 202 uses the HDD 211 and various recording media mounted on the general-purpose drive 212 as storage locations for various data. The CPU 202 displays a graphical user interface (GUI) provided by a program on a display device 210 and receives inputs such as user instructions via an input device 213.

[0017] Next, the functional configuration of the image processing device 201 in this embodiment will be described with reference to Fig. 3. The image processing device 201 is made up of a left / right eye position information acquisition unit 301, a correction amount determination unit 302, and a correction unit 303.

[0018] The left and right eye position information acquisition unit 301 acquires position information of the left and right eyes of the viewer wearing the HMD 101 in response to instructions from the CPU 202. The position information of the viewer's left and right eyes can be acquired, for example, from an image captured by the eye camera 110 provided in the HMD 101. Fig. 4 is a schematic diagram for explaining the processing of the left and right eye position information acquisition unit 301, Fig. 4(a) is an example of an image captured by the eye camera 110, and Fig. 4(b) and Fig. 4(c) show examples of acquisition of position information of the viewer's left and right eyes, respectively.

[0019] First, in FIG. 4(a), an eye image 401 is composed of a left imaging area 402 corresponding to the left eye eyepiece 104 and a right imaging area 403 corresponding to the right eye eyepiece 105. When a viewer wears the HMD 101, the viewer's left eye is imaged in the left imaging area 402, and the viewer's right eye is imaged in the right imaging area 403. The left / right eye position information acquisition unit 301 detects the viewer's left eye and right eye from the left imaging area 402 and right imaging area 403, respectively, and acquires the eye position coordinates within each imaging area as position information. The left eye position 404 and right eye position 405 can detect, for example, the position of the pupil at the center of the cornea. FIG. 4(b) is an enlarged view of the left imaging area 402. If the image size of the left imaging region 402 is 1000 x 1000 pixels and the coordinates of the center of the imaging region are the origin (x, y) = (0, 0), then the coordinates of the left eye position 404 can be expressed as (x, y) = (-100, 100). Figure 4(c) is an enlarged view of the right imaging region 403, and similar to Figure 4(b), the position coordinates of the right eye position 405 are also obtained. For example, the coordinates of the right eye position 405 can be expressed as (x, y) = (-10, 0).

[0020] In response to instructions from CPU 202, correction amount determination unit 302 determines the amount of sharpness correction for the left and right images based on the positions of the viewer's left and right eyes acquired by left and right eye position information acquisition unit 301. Specifically, the amount of correction to correct the sharpness of the left image to be displayed on left-eye display 107 is determined based on left eye position 404 acquired by left and right eye position information acquisition unit 301. Similarly, the amount of correction to correct the sharpness of the right image to be displayed on right-eye display 108 is determined based on right eye position 405 acquired by left and right eye position information acquisition unit 301. The processing of correction amount determination unit 302 will be described later.

[0021] Correction unit 303 corrects the image based on the amount of correction determined by correction amount determination unit 302 in response to instructions from CPU 202. Specifically, correction unit 303 corrects the image to be projected onto left-eye display 107 using the amount of correction to the left-side image determined by correction amount determination unit 302. Similarly, correction unit 303 corrects the image to be projected onto right-eye display 108 using the amount of correction to the right-side image determined by correction amount determination unit 302.

[0022] <Description of the processing by the correction amount determination unit 302> Here, we will explain the processing of the correction amount determination unit 302. The correction amount determination unit 302 calculates the amount of positional deviation between the ideal eye position relative to the eyepiece lens and the viewer's eye position acquired by the left / right eye position information acquisition unit 301, and determines the amount of sharpness correction for the image based on this amount of positional deviation. The correction amount determination unit 302 determines the amount of sharpness correction for each of the viewer's left and right eyes.

[0023] The processing of the correction amount determination unit 302 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a schematic diagram for explaining the positional deviation of the viewer's eyes, in which Fig. 5(a) shows the eye-captured image 401 when the viewer is wearing the HMD 101, and Fig. 5(b) is a table showing the ideal position coordinates of the left and right eyes in Fig. 5(a) and the position coordinates of the viewer's left and right eyes.

[0024] In FIG. 5( a), the ideal eye positions at which the left eye eyepiece 104 and the right eye eyepiece 105 allow the user to properly view an image are shown as an ideal left eye position 501 and an ideal right eye position 502. If the ideal left eye position 501 is the center of the left-side image capturing area 402, the coordinates of the ideal left eye position 501 in this embodiment are (x, y) = (0, 0) as shown in FIG. 5( b). Similarly, if the ideal right eye position 502 is the center of the right-side image capturing area 403, the coordinates of the ideal right eye position 502 in this embodiment are (x, y) = (0, 0) as shown in FIG. 5( b). The viewer's left eye position 404 is (x, y) = (−100, 100) as acquired by the left / right eye position information acquisition unit 301, and the viewer's right eye position 405 is (x, y) = (−10, 0). The left eye ideal position 501 and left eye position 404 can be used to calculate the left eye position shift amount (Δx, Δy)=(−100, 100), and the right eye ideal position 502 and right eye position 405 can be used to calculate the right eye position shift amount (Δx, Δy)=(−10, 0).

[0025] At this time, the viewer is wearing the HMD 101 in a position that is shifted from the ideal position, and the right eye is relatively close to the ideal right eye position 502, while the left eye is relatively far from the ideal left eye position 501.

[0026] Next, the amount of sharpness correction for the left-eye image and the right-eye image is determined based on the amount of misalignment between the left and right eyes. FIG. 6 is a schematic diagram showing the amount of sharpness correction relative to the amount of misalignment between the eyes. In FIG. 6, the horizontal axis represents the amount of misalignment in the x-direction, and the vertical axis represents the amount of misalignment in the y-direction, with the pattern in the graph representing the strength of sharpness correction. Note that sharpness correction here refers to, for example, edge enhancement processing, which is performed by applying a filter that emphasizes high-frequency regions to the image. The amount of correction can be changed in stages by setting multiple weights for the filter coefficients. FIG. 6 shows that the higher the density of each pattern, the greater the amount of sharpness correction. That is, in this embodiment, the greater the amount of misalignment between the eyes, the greater the amount of sharpness correction. This is because the greater the deviation between the eye positions from the ideal position, the more the sharpness deteriorates, and therefore the greater the amount of misalignment, the greater the amount of sharpness correction required.

[0027] Additionally, areas where the misalignment amount exceeds a certain level (the areas indicated by the highest density patterns in FIG. 6 ) are indicated by solid black and are shown as error areas. This means that if the eye misalignment amount is too large, sharpness correction will not be performed. In this case, the viewer is notified to reattach the HMD 101, and the left / right eye position information acquisition unit 301 performs processing again after reattaching the HMD 101. The graph origin (Δx, Δy)=(0,0) in FIG. 6 indicates the ideal eye position because the misalignment amount is zero. Since there is almost no degradation in sharpness at the ideal eye position, sharpening correction processing will not be performed. Point 601 indicating the left eye misalignment amount in FIG. 6 corresponds to the misalignment amount (Δx, Δy)=(-100,100) of the left eye position 404, and point 602 indicating the right eye misalignment amount corresponds to the misalignment amount (Δx, Δy)=(-10,0) of the right eye position 405. Therefore, in this embodiment, <edge enhancement amount: 2> is applied to the left image, and <edge enhancement amount: 0> is applied to the right image.

[0028] In Figure 5(a), the left eye is more displaced from the ideal position than the right eye, so the amount of sharpness correction for the left eye is greater than that for the right eye. In this way, the amount of sharpness correction is determined according to the amount of sharpness deterioration caused by the displacement of the eye position from the ideal position, so it is possible to produce images with equal sharpness for the left and right eyes.

[0029] According to this embodiment, it is possible to reduce the difference in sharpness between the left and right eyes that occurs due to misalignment of the HMD 101, thereby reducing the viewer's discomfort and visually induced motion sickness.

[0030] While FIG. 6 shows the sharpness correction amount increasing along concentric circles relative to the amount of misalignment, the shape is not limited to this, and the magnitude of the correction amount may be anisotropic. Although no sharpness correction processing is performed near the origin in FIG. 6, the sharpness correction amount may also be set near the origin. In this case, the correction unit 303 performs correction so that the sharpness is improved for both the left-eye and right-eye images. In this embodiment, the sharpness correction amount is determined so that the left-eye and right-eye images have approximately the same sharpness, but the method for determining the correction amount is not limited to this. For example, the viewer's dominant eye may be obtained separately via the input device 213, and the sharpness correction amount may be increased for the viewer's dominant eye.

[0031] Next, the processing flow of the image processing device 201 in this embodiment will be described with reference to the flowchart in Fig. 7. The CPU 202 reads out a program that implements the flowchart shown in Fig. 7 stored in the ROM 204 or HDD 211, and executes it using the RAM 203 as a work area. This allows the CPU 202 to fulfill the roles of the various functional components shown in Fig. 3. In the flowcharts that follow, each process (step) will be represented by "S."

[0032] In S701, the left and right eye position information acquisition unit 301 acquires position information of the left eye and the right eye of the viewer wearing the HMD 101. The position information of the left eye and the right eye can be acquired using, for example, the eye camera 110 of the HMD 101.

[0033] In S702, correction amount determination unit 302 uses the left eye position information acquired in S701 to calculate the amount of positional deviation from ideal left eye position 501, and determines the amount of sharpness correction for the left image based on the amount of positional deviation. Similarly, using the right eye position information acquired in S701, correction amount determination unit 302 calculates the amount of positional deviation from ideal right eye position 502, and determines the amount of sharpness correction for the right image based on the amount of positional deviation.

[0034] In S703, the sharpness of each of the left image and the right image is corrected based on the sharpness correction amount for each of the left image and the right image acquired in S702. Note that when the processing of S703 ends, the processing shown in FIG. 7 ends.

[0035] The processes from S701 to S703 in this embodiment may be automatically performed when the viewer wears the HMD 101. Alternatively, the image processing device 201 may be provided with a "wearing position deviation reduction mode," and the viewer may instruct the image processing device 201 via the input device 213 to switch between performing and not performing the processes.

[0036] As described above, according to this embodiment, the sharpness of the left and right images is corrected based on the amount of misalignment between the left and right eyes of a viewer wearing an HMD. As a result, even if the viewer's eyes are misaligned when wearing an HMD, the sharpness of the images is not reduced, and the viewer can view images with approximately the same level of sharpness in the left and right images. Therefore, since the difference in sharpness between the left and right images due to misalignment in the viewing position is reduced, the viewer can effectively reduce discomfort and visually induced motion sickness when viewing images with an HMD.

[0037] <Modification of the First Embodiment> Here, a modification of the first embodiment will be described. The left and right eye position information acquisition unit 301 in the first embodiment may determine whether the viewer's eyes are fatigued from the eye-captured image 401 captured by the eye camera 110, and change the correction amount determined by the correction amount determination unit 302 based on the determination result. For example, when the left eye position 404 and the right eye position 405 cannot be detected in the eye-captured image 401 for a certain period of time or more (i.e., a predetermined threshold or more) due to the viewer closing their eyes, it is determined that the viewer's eyes are fatigued.

[0038] However, the determination of fatigue is not limited to this, and for example, fatigue may be determined from the frequency of the viewer's blinking (i.e., the number of times the left eye position 404 and the right eye position 405 become undetectable within a certain period (i.e., a predetermined threshold or more).Fatigue may also be determined based on the frequency of the viewer's blinking (i.e., the number of times the blinking is equal to or greater than a predetermined threshold).Fatigue may also be determined to be due to bloodshot eyes if the contrast between the whites and pupils of the eyes is below a certain level.

[0039] Since eye fatigue can cause images to appear blurred, when the left / right eye position information acquisition unit 301 determines that fatigue exists, the correction amount determination unit 302 increases the amount of correction for sharpness. This modification makes it possible for a viewer to view images clearly even if they have eye fatigue.

[0040] <Embodiment 2> In the first embodiment, a mode has been described in which the sharpness of the left and right images is corrected based on the amount of misalignment between the left and right eyes of a viewer wearing an HMD. In the first embodiment, the misalignment of the HMD is cited as a factor that causes a difference in sharpness between the left and right images, but it is also possible that the visual characteristics of the viewer's left and right eyes are different. For example, if the viewer's eyesight differs from that of the left eye, a perceptual difference due to visual acuity will remain even if sharpness due to misalignment of the eyes is corrected.

[0041] Therefore, in this embodiment, a mode of performing sharpness correction taking into consideration not only the amount of deviation of the eye position when wearing the HMD but also the visual characteristics of the viewer will be described. Note that the hardware configuration of the image processing device 201 according to this embodiment is the same as that of the first embodiment, and therefore a description thereof will be omitted.

[0042] The functional configuration of the image processing device 201 in this embodiment will be described with reference to Fig. 8. The image processing device 201 in this embodiment has a left / right eye position information acquisition unit 301, a correction amount determination unit 302, and a correction unit 303, just like in the first embodiment, and further has a visual information acquisition unit 801 in this embodiment. The left / right eye position information acquisition unit 301 and the correction unit 303 perform the same processes as in the first embodiment, and therefore description thereof will be omitted. The correction amount determination unit 302 performs processes that are partly different from those in the first embodiment.

[0043] The visual information acquisition unit 801 acquires visual information from the viewer's left eye and right eye in response to instructions from the CPU 202. The visual information from the viewer's left eye and right eye is information that indicates the viewer's visual sensitivity to sharpness, such as visual acuity and contrast sensitivity. The visual information acquisition unit 801 presents user interface (UI) screens such as those shown in FIGS. 9(a) and 9(b) on the left-eye display 107 and the right-eye display 108, and acquires visual information from the viewer via the input device 213.

[0044] Fig. 9(a) is a visual acuity input UI screen 901 that allows the viewer to directly input their visual acuity, and Fig. 9(b) is a visual acuity measurement UI screen 905 that acquires the viewer's visual acuity by measuring it on the UI screen. The information acquired in both Fig. 9(a) and (b) is the visual acuity of the left and right eyes, and either Fig. 9(a) or (b) can be used.

[0045] The visual acuity input UI screen 901 in FIG. 9(a) has a left eye visual acuity selection section 902, a right eye visual acuity selection section 903, and a confirm button 904. For example, when a viewer wears the HMD 101, the visual acuity input UI screen 901 is displayed and accepts visual acuity input from the viewer. The left eye visual acuity selection section 902 and the right eye visual acuity selection section 903 are, for example, drop-down lists that allow visual acuity to be selected in increments of 0.1. The viewer selects the visual acuity of their left eye and right eye from the list and confirms it by clicking the confirm button 904. The visual information acquisition section 801 can acquire the visual acuity input into the left eye visual acuity selection section 902 and the right eye visual acuity selection section 903 as the viewer's visual information.

[0046] The visual acuity measurement UI screen 905 in FIG. 9(b) has a Landolt ring display section 906, an orientation selection section 907, and an enter button 904. For example, when the viewer wears the HMD 101, the visual acuity measurement UI screen 905 is displayed only on either the left-eye display 107 or the right-eye display 108, and the display on which the visual acuity measurement UI screen 905 is not displayed is darkened. The Landolt ring display section 906 displays a Landolt ring for visual acuity testing, and the viewer selects the orientation of the slits in the Landolt ring using the orientation selection section 907 and presses the enter button 904 to enter the selection. If the orientation set in the orientation selection section 907 is correct for the orientation of the slits displayed in the Landolt ring display section 906, the Landolt ring displayed in the Landolt ring display section 906 is updated, and the viewer is prompted to select the orientation of the slits in the Landolt ring again. The Landolt ring starts out at a size equivalent to 0.1 degrees of visual acuity, and if the answer to the slit direction is correct, the size of the Landolt ring is reduced so that the visual acuity increases by 0.1 degrees. This process is repeated to measure the viewer's visual acuity. After measuring the visual acuity of one eye, the visual acuity of the other eye is measured, thereby obtaining the visual acuity of each eye. The visual information acquisition unit 801 acquires the visual acuity measured in this way as the viewer's visual information.

[0047] In response to instructions from CPU 202, correction amount determination unit 302 determines the amount of sharpness correction for the left and right images based on the positions of the viewer's left and right eyes acquired by left and right eye position information acquisition unit 301 and the visual information for the viewer's left and right eyes acquired by visual information acquisition unit 801. Specifically, correction amount determination unit 302 determines the amount of correction to correct the sharpness of the left image to be displayed on left-eye display 107 based on left eye position 404 acquired by left and right eye position information acquisition unit 301 and the visual information for the viewer's left eye acquired by visual information acquisition unit 801. Similarly, correction amount determination unit 302 determines the amount of correction to correct the sharpness of the right image to be displayed on right-eye display 108 based on right eye position 405 acquired by left and right eye position information acquisition unit 301 and the visual information for the viewer's right eye acquired by visual information acquisition unit 801.

[0048] The processing of the correction amount determination unit 302 in this embodiment will be described. The correction amount determination unit 302 in this embodiment performs the same processing as in the first embodiment up to calculating the amount of deviation between the ideal eye position relative to the eyepiece lens described in the first embodiment and the viewer's eye position.

[0049] In this embodiment, the amount of sharpness correction is determined based on the amount of eye positional misalignment and the visual information acquired by the visual information acquisition unit 801. For example, assume that the amount of positional misalignment of the viewer's left eye ((Δx, Δy)=(−100, 100)) and the amount of positional misalignment of the right eye ((Δx, Δy)=(−10, 0)) described in the first embodiment are such that the viewer's visual acuity acquired by the visual information acquisition unit 801 is 1.0 for the left eye and 0.8 for the right eye. Because the deterioration in sharpness caused by the amount of eye positional misalignment is greater for the left eye than for the right eye, the sharpness of the right-hand image is higher than that of the left-hand image, but the viewer's visual acuity is lower for the right eye than for the left eye. In this case, the sharpness of both the left and right-hand images projected onto the viewer's retina will be reduced, and correction that reflects only the amount of eye positional misalignment may result in a difference in sharpness between the left and right eyes. Therefore, in this embodiment, the amount of sharpness correction is determined so that the left and right eyes see the same image by performing the same amount of correction on the left and right eyes.

[0050] As an example of a correction method, in addition to the sharpness correction according to the eye position described with reference to Fig. 6, the amount of edge enhancement can be increased by 1 for every 0.1 decrease in the viewer's visual acuity from 1.0. Therefore, in this embodiment, <edge enhancement: 2> can be applied to the left image and <edge enhancement: 2> can be applied to the right image.

[0051] The processing flow of the image processing device 201 in this embodiment will be described with reference to the flowchart in Fig. 10. The processing in S701 is the same as in the first embodiment, and its description will be omitted. When the processing in S701 ends, the process proceeds to S1002.

[0052] In S1002, the visual information acquisition unit 801 acquires visual information for the left eye and right eye of the viewer wearing the HMD 101. The visual information for the left eye and right eye can be acquired, for example, by using UI screens displayed on the left-eye display 107 and right-eye display 108 of the HMD 101 and by the viewer's input via the input device 213.

[0053] In S1003, the correction amount determination unit 302 determines the amount of sharpness correction for the left image based on the left eye position information acquired in S701 and the left eye visual information acquired in S1002. Similarly, the correction amount determination unit 302 determines the amount of sharpness correction for the right image based on the right eye position information acquired in S701 and the right eye visual information acquired in S1002.

[0054] The process of S703 is the same as that of the first embodiment, and therefore the description thereof will be omitted. When the process of S703 is completed, the process shown in FIG.

[0055] As described above, in this embodiment, sharpness correction is performed taking into account not only the amount of misalignment of the viewer's eyes when wearing the HMD, but also the viewer's visual characteristics. As a result, even if the viewer has different visual acuity in the left and right eyes and there is a misalignment when wearing the HMD, the viewer can view images with approximately the same sharpness in the left and right images. Therefore, since the difference in sharpness between the left and right images is reduced, it is possible to effectively reduce the viewer's discomfort and visually induced motion sickness when viewing images with an HMD.

[0056] <Embodiment 3> In the second embodiment, the sharpness of the left and right images is corrected based on the amount of misalignment between the left and right eyes of a viewer wearing an HMD and the visual acuity of the left and right eyes. In the present embodiment, the difference between the left and right images is made less perceptible by reducing the amount of parallax between the images when the difference between the left and right images is large.

[0057] First, because the left and right eyes of a human being are spaced about 5 to 7 centimeters apart, there is a horizontal shift (parallax) between the images seen by the left and right eyes. This parallax is used as a clue to perceive the depth of an object; when the amount of parallax is large, the distance from the eye to the viewed object is perceived as being close, and the viewed object appears to protrude. When the image difference between the left and right images is large, it is thought that the image difference is more easily perceived for a viewed object at a close distance than for a viewed object at a distant distance. Therefore, in this embodiment, a process is described for reducing the amount of parallax so that the viewed object appears farther away when the difference between the left and right images is large.

[0058] The hardware configuration of the image processing device 201 according to this embodiment is the same as that of the first embodiment, and therefore a description thereof will be omitted. The functional configuration of the image processing device 201 will be described with reference to Fig. 11. As in the first embodiment, the image processing device 201 according to this embodiment includes a left / right eye position information acquisition unit 301, a correction amount determination unit 302, and a correction unit 303, and further includes a comparison unit 1101 and a parallax amount change unit 1102 in this embodiment. The left / right eye position information acquisition unit 301, the correction amount determination unit 302, and the correction unit 303 perform the same processes as in the first embodiment, and therefore a description thereof will be omitted.

[0059] The comparison unit 1101 compares the left image with the right image and determines whether or not to perform processing in the parallax amount modification unit 1102. Here, the comparison of the left image with the right image can be performed, for example, by performing a two-dimensional Fourier transform on each image and comparing the frequency characteristics averaged in frequency space. If the difference in frequency characteristics between the left image and the right image is equal to or greater than an arbitrary threshold, the parallax amount modification unit 1102 determines to perform processing, and if the difference is less than the threshold, it determines not to perform processing.

[0060] Note that the comparison between the left and right images is not limited to a comparison of frequency characteristics. For example, the amount of edge enhancement for the left image calculated by the correction amount determination unit 302 may be compared with the amount of edge enhancement for the right image, and a determination may be made as to whether the difference is equal to or greater than a threshold. Since the amount of noise in an image tends to increase as the amount of edge enhancement increases, if the difference in the amount of edge enhancement between the left and right images is large, the difference between the left and right images may also be large.

[0061] When it is determined based on the comparison result by the comparison unit 1101 that processing in the parallax amount modification unit 1102 is to be performed, the parallax amount modification unit 1102 performs processing to modify the amount of parallax between the left and right images. FIG. 12 is a schematic diagram illustrating processing by the parallax amount modification unit 1102. FIG. 12(a) shows a left image 1201 and a right image 1202 before the amount of parallax is modified. FIG. 12(b) shows the geometric conditions between the object 1203 and the virtual cameras (left virtual camera 1204 and right virtual camera 1205) corresponding to FIG. 12(a). FIG. 12(c) shows the left image 1201 and the right image 1202 after the amount of parallax has been modified. FIG. 12(d) shows the geometric conditions between the object 1203 and the virtual cameras (left virtual camera 1204 and right virtual camera 1205) corresponding to FIG. 12(c).

[0062] 12(a), object 1203 in left image 1201 and right image 1202 has a large amount of parallax and is perceived as popping out compared to objects with a small amount of parallax. In this case, the geometric relationship between object 1203 and left virtual camera 1204 and right virtual camera 1205 is as shown in FIG. 12(b), and the viewing distance between the object and the virtual cameras, as indicated by the arrows in the figure, is relatively short.

[0063] The parallax amount modification unit 1102 modifies the image so that the subject 1203 moves away from the virtual camera as shown in Fig. 12(d). For example, if the left image 1201 and the right image 1202 are images obtained in a CG virtual space, the viewing distance is increased by moving the positions of the left virtual camera and the right virtual camera away from the subject 1203, and the amount of parallax between the left image 1201 and the right image 1202 is reduced. The left image 1201 and the right image 1202 with the amount of parallax modified in this way are shown in Fig. 12(c).

[0064] In addition to changing the position of the virtual camera, the processing of the parallax amount change unit 1102 can also eliminate the amount of parallax by making the left image 1201 and the right image 1202 the same image. For example, one of the images can be projected onto both the left-eye display 107 and the right-eye display 108.

[0065] The processing flow of the image processing device 201 in this embodiment will be described with reference to the flowchart in Fig. 13. The processing from S701 to S703 is the same as in the first embodiment, and therefore a description thereof will be omitted. After the processing of S703 is completed, the processing proceeds to S1304.

[0066] In S1304, the comparison unit 1101 compares the difference between the left image 1201 and the right image 1202, and determines whether or not to perform processing in the parallax amount modification unit 1102. If processing is to be performed, the process proceeds to S1305, and if processing is not to be performed, the process shown in FIG.

[0067] In S1305, the amount of parallax between the left image 1201 and the right image 1202 is changed so as to reduce the amount of parallax.

[0068] When the process of S1305 is completed, the process shown in FIG. 13 is terminated.

[0069] As described above, in this embodiment, the left image and the right image are compared, and if it is determined that the image difference is large, a process of reducing the amount of parallax is performed. By reducing the amount of parallax, even if there is a difference between the left and right images, it is possible to relatively reduce the perception of the difference. Therefore, since the perception of the image difference between the left image and the right image is reduced, it is possible to effectively reduce discomfort and visually induced motion sickness experienced by the viewer when watching images using an HMD.

[0070] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0071] The present disclosure also includes the following configurations and methods.

[0072] [Configuration 1] a position information acquisition means for acquiring position information of the left eye and the right eye of a wearer wearing the head mounted display; a correction means for correcting sharpness of a left image to be projected onto the left eye of the wearer and a right image to be projected onto the right eye of the wearer in the head mounted display, based on position information of the left eye and the right eye of the wearer, respectively; 1. An image processing device comprising:

[0073] [Configuration 2] the correction means has a correction amount determination means for determining a correction amount for correcting the sharpness, The correction amount determining means The image processing device according to configuration 1, wherein the head-mounted display determines a sharpness correction amount for the left-side image based on a difference between predetermined position information of the left eye at which the left-side image can be viewed and position information of the left eye of the wearer.

[0074] [Configuration 3] the correction means has a correction amount determination means for determining a correction amount for correcting the sharpness, The correction amount determining means The image processing device according to configuration 1, characterized in that in the head-mounted display, a sharpness correction amount for the right-side image is determined from a difference between predetermined position information of the right eye at which the right-side image can be properly viewed and position information of the right eye of the wearer.

[0075] [Configuration 4] 3. The image processing device according to configuration 2, wherein the correction amount set by the correction amount determination means for the left image and the correction amount for the right image have different values.

[0076] [Configuration 5] 4. The image processing device according to configuration 3, wherein the correction amount set by the correction amount determination means for the left image and the correction amount for the right image have different values.

[0077] [Configuration 6] visual information acquisition means for acquiring visual information for each of the wearer's left eye and right eye; The correction means corrects the sharpness based on the position information and the visual information. 6. The image processing device according to any one of configurations 1 to 5, wherein:

[0078] [Configuration 7] a comparison means for comparing the left image with the right image; a parallax amount changing means for reducing the amount of parallax between the left image and the right image based on the comparison result by the comparing means; 7. The image processing device according to any one of configurations 1 to 6, comprising:

[0079] [Configuration 8] A program for causing a computer to function as each of the means of the image processing device according to any one of the first to seventh configurations. [Explanation of symbols]

[0080] 301 Left and right eye position information acquisition unit 302 Correction amount determination unit 303 Correction Unit

Claims

1. a position information acquisition means for acquiring position information of the left eye and the right eye of a wearer wearing the head mounted display; a correction means for correcting sharpness of a left image to be projected onto the left eye of the wearer and a right image to be projected onto the right eye of the wearer in the head mounted display, based on position information of the left eye and the right eye of the wearer, respectively; 1. An image processing device comprising:

2. the correction means has a correction amount determination means for determining a correction amount for correcting the sharpness, The correction amount determining means 2. The image processing device according to claim 1, wherein the amount of sharpness correction for the left image is determined from the difference between predetermined position information of the left eye at which the left image can be viewed and position information of the left eye of the wearer in the head-mounted display.

3. the correction means has a correction amount determination means for determining a correction amount for correcting the sharpness, The correction amount determining means 2. The image processing device according to claim 1, wherein the amount of sharpness correction for the right-side image is determined from the difference between predetermined position information of the right eye at which the right-side image can be properly viewed and position information of the wearer's right eye in the head-mounted display.

4. 3. The image processing device according to claim 2, wherein the correction amount set by the correction amount determining means is different from the correction amount for the left image and the correction amount for the right image.

5. 4. The image processing device according to claim 3, wherein the correction amount set by the correction amount determining means is different from the correction amount for the left image and the correction amount for the right image.

6. visual information acquisition means for acquiring visual information for each of the wearer's left eye and right eye; The correction means corrects the sharpness based on the position information and the visual information.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

7. a comparison means for comparing the left image with the right image; a parallax amount changing means for reducing the amount of parallax between the left image and the right image based on the comparison result by the comparing means; 4. The image processing device according to claim 1, further comprising:

8. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 3.

9. a position information acquiring step of acquiring position information of the left eye and the right eye of a wearer wearing the head mounted display; and a correction step of correcting the sharpness of a left-side image to be projected into the left eye of the wearer and a right-side image to be projected into the right eye of the wearer in the head-mounted display based on position information of the wearer's left eye and right eye, respectively.

Images