Image processing apparatus, image processing method, and program

Abstract

To enable a user to easily determine whether a head-mounted display device is mounted at a correct position.SOLUTION: An image processing apparatus according to an embodiment of the present disclosure includes acquisition means for acquiring information indicating a position of an eye of a user wearing a head-mounted display device, and display control means for causing the head-mounted display to display, together with the eye position information, guide information for the user to perform alignment of the eye. The display device includes imaging means for imaging the eyes of the user, and the acquisition means acquires the eye position information on the basis of a captured image captured by the imaging means.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to alignment of a wearing position of a head-mounted display device. [Background technology]

[0002] Head-mounted displays (HMDs) are available, which are display devices that allow users to view images while wearing them on their heads. Video see-through HMDs are equipped with a display and two eyepieces, one for each of the user's eyes. Users can view images displayed on the display by looking into the display through the eyepieces. HMDs also have straps for attaching them to the head and an adjustment unit for adjusting the length of the straps, so that users can view images without having to constantly hold the HMD in their hands. However, if the HMD is not worn in the appropriate position, the images may appear blurred or otherwise be difficult to view properly.

[0003] To address this issue, Patent Document 1 describes a method in which different positioning images are displayed on a left-eye display and a right-eye display, and the user can determine whether the wearing position is appropriate by observing them with both eyes. Specifically, horizontal lines are displayed on one display and vertical lines on the other, and when the two images are observed with both eyes, the user determines whether the vertical and horizontal lines appear to overlap correctly, allowing the user to determine whether the wearing position is appropriate. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 09-304729 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the display method described in Patent Document 1, the user must determine whether the positional relationship between the left and right images is correct without knowing the correct positional relationship between the two. As a result, it is not easy for an inexperienced user to determine whether the image is being viewed correctly or whether the HMD is being worn in the correct position. [Means for solving the problem]

[0006] The image processing device of the present disclosure includes an acquisition means for acquiring eye position information indicating the eye position of a user wearing a head-mounted display device, and a display control means for displaying guide information for the user to align their eyes on the display device together with the eye position information. [Effects of the Invention]

[0007] The image processing device of the present disclosure makes it easier for the user to determine whether the head-mounted display device is being worn in the correct position. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an image processing system including an HMD. [Figure 2] FIG. 1 is a diagram showing the appearance of an HMD. [Figure 3] FIG. 2 is a diagram illustrating the internal configuration of an HMD. [Figure 4] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 5] 1 is a block diagram showing a functional configuration of an image processing apparatus according to a first embodiment; [Figure 6] FIG. 3 is a diagram illustrating a guide image in the first embodiment. [Figure 7] 3 is a flowchart showing a processing flow in the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a guide image displayed in a virtual space. [Figure 9] FIG. 10 is a diagram illustrating a guide image in the second embodiment. [Figure 10]FIG. 10 is a block diagram showing the functional configuration of an image processing apparatus according to a second embodiment. [Figure 11] 10 is a flowchart showing a processing flow according to a second embodiment. [Figure 12] 10A and 10B are diagrams illustrating a deviation between the user's eye and a target position. [Figure 13] FIG. 11 is a diagram illustrating a guide display in the third embodiment. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of an image processing apparatus according to a third embodiment. [Figure 15] 10 is a flowchart showing a processing flow according to a third embodiment. [Figure 16] FIG. 10 is a diagram showing correspondence between divided areas and numbers in an eye image. [Figure 17] FIG. 10 is a diagram showing candidates for guide information. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Note that the same components will be described with the same reference numerals.

[0010] First Embodiment The image processing device of the present disclosure displays, on an HMD, guide information for the user to align their eyes together with eye position information indicating the position of the user's eyes. In the first embodiment, the eye position information is acquired from a captured image captured by an imaging means provided in the HMD for capturing an image of the user's eyes. Furthermore, as the guide information, a target display is displayed at a target position that serves as an indicator for the user's eye alignment. An example will be described in which the target display is superimposed on the captured image of the user's eyes and displayed on the same screen.

[0011] (Image processing system configuration) FIG. 1 is a diagram showing the system configuration of an HMD system 1 as an example of an image processing system including an image processing device according to the present disclosure. The HMD system 1 includes an image processing device 100 and an HMD 101, which are communicatively connected via a transmission path 103 to communicate image data, control signals, and the like. The transmission path 103 includes a video signal line such as an HDMI (registered trademark) cable and a data signal line such as a USB cable. In addition, input devices such as a controller and a keyboard are communicatively connected to the image processing device 100 to receive input from a user. The communication connection between the HMD 101 and the image processing device 100 and the communication connection between the image processing device 100 and the input devices may be a wired connection such as a USB cable, or a wireless connection such as Bluetooth (registered trademark).

[0012] The HMD 101 is an output device of the HMD system 1 and is a head-mounted display device. The HMD 101 is worn on the head of a user, and allows the user's left and right eyes to observe enlarged virtual images of a left-eye display image and a right-eye display image input from the image processing device 100. The image processing device 100 acquires the orientation of the head of the user wearing the HMD 101, i.e., posture information of the HMD 101, generates a left-eye display image and a right-eye display image according to the posture information, and outputs them to the HMD 101.

[0013] FIG. 2(a) is a perspective view showing the front surface of the HMD 101, and FIG. 2(b) is a perspective view showing the back surface of the HMD 101. The surface that contacts the user's head is referred to as the back surface. As shown in FIG. 2(a), the main body of the HMD 101 is provided with a pair of cameras 201L and 201R that can capture images of the surroundings of the user wearing the HMD 101. The HMD 101 is also provided with a strap 210 for attaching the HMD 101 to the user's head and an adjustment unit 211 for adjusting the attachment position. The adjustment unit 211 is formed, for example, with a hook-and-loop fastener or the like, and allows fine adjustment of the length of the strap 210. The adjustment unit 211 is not limited to a hook-and-loop fastener, and may be any type of adjustment mechanism that allows the user to adjust the attachment position of the HMD 101. For example, a dial-type length adjustment mechanism may be used instead of the hook-and-loop fastener. The number of straps 210 and adjustment units 211 is not limited to one each, and multiple straps 210 and adjustment units 211 may be provided. The strap may be connected not only around the back of the head but also across the top of the head. Furthermore, although the HMD 101 shown in FIGS. 1 and 2 is a goggle type, it is not limited to this and may be a sunglasses type or other type.

[0014] As shown in Figure 2(b), a pair of eyepieces 204L and 204R are provided on the back of the main body of the HMD101, and inside the main body are imaging devices (hereinafter referred to as eye cameras) 207L and 207R that can capture images of the eyes of a user wearing the HMD101, a display panel 203, an IMU (Inertial Measurement Unit) 202 that can acquire posture information of the HMD101, and a proximity sensor 206.

[0015] 3(a) is a diagram showing the internal configuration of the HMD 101. As described above, the HMD 101 includes multiple cameras 201L and 201R, eye cameras 207L and 207R, a proximity sensor 206, a display panel 203, eyepieces 204L and 204R, an IMU 202, etc. In the following description, when there is no need to distinguish between the left and right of the cameras 201L and 201R, the eye cameras 207L and 207R, and the eyepieces 204L and 204R, they will be denoted by the reference numerals 201, 207, and 204, respectively.

[0016] As shown in Figure 3(a), eyepiece 204L and camera 201L are arranged in this order, starting from the one closest to the eye, so as to face the user's left eye. Eyepiece 204R and camera 201R are arranged in this order, starting from the one closest to the eye, so as to face the user's right eye. A display panel 203 is disposed between eyepiece 204 and camera 201.

[0017] The camera 201 is an RGB camera that captures images of the real space around the user. The images captured by the camera 201 are sequentially transmitted to the image processing device 100 and used to generate, for example, an augmented reality image (AR image). The camera 201 is a so-called stereo camera in which two cameras are arranged on the left and right at a known distance. The number of cameras 201 is not necessarily limited to two, and may be one. Also, a camera for position tracking may be provided separately from the camera 201 for generating AR images.

[0018] In the present embodiment, a configuration example is shown in which the camera 201 for generating AR images is also used as a camera for position tracking. However, a camera for position tracking may be provided in addition to the camera 201 for generating AR images. Images of the user's surroundings captured by the camera 201 may be used to estimate the user's position and generate an environment map by Visual SLAM (Simultaneous Localization and Mapping). The user's position and orientation are expressed, for example, in 6DoF (degrees of freedom). Specifically, the self-position and orientation are expressed by forward / back, up / down, left / right, pitch, yaw, and roll. Note that the method for estimating the user's position is not limited to Visual SLAM using multiple cameras, and may be performed using a distance sensor such as LiDAR or the IMU 202.

[0019] The eye camera 207 is an imaging device that captures an image of a user's eye by projecting light onto the cornea of ​​the user's eye and capturing the reflected light. The eye camera 207 includes an imaging device 207L that captures an image of the left eye and an imaging device 207R that captures an image of the right eye. FIG. 3(b) is a schematic diagram illustrating eye imaging by the eye camera 207. As shown in the figure, the eye camera 207 includes an illumination light source 311, a light splitter 312, a light-receiving lens 313, and an imaging element 314 with a two-dimensional array of photoelectric elements such as a CMOS. The illumination light source 311 is a light source that projects light onto the eye 300. For example, it may consist of multiple infrared light-emitting diodes and is arranged around the eyepiece 204. Note that the arrangement of the components of the eye camera 207 is not limited to the example shown in FIG. 3(b). An image of the illuminated eye and its surroundings passes through the eyepiece 204, is reflected by the light splitter 312, and is then focused on the imaging element 314 by the light-receiving lens 313. The light receiving lens 313 positions the pupil of the user's eye 300 and the image sensor 314 in a complementary imaging relationship. The image sensor 314 outputs an image including the eye 300 within its imaging range to the image processing device 100. The image processing device 100 combines the images of the left and right eyes obtained from the eye cameras 207R and 207L to generate an eye image in which both eyes appear on one screen (FIG. 6(a)). In the following explanation, an image in which both eyes appear is referred to as an eye image. The eye image is generated so that the user's right eye appears on the right side and the left eye appears on the left side. In other words, the eye image generated is an image that looks as if the user were looking at their own eyes reflected in a mirror.

[0020] In this embodiment, the eye camera 207 is fixedly mounted on the HMD 101, and the imaging range of the eye camera 207 is always the same. The eye camera 207 captures an image whose imaging range includes at least one of the user's eyes. More preferably, the eye camera 207 captures an image whose imaging range includes one of the user's eyes and the area around that eye. Furthermore, the eye camera 207 captures an image whose imaging range includes a portion of the HMD 101 and one of the user's eyes. In this case, by observing the image of the eye, the user can easily grasp the relative position of the eye with respect to the portion of the HMD 101 shown in the image. The portion of the HMD 101 may be, for example, the outer edge of the eyepiece 204. In other words, the eye camera 207L captures an image whose imaging range includes the eyepiece 204L and the user's left eye, and outputs the image to the image processing device 100. Similarly, the eye camera 207R captures an image including the eyepiece 204R and the user's right eye in its imaging range, and outputs the image to the image processing device 100.

[0021] It should be noted that the eye camera 207 does not necessarily have to be provided for each eye, and may be configured as a single camera that captures images of both eyes simultaneously. The installation location is also not limited to the example in FIG. 2(b). The operations of the camera 201 and the eye camera 207 are controlled by the CPU 401 of the image processing device 100.

[0022] The proximity sensor 206 is a sensor that is provided, for example, on the surface of the housing of the HMD 101 that comes into contact with the user's head and detects that the HMD 101 is being worn by the user. The proximity sensor 206 outputs a signal indicating that wearing of the HMD 101 has been detected when the distance between the user's head and the proximity sensor 206 is smaller than a predetermined threshold.

[0023] The display panel 203 is configured with a display panel such as a liquid crystal panel or an organic EL panel. Furthermore, eyepieces 204L and 204R corresponding to the left and right eyes, respectively, are arranged in front of the display panel 203. The user of the HMD 101 can observe an enlarged virtual image of the display image displayed on the display panel 203 through these eyepieces 204L and 204R. Note that although FIGS. 1 to 3 show an example of one display panel 203, two displays corresponding to the left and right eyes may be provided at positions spaced a predetermined distance from the eyepieces 204L and 204R, respectively.

[0024] The image processing device 100 performs processing to generate a display image for the left eye and a display image for the right eye, outputs these images to the HMD 101, and displays them on the display panel 203 of the HMD 101. By providing an appropriate parallax between the display image for the left eye and the display image for the right eye, it is possible to give the user an image perception with a sense of depth.

[0025] The HMD system 1 of this embodiment will be described as a system configuration in which the image processing device 100 and the HMD 101 are configured separately. However, the HMD system 1 may also be configured as an integrated HMD system, for example, in which the image processing device 100 is included inside the HMD 101. The configuration of the HMD 101 is not limited to the examples shown in FIGS. 2 and 3. For example, the HMD 101 may not be equipped with a camera 201 for generating a display image. The HMD 101 may also be equipped with a camera for position tracking in addition to the camera 201 for generating a display image. The HMD 101 may further include a distance sensor, a speaker for outputting audio, a microphone for inputting audio, a vibrator for generating vibrations, an LED lamp for indicating the status of the device, etc.

[0026] 4 is a diagram showing an example of the configuration of an image processing device 100 according to the present disclosure. The image processing device 100 has a CPU 401, a GPU 402, a RAM 403, a ROM 404, an HDD 405, a general-purpose interface (I / F) 406, an output I / F 407, a network I / F 408, and an input I / F 409. These components are connected via a system bus 410.

[0027] The CPU 401 comprehensively controls the entire HMD system 1. The CPU 401 is a processor that controls the entire system by reading and executing a system program stored in the ROM 401 or HDD 405. The CPU 401 also realizes the operation of this embodiment by reading and executing an application program stored in the ROM 404 or HDD 405. Although FIG. 4 shows one CPU, the system may be configured with multiple CPUs.

[0028] The GPU 402 is a processor that performs image processing upon receiving an instruction from the CPU 401. The GPU 402, for example, performs rendering of CG (computer graphics) and generates a display image to be displayed on the display of the HMD 101. The display image may be CG only, or may be an AR image in which CG, which is a virtual object, is superimposed on a real image acquired from the camera 201 of the HMD 101. Although FIG. 4 shows one GPU, the system may be configured with multiple GPUs. In this embodiment, the GPU 402 generates a guide image upon receiving an instruction from the CPU 401. The process of generating a guide image will be described later.

[0029] The RAM 403 is a general-purpose RAM and is used as a work memory for temporarily storing various pieces of information when, for example, the CPU 401 executes a program. The ROM 404 is a general-purpose ROM and stores, for example, programs executed by the CPU 401 and the GPU 402. The HDD (hard disk) 405 is a secondary storage device such as an optical disk drive and is connected to the CPU 401 via an I / F such as a serial ATA (SATA). The HDD 405 includes a storage medium for storing image data, results of various processes, other data, and various programs executed by the CPU 401 and the GPU 402. The HDD 405 may be an SSD or flash memory.

[0030] The general-purpose I / F 406 is a serial bus interface such as USB or IEEE1394, and connects the HMD 101 and other peripheral devices. The general-purpose I / F 406 connects the camera 201, eye camera 207, alignment camera, and IMU 202 of the HMD 101. The CPU 101 can acquire images captured by the camera 201 and eye camera 207 of the HMD 101 via the general-purpose I / F 406. The CPU 101 can also read posture information of the HMD 101 from the IMU 202 via the general-purpose I / F 406. The posture information can also be input via a mouse, keyboard, camera, etc.

[0031] The output I / F 407 is a video output interface such as DVI, HDMI, display port, etc. The CPU 401 transmits display data such as a guide image and an AR image to the HMD 101 via the output I / F 408 and displays them on the display panel 203. The output I / F 407 is also used to output audio to a speaker (not shown).

[0032] The network I / F 408 is an interface for establishing a communication connection to a LAN or the Internet under the control of the CPU 401. The image processing device 100 can establish a communication connection with an HMD system 1 used by another person via the network, or with an external content distribution server, etc. The system bus 410 controls the flow of data throughout the image processing device 100.

[0033] The input I / F 409 is a serial bus interface such as USB or IEEE1394, and connects to an input device 411 such as a keyboard, mouse, touch panel, or dedicated controller for the HMD. The CPU 101 reads data from the input device 411 via the input I / F 409.

[0034] The configuration of the image processing device 100 is not limited to the above example, and may be changed as appropriate depending on the device configuration or functions of the HMD 101 to be connected.

[0035] (HMD functional configuration) Fig. 5 is a diagram showing the functional configuration of the image processing device 100 in the first embodiment. The image processing device 100 has an eye position information acquisition unit 501, a target position acquisition unit 502, a guide image generation unit 503, and a display control unit 504. Program modules corresponding to the components shown in Fig. 5 are included in an application program. The CPU 401 or the GPU 402 executes each program module to function as each component shown in Fig. 5. The same applies to each embodiment described later in this specification.

[0036] The eye position information acquisition unit 501 acquires information indicating the eye positions of a user wearing the HMD 101. In the following description, information indicating the eye positions of the user is referred to as eye position information. In the first embodiment, the eye position information acquisition unit 501 acquires eye position information based on an image of the user's eyes captured by the eye camera 207 provided in the HMD 101. The eye position information acquisition unit 501 generates an eye image 601 from the captured image acquired from the eye camera 207, stores the image in the RAM 403, and outputs the image to the guide image generation unit 503. The eye position information acquisition unit 501 may also analyze the image of the user's eyes acquired from the eye camera 207 and acquire pixel coordinates of representative positions of each of the user's left and right eyes in the captured image as eye position information. In this case, the representative eye positions are, for example, pixel coordinates of the centers of the pupils in the captured image. Specifically, the feature information acquisition unit 1001 detects black, round areas included in the eye image by image processing. Then, by calculating the center position of the detected area, pixel coordinates corresponding to the representative position of the eye are obtained. Note that the method of obtaining pixel coordinates corresponding to the representative position of the user's eye is not limited to this, and known eye tracking technology may also be used. Furthermore, the representative position of the eye is not limited to the center of the pupil, and may be another part. For example, the center of the iris of the eye or a position indicating the end point of the eye, such as the inner corner or outer corner of the eye, may be used as the representative position. Alternatively, the center point of the line connecting the inner corner and outer corner of the eye may be used as the representative position of the eye. The representative position of the eye may be any point that can be identified as part of the eye from the eye image.

[0037] FIG. 6(a) is a diagram showing an example of an eye image 601. The eye image 601 is generated based on an image captured by the eye camera 207 as described above. The eye image 601 is generated so that a region 604R including a right eye 607R is output on the right side of the eye image 601, and a region 604L including a left eye 607L is output on the left side of the eye image 601. In other words, the eye image 601 is generated so that the user wearing the HMD 101 can check their own eyes as if looking in a mirror. The region 604R is an area corresponding to the right eyepiece 204R, and the region 604L is an area corresponding to the left eyepiece 204L. In other words, the eye image 601 is captured so that the outer edge of the eyepiece 204 is included in the capturing range.

[0038] The target position acquisition unit 502 acquires information indicating the target position that is pre-stored in the HDD 405. The target position is a position that the user uses as an index for eye alignment, and is a specific position in the captured image captured by the eye camera 207 that has been determined in advance. The specific position is expressed, for example, as coordinates in the captured image. The target position is also determined to be a position that corresponds to a part of the head-mounted display. For example, it is determined as a position on the captured image that corresponds to the center position of the eyepiece 204. In this embodiment, the information indicating the target position is defined as pixel coordinates (x, y) in the eye image 601 generated by the eye position information acquisition unit 501, as follows:

[0039] Left eye target pixel coordinates = (xtarget_L, ytarget_L) Right eye target pixel coordinates = (xtarget_R, ytarget_R)

[0040] The above-mentioned coordinates (x, y) are expressed with the horizontal direction as the x-axis and the vertical direction as the y-axis in the eye image 601. In the guide images 612 and 613 shown in Figures 6(b) and 6(c), the center position of the target display 608L is the pixel coordinate corresponding to the target position for the left eye, and the center position of the target display 608R is the pixel coordinate corresponding to the target position for the right eye. In the following description, when there is no need to distinguish between the left and right of the target displays 608L and 608R, they will be referred to as 608.

[0041] In this embodiment, the eye image 601 is positioned in advance so that, for example, the center point of an area 604 corresponding to the eyepiece 204 is the target position. Image generation conditions and display conditions are determined so that when the user's eye is aligned with the target position, the images with binocular parallax displayed on the display panel 203 are fused sharply into a single stereoscopic image.

[0042] The method of expressing the target position is not limited to the example of expressing it in pixel coordinates as described above, and any method may be used as long as the target position of the eye in the eye image 601 is clear. For example, it may be expressed as a ratio in the entire eye image without using pixel coordinates. In this case, the upper left coordinate of the eye image may be set as the origin (0,0) and the lower right coordinate as (1.0,1.0), and the left and right target positions may be expressed as a decimal ratio.

[0043] The guide image generation unit 503 generates a guide image for the user wearing the HMD 101 to align their eyes. The guide image includes information indicating the target position, for example, a target display 608, and is displayed together with information indicating the user's current eye position (eye position information). That is, the guide image is generated so that the target position, which is guide information, and the eye position information are displayed on the same screen. Specifically, the guide image generation unit 503 generates guide images 612 and 613 by superimposing a target display 608 indicating the target position on an image 601 of the user's eye captured by the eye camera 207 and acquired by the eye position information acquisition unit 501. The guide images 612 and 613 are generated and updated in real time while the eye image is being captured.

[0044] The display control unit 504 outputs the guide images 612 and 613 generated by the guide image generation unit 503 to the HMD 101 via the output I / F 407, and causes them to be displayed in real time on the display panel 203 of the HMD 101. The display method will be described later.

[0045] 6(b) and (c) are diagrams showing examples of guide images 612 and 613 displayed in the first embodiment. Each of the guide images 612 and 613 shown in the diagrams represents one frame of a moving image. The guide images 612 and 613 also include an eye image that reflects in real time the image captured by the eye camera 207 and acquired by the eye position information acquisition unit 501, and a target display 608 that indicates the target position as guide information.

[0046] As shown in FIGS. 6(b) and 6(c), guide images 612 and 613 include superimposed images 602 and 603 in which a left eye target display 608L and a right eye target display 608R are superimposed on an image of the eyes. Specifically, the left eye target display 608L is superimposed on an area 604L corresponding to the left eyepiece 204L, and the right eye target display 608R is superimposed on an area 604R corresponding to the right eyepiece 204R. FIG. 6(b) shows a state in which the position of the user's eyes is misaligned with the target display 608, while FIG. 6(c) shows a state in which the position of the user's eyes is aligned with the target display 608. While viewing the guide images 612 and 613, the user moves the position of the HMD 101 body and adjusts the strap 210 and adjustment unit 211 so that the target displays 608L and 608R overlap with their own eyes in the images.

[0047] Note that the display forms of the guide images 612 and 613 shown in FIGS. 6(b) and 6(c) are merely examples. In FIG. 6, the target display 608 indicating the target position is represented using a "circle" symbol. However, this is not limiting; any other target display, such as a "+" or an "x," may be used as long as it can clearly indicate the target position. Furthermore, the size of the target display 608 is not limited to the size indicated by the guide images 612 and 613, and may be any size as long as the user can simultaneously view the entire target display in the guide images 612 and 613 and the position of the user's eyes using the HMD 101. However, if the size of the target display 608 is large enough to cover the entire area 604 of the eyepiece 204, the user will not know in which direction to move the HMD 101. Therefore, it is preferable that the size of the target display 608 be such that it can be displayed at a predetermined distance from the outer edges of the areas 604L and 604R. The distance may be, for example, approximately the distance from the inner corner to the outer corner of the eye in the eye image or the diameter of the iris. If the size of the target display 608 is set to about the size of the iris of the eye, it becomes easier to align the pupil position with the target position. Also, if the size of the target display 608 is set to about the size of the pupil, it becomes easier to align the pupil position with the target position.

[0048] Fig. 7 is a flowchart showing the flow of processing executed by the image processing device 100 of the first embodiment. The image processing device 100 executes a series of processes shown in the flowchart of Fig. 7 by having the CPU 401 execute a program stored in the ROM 403 using the RAM 402 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 401, and the image processing device 100 may be configured so that some or all of the processes are executed by one or more processing circuits (ASICs) other than the GPU 402 and CPU 401. In the following description, the symbol S represents a step.

[0049] The process shown in FIG. 7 starts when it is detected that the HMD 101 is being worn, and is executed in real time until the user completes adjusting the wearing position of the HMD 101. The CPU 401 determines whether the user has worn the HMD 101 based on a signal indicating whether wearing has been detected, which is input from the proximity sensor 206 of the HMD 101. Specifically, if the signal input from the proximity sensor 206 is a signal indicating that wearing has been detected, the CPU 401 determines that the user has worn the HMD 101. Alternatively, the CPU 401 may determine that the user has worn the HMD 101 when it receives an instruction to display a guide image through a user operation. The CPU 401 also determines whether the user has completed adjusting the wearing position based on a change in posture information input from the IMU 202 of the HMD 101. Specifically, if the amount of change in the posture information input from the IMU 202 is greater than a threshold, the CPU 401 determines that the user has completed adjusting the wearing position. Alternatively, the CPU 401 may determine that the adjustment is complete when an instruction to end the display of the guide image is received through a user operation.

[0050] In S701, the eye position information acquisition unit 501 acquires image data captured by the eye camera 207. The eye position information acquisition unit 501 generates an eye image 601 based on the acquired image data. As shown in FIG. 6(a) described above, the eye image 601 includes areas 604L and 604R corresponding to the eyepieces 204L and 204R and the user's eyes 607L and 607R in its imaging range. Note that the eye image 601 is not limited to the example shown in FIG. 6(a), and any image may be used as long as it is captured so that information about the position of the user's eyes can be acquired. The eye position information acquisition unit 501 outputs the eye image 601 to the guide image generation unit 503.

[0051] In S702, the target position acquisition unit 502 acquires information on the target position stored in the HDD 405. As described above, for example, the target pixel coordinates of the left eye=(xtarget_L, ytarget_L) and the target pixel coordinates of the right eye=(xtarget_R, ytarget_R) are acquired. The target position acquisition unit 502 outputs the acquired information on the target position to the guide image generation unit 503.

[0052] In S703, the guide image generation unit 503 generates guide images 612 and 613 using the eye image 601 acquired from the eye position information acquisition unit 501 and the target position information acquired from the target position acquisition unit 502. An example of a method for generating guide images will be described below.

[0053] First, the guide image generating unit 503 displays a target display superimposed on pixel coordinates (xtarget_L, ytarget_L), (xtarget_R, ytarget_R) indicating the target position in a two-dimensional pixel coordinate plane corresponding to the eye image 601. In this specification, superimposing a display is referred to as superimposing. As described above, the target display 608 may be any mark such as a "circle" symbol. Note that the user may specify the type of mark to be used as the target display. In the following description, the eye images with the target displays 608L, 608R superimposed thereon are referred to as superimposed images 602, 603.

[0054] Next, the guide image generation unit 503 converts the display size of the superimposed images 602 and 603 on the HMD 101 so that they are appropriate for the user's viewing on the HMD 101. If the display size is too large, the user will need to move their head and shift their line of sight to confirm the target position, which is inconvenient. On the other hand, if the display size is too small, it may be difficult to visually recognize the mark or the position of the user's eyes. Therefore, the field of view θ, which indicates the display size of the superimposed images 602 and 603, is preferably determined to be within a range from a lower limit θ2 to an upper limit θ1. This range of field of view θ can be determined based on the effective field of view of an average person. Here, the effective field of view of a person is a field of view at which a person can comfortably recognize an object without moving their head. Assuming that the effective field of view of an average person is 30 degrees, setting the upper limit θ1 to 30 degrees, which corresponds to the effective field of view of a person, allows the user to visually recognize the superimposed images 602 and 603 without changing their line of sight or posture while adjusting the wearing position of the HMD 101, making adjustment easier. The lower limit θ2 is set to a viewing angle at which the user can comfortably view the superimposed images 602 and 603. By setting the display size of the superimposed images 602 and 603 to a viewing angle θ between the lower limit θ2 and the upper limit θ1, the user can adjust the wearing position of the HMD 101 as if moving the target display 608 to match their own eyes reflected in a mirror.

[0055] A method for converting the display size will now be described in detail. The guide image generation unit 503 calculates the resolution of the display panel 203 included within the viewing angle θ set as described above, and converts the resolution of the superimposed images 602 and 603 to the calculated resolution. Furthermore, the guide image generation unit 503 generates a background image 609 with the same resolution as the overall resolution of the display panel 203 (hereinafter referred to as panel resolution), and arranges the converted superimposed images 602 and 603 so that their centers overlap the center of the background image 609. In this way, guide images 612 and 613 of an appropriate display size are generated. The center of the background image 609 is set to a position on the display panel 203 corresponding to the center of the line connecting the center of the left eyepiece 204L and the center of the right eyepiece 204R. By aligning the center of the background image 609 with the center of the superimposed image, the superimposed images 602 and 603 are displayed at the center of the user's line of sight. The resolution of the superimposed image can be calculated using the following equation 1.

[0056] Overlay image resolution = Panel resolution [pix] ÷ HMD viewing angle [degree] × viewing angle θ [degree] (1)

[0057] Thereafter, the guide image generation unit 503 creates a background image 609 that is a single color and has the same resolution as the panel resolution of the HMD 101. The converted superimposed images 602 and 603 are superimposed on the background image 609 so that the center of the created background image 609 coincides with the center of the converted superimposed images 602 and 603.

[0058] 6, the superimposed images 602 and 603 are generated so that the areas other than the eyepiece areas 604L and 604R are black. Also, a full-color background image 609 corresponding to the display panel 203 is generated in gray. Note that this example is not limiting and any color may be generated. Also, the example is not limited to the case where the resolution of the background image that forms the background of the guide images 612 and 613 is the resolution of the display panel 203, and the guide images may be of any size as long as the entire superimposed images 602 and 603 are displayed within the display panel 203.

[0059] Furthermore, the display form of the guide image is not limited to the example of Fig. 6. For example, a plate-shaped object with superimposed images 602 and 603 pasted in a three-dimensional space may be created using CG (computer graphics), and the three-dimensional space may be subjected to CG rendering to generate the guide image. In addition, the display form of the guide image is arbitrary.

[0060] FIG. 8 is a diagram illustrating an example of a guide image displayed in a three-dimensional space. The coordinates shown in FIG. 8 are assumed to be a virtual space (Vx, Vy, Vz) in which a user 801 of the HMD 101 is located. The guide image generation unit 503 sets the floor position where the user 801 is standing as the origin and places a plate-shaped object 802 at a position a predetermined distance from the origin. The plate-shaped object 802 is placed so that its plate surface is perpendicular to the line of sight of the user 801. The size of the plate-shaped object 802 is adjusted according to the position of the plate-shaped object 802 so that it falls within the above-mentioned viewing angle range θ1 to θ2. The guide image generation unit 503 pastes superimposed images 602 and 603 on the surface of the plate-shaped object 802 facing the user 801. In the virtual space created in this way, the guide image generation unit 503 places two virtual cameras corresponding to the left and right eyes of the user 801 and creates (renders) images viewed from the virtual cameras according to the resolution and viewing angle of the display panel 203 of the HMD 101. The image created in this way becomes the guide image in the example of Fig. 8. This is the same image generation method as when the HMD 101 generates an image that represents the VR space.

[0061] The guide image generating unit 503 outputs the generated guide image to the display control unit 504 .

[0062] In S704, the display control unit 504 converts the data of the guide image generated by the guide image generation unit 503 so that it looks appropriate when viewed on the HMD 101, and outputs the converted guide image data as display data to the HMD 101. The conversion process into display data includes color conversion process suitable for the display of the HMD 101, correction process for correcting distortion of the eyepiece 204 of the HMD 101, and the like. The HMD 101 displays the display data received from the display control unit 504 on the display panel 203.

[0063] The CPU 401 of the image processing device 100 repeats the processes of S701 to S704 until the user completes the adjustment of the wearing position of the HMD 101, and ends this flowchart when the user completes the adjustment of the wearing position of the HMD 101.

[0064] As described above, the image processing device 100 of the first embodiment acquires an image of the user's eyes captured by an imaging device provided in the HMD 101, and displays a guide image in real time in which a target display indicating a target position is superimposed on the image of the eyes. The guide image displays the user's own eye position and the target display on the same screen, making it easier for the user to grasp the positional relationship between the user's own eye position and the target position. This makes it easier to determine the direction and amount of adjustment to be made, and allows the user to easily adjust the wearing position of the HMD 101. The image processing device 100 of the present disclosure makes it easier for the user to wear the HMD 101 in the correct position.

[0065] In the first embodiment, an example was shown in which guide images 612 and 613 were generated by superimposing a target display 608 indicating a target position on an image 601 of the user's eye captured by the eye camera 207. However, the method of generating guide images is not limited to this. For example, a guide image may be generated by superimposing eye position information indicating the position of the user's eye on a reference image in which a target display is displayed at a specific position. This can be achieved by previously associating the positional relationship between the range of the reference image and the imaging range of the captured image. In addition, in the first embodiment, an example was shown in which the image of the user's eye included in the image captured by the eye camera 207 was displayed as eye position information on the guide image. However, the eye position information may also be displayed using a mark or the like. Then, the guide image generation unit 503 generates a guide image by drawing a mark indicating the eye at the acquired coordinate position in real time and drawing the target display. Furthermore, it is preferable that the mark indicating the position of the user's eye be represented using a mark of a different type from the mark used in the target display. Furthermore, it is preferable that a pseudo-eye image be displayed instead of a mark so that the two can be distinguished from each other.

[0066] Second Embodiment The image processing device 100 in the first embodiment displays, as a guide image, an image in which a target position is superimposed on an image of the user's eye. However, each user's eye has different characteristics. For example, the shape, size, and position of the left and right eyes vary from user to user. Due to such individual eye characteristics, a user may have difficulty perfectly aligning the position of their eye with the target position. Therefore, in the second embodiment, a process is added to correct the target position according to information indicating the characteristics of each user's eye (hereinafter referred to as feature information). Below, an overview of the correction process in the second embodiment and the significance of this correction process will be described.

[0067] 9(a) and 9(b) are diagrams showing guide images 901 and 902 created by the method of the first embodiment. The guide image 901 in FIG. 9(a) shows a state in which the positions of target displays 908L and 908R are aligned with the positions of the user's eyes 907R and 907L. On the other hand, the guide image 902 in FIG. 9(b) shows a state in which the positions of the user's eyes 907R and 907L are misaligned with the positions of the target displays 908L and 908R. More specifically, the vertical positions are aligned, but the horizontal positions are misaligned. The horizontal misalignment shown in FIG. 9(b) occurs due to the difference between the distance between the user's eyes (hereinafter referred to as interpupillary distance) and the distance between the left and right eyepieces 204 of the HMD 101 (hereinafter referred to as inter-lens distance). The distance between the user's eyes will be referred to as interpupillary distance in the following description. The distance between the left and right eyepieces 204 will be referred to as inter-lens distance in the following description. If the interpupillary distance of the user matches the inter-lens distance of the HMD 101, the user can align their eyes with the predetermined target position. On the other hand, if the interpupillary distance of the user does not match the inter-lens distance of the HMD 101, the user cannot freely change their interpupillary distance, and therefore the user cannot completely align their eyes with the target position.

[0068] Therefore, the image processing device 100 of the second embodiment corrects the target position based on the characteristic information of the user's eyes. Then, as shown in Fig. 9(c), a guide image 903 is generated by superimposing target displays 909L and 909R at the corrected target position on the image of the eyes acquired from the eye camera 207, and the guide image 903 is displayed on the display panel 203. This allows the user to easily determine whether the HMD 101 is worn in the correct position even if the inter-lens distance does not match the user's interpupillary distance.

[0069] FIG. 10 is a block diagram showing the functional configuration of an image processing device 100A according to the second embodiment. The image processing device 100A includes an eye position information acquisition unit 501, a feature information acquisition unit 1001, a correction unit 1002, a target position acquisition unit 502, a guide image generation unit 503, and a display control unit 504. The same components and processes as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and their descriptions are omitted. The hardware configurations of the image processing device 100A and the HMD 101 are the same as those of the image processing device 100 and the HMD 101 in the first embodiment. The image processing device 100A functions as each unit shown in FIG. 10 and executes a series of processes shown in the flowchart of FIG. 11 by the CPU 401 executing a program stored in the ROM 403 using the RAM 402 as a work memory. It should be noted that not all of the processes described below need to be executed by the CPU 401; the image processing device 100A may be configured so that part or all of the processes are performed by one or more processing circuits other than the CPU 401.

[0070] 10, differences from the first embodiment will be described. The eye position information acquisition unit 501 outputs the acquired image of the user's eye to the guide image generation unit 503, and also outputs it to the feature information acquisition unit 1001. The target position acquisition unit 502 outputs information on the target position not to the guide image generation unit 503 but to the correction unit 1002. The guide image generation unit 503 generates a guide image based on the information on the corrected target position acquired from the correction unit 1002.

[0071] The feature information acquisition unit 1001 analyzes the acquired image of the user's eyes to acquire feature information of the user's eyes. The feature information is, for example, the above-mentioned interpupillary distance. The feature information acquisition unit 1001 outputs the feature information to the correction unit 1002. A method for calculating the interpupillary distance will be described later.

[0072] The correction unit 1002 corrects the target position information acquired from the target position acquisition unit 502 in accordance with the characteristic information of each user's eye. Specifically, the correction unit 1002 corrects the target position information based on the pupil width information, which is the characteristic information acquired from the characteristic information acquisition unit 1001. The direction of correction is the horizontal direction. The amount of correction is the difference between the pupil width of the user and the distance between the left and right target positions. When making the correction, the correction unit 1002 corrects the target positions equally on the left and right. Alternatively, the correction unit 1002 may fix one of the left and right eyes and make the correction by shifting the target position of only the other eye. Details of the correction process will be described later.

[0073] Fig. 11 is a flowchart illustrating the processing executed by the image processing device 100A of the second embodiment. In the flowchart of Fig. 11, processing of S1101 to S1102 is added after S702 in the flowchart of the first embodiment (Fig. 7). The following mainly describes the processing that differs from Fig. 7.

[0074] In S701, the eye position information acquisition unit 501 acquires an image of the user's eyes from the eye camera 207, as in the first embodiment. The eye position information acquisition unit 501 generates an image of the eyes including the left and right eyes based on the acquired captured image, and outputs the image to the guide image generation unit 503 and the feature information acquisition unit 1001.

[0075] In S702, the target position acquisition unit 502, similar to the first embodiment, acquires information on the target position stored in the HDD 405. Next, the process proceeds to S1101.

[0076] In S1101, the feature information acquisition unit 1001 acquires feature information of the user's eyes from the image of the eyes acquired from the eye position information acquisition unit 501. As described above, the feature information acquisition unit 1001 acquires the interpupillary distance between the user's left and right eyes in the image of the eyes acquired from the eye position information acquisition unit 501 as the feature information of the eyes.

[0077] Here, a method for acquiring the interpupillary distance will be described. The feature information acquisition unit 1001 first acquires pixel coordinates of the representative positions of each of the user's left and right eyes in the eye image. As the representative positions of the eyes, for example, the pixel coordinates (x, y) of the centers of the pupils of the eyes in the eye image 601 are determined. The representative positions of the eyes are defined as follows:

[0078] Pixel coordinates of the representative position of the left eye = (xeye_L, yeye_L) Pixel coordinates of the representative position of the right eye = (xeye_R, yeye_R)

[0079] The feature information acquisition unit 1001 analyzes the acquired eye images and acquires pixel coordinates corresponding to the centers of the pupils, which are representative positions for each of the left and right eyes. Specifically, the feature information acquisition unit 1001 detects black, round areas included in the eye images through image processing. Then, the feature information acquisition unit 1001 acquires pixel coordinates corresponding to the representative positions of the eyes by calculating the center positions of the detected areas. Note that the method for acquiring pixel coordinates corresponding to the representative positions of the user's eyes is not limited to this, and known eye tracking technology may also be used. Furthermore, the representative positions of the eyes are not limited to the centers of the pupils and may be other parts. For example, the representative positions may be the center of the iris or positions indicating the end points of the eyes, such as the inner and outer corners of the eyes. Alternatively, the representative position of the eyes may be the center point of the line connecting the inner and outer corners of the eyes. Note that if the eye position information acquisition unit 501 has acquired pixel coordinates corresponding to the representative positions of the eyes, the feature information acquisition unit 1001 acquires the pixel coordinates corresponding to the representative positions of the eyes acquired by the eye position information acquisition unit 501 as eye position information.

[0080] The feature information acquisition unit 1001 calculates the distance between the pixel coordinates (xeye_L, yeye_L) of the representative position of the user's left eye and the pixel coordinates (xeye_R, yeye_R) of the representative position of the user's right eye. In the following description, the distance is assumed to be the Euclidean distance. Hereinafter, the calculated distance is referred to as the user's interpupillary distance W_user. The distance indicating the user's interpupillary distance W_user is expressed in units of pixels. The feature information acquisition unit 1001 outputs information about the user's interpupillary distance W_user to the correction unit 1002 as feature information.

[0081] In S1002, the correction unit 1002 corrects the target position acquired from the target position acquisition unit 502 based on the feature information of the user's eye acquired from the feature information acquisition unit 1001. Specific steps of the correction process will be described below with reference to FIG. 12. FIG. 12(a) shows an image in which target indicators 908L and 908R are superimposed on the image of the user's eye at the target position before correction. FIG. 12(b) shows an image in which target indicators 1208L and 1208R are superimposed on the image of the user's eye at the target position after correction.

[0082] The correction unit 1002 calculates the horizontal distance between the pixel coordinates (xtarget_L, ytarget_L) of the left eye 908L and the pixel coordinates (xtarget_R, ytarget_R) of the right eye 908R of the target position acquired from the target position acquisition unit 502. Hereinafter, this distance will be referred to as the left-right width W_target of the target position. The distance indicating the left-right width W_target of the target position is expressed in units of pixels.

[0083] Furthermore, the correction unit 1002 calculates the difference between the user's pupil width W_user and the left-right width W_target of the target position. Finally, the correction unit 1002 adds or subtracts the number of pixels Δd obtained by dividing the difference between the user's pupil width W_user and the left-right width W_target of the target position by 2 to or from the x-coordinate value of the pixel coordinates indicating the left and right target positions. This obtains the pixel coordinates of the corrected target positions.

[0084] If the user's eye width W_user is narrower than the left-right width W_target of the target position, Left corrected target position = (xtarget_L + Δd, ytarget_L) Corrected right target position = (xtarget_R-Δd, ytarget_R) If the user's eye width W_user is wider than the left-right width W_target of the target position, Corrected left target position = (xtarget_L-Δd, ytarget_L) Corrected right target position = (xtarget_R+Δd,ytarget_R) where Δd=|W_target-W_user| / 2

[0085] For the left eye, if the user's interpupillary width W_user is narrower than the left-right width W_target of the target position, the correction unit 1002 adds Δd to the corrected target position. In other words, it shifts to the right. If the user's interpupillary width W_user is wider than the left-right width W_target of the target position, the correction unit 1002 subtracts Δd. In other words, it shifts to the left. For the right eye, the addition and subtraction are reversed from the left eye. If the user's interpupillary width W_user is narrower than the left-right width W_target of the target position, the correction unit 1002 subtracts Δd and shifts to the left. If the user's interpupillary width W_user is wider than the left-right width W_target of the target position, the correction unit 1002 adds Δd and shifts to the right.

[0086] As a result of correcting the target positions in this manner, it is possible to correct the display positions of the target displays 1208L and 1208R in accordance with the characteristic information of the user's individual eyes, as shown in the guide image 903 in FIG. 9(c). Note that in this embodiment, the method of correcting the target positions involves correcting both eyes equally by Δd, but this is not limited to this. For example, it is also possible to correct only one of the left and right eyes without correcting the other. In this case, the correction amount is twice Δd, that is, |W_target-W_user|. Furthermore, the correction ratio between the left and right may be changed. However, it is considered that the appearance of AR images and VR images on the HMD 101 is better when the left and right eyes are corrected equally compared to when the left and right eyes are corrected unevenly.

[0087] The correction unit 1002 outputs information on the corrected target position to the guide image generation unit 503. The guide image generation unit 503 generates an image in which a target display indicating the corrected target position is superimposed on the eye image acquired from the eye position information acquisition unit 501, and outputs the image to the display control unit 504. The display control unit 504 outputs the guide image generated by the guide image generation unit 503 to the HMD 101 via the output I / F 407, and causes it to be displayed on the display panel 203 of the HMD 101. The method of generating a guide image in the guide image generation unit 503 and the display control method in the display control unit 504 are the same as those in the first embodiment.

[0088] As described above, according to the image processing device 100A of the second embodiment, the target position is corrected in accordance with the characteristics of the user's eyes, and a guide image is generated in which the corrected target position is superimposed on the image of the user's eyes. This allows a user whose interpupillary distance does not match the width of the left and right eyepieces 204L, 204R of the HMD 101 to easily determine whether or not the HMD 101 is being worn in the correct position.

[0089] 11, the feature information acquisition unit 1001 calculates the user's eye width W_user and the left-right width W_target of the target position each time an image of the user's eyes is acquired, but this is not a limitation. The user's eye width W_user and the left-right width W_target of the target position may be calculated only once at the start of the flowchart of FIG. 11, and the correction amount Δd may be determined to correct the target position. Furthermore, in the above description, the feature information acquisition unit 1001 determines the representative positions of the user's eyes from the captured image, but this is not a limitation. The eye position information acquisition unit 501 may determine the representative positions of the user's eyes from the captured image.

[0090] <Third embodiment> In addition to the processing of the second embodiment, the image processing device 100B of the third embodiment displays, as guide information, instruction information that is an instruction for the user to adjust the wearing position of the HMD 101 in accordance with the current eye position of the user.

[0091] FIG. 13 is a diagram illustrating a guide display in the third embodiment. FIG. 13 is a diagram illustrating an example in which instruction information is displayed on a guide image. A guide image 1301 shown in FIG. 13(a) illustrates a state in which the positions of the user's eyes 1307L and 1307R are both shifted downward relative to target displays 1308L and 1308R indicating the target positions. The guide image 1301 displays, as instruction information 1302, an adjustment method that the user should perform in this state. The instruction information 1302 is a text message that indicates the adjustment direction in words, such as "Please move the HMD down." A guide image 1305 shown in FIG. 13(b) illustrates a state in which the target display 1308L and the user's eye 1307L coincide with each other for the left eye, and the target display 1308R and the user's eye 1307R are shifted from each other for the right eye. The guide image 1305 displays, as instruction information 1303 and 1304, an adjustment method that the user should perform in this state. Specifically, instruction information 1303 for the left eye, "Do not move the left side," and instruction information 1304 for the right eye, "Push the right side down," are displayed.

[0092] In this way, the image processing device 100B of the third embodiment adds a target display indicating the target position to a guide image that shows the user's current eye position, and displays instruction information as guide information. This allows the user to efficiently adjust the wearing position of the HMD 101. Note that the instruction information is not limited to being expressed by the text message described above, and may be expressed in any way that allows the user to recognize the adjustment method. For example, a symbol such as an arrow may be used to indicate the adjustment direction of the HMD 101 so that the user can visually recognize it.

[0093] FIG. 14 is a block diagram showing the functional configuration of an image processing device 100B according to the third embodiment. The image processing device 100B includes an eye position information acquisition unit 501, a feature information acquisition unit 1001, a correction unit 1002, a target position acquisition unit 502, a guide image generation unit 503, an instruction information generation unit 1401, and a display control unit 504. The same components and processes as those in the first and second embodiments are denoted by the same reference numerals as those in the first and second embodiments, and their descriptions will be omitted. The hardware configurations of the image processing device 100B and the HMD 101 are the same as those of the image processing device 100 and the HMD 101 according to the first embodiment. The image processing device 100B functions as the components shown in FIG. 14 and executes a series of processes shown in the flowchart of FIG. 15 by the CPU 401 executing a program stored in the ROM 403 using the RAM 402 as a work memory. It is not necessary for all of the processes described below to be executed by the CPU 401, and the image processing device 100B may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 401.

[0094] 14, differences from the second embodiment will be described. The feature information acquisition unit 1001 outputs feature information of the user's eye to the correction unit 1002 and also to the instruction information generation unit 1401. The correction unit 1002 outputs information on the corrected target position to the guide image generation unit 503 and also to the instruction information generation unit 1401. The display control unit 504 acquires guide image data from the guide image generation unit 503, and acquires instruction information from the instruction information generation unit 1401.

[0095] The instruction information generation unit 1401 generates instruction information that is an instruction for the user to adjust the wearing position of the HMD 101. The instruction information may be an instruction for either the left eye or the right eye of the user, or may be an instruction for both eyes. The instruction information may also be information indicating that adjustment of the wearing position is not necessary. If adjustment of the wearing position is necessary, the instruction information includes information instructing the user in the direction in which to adjust the wearing position of the HMD 101. The instruction information may also be a text message, or an image or symbol such as an arrow indicating a direction. The instruction information may be written in Japanese or in another language such as English. A configuration may also be adopted in which the user selects the language. A method for generating instruction information will be described later.

[0096] Fig. 15 is a flowchart for explaining the processing executed by the image processing device 100B of the third embodiment. In the flowchart of Fig. 15, the processing of S1501 to S1502 is added after S703 in comparison with the flowchart of the second embodiment (Fig. 11). This flowchart will be explained focusing on the processing newly added in the third embodiment.

[0097] In S1501, the instruction information generation unit 1401 generates instruction information from information on the position of the user's eyes acquired from the feature information acquisition unit 1001 and information on the corrected target position acquired from the correction unit 1002. The generation of instruction information will be specifically described below. In the following description, the target position is the corrected target position.

[0098] First, the instruction information generation unit 1401 acquires the direction of pixel coordinates (xeye_L, yeye_L) indicating the position of the user's left eye, based on pixel coordinates (xtarget_L, ytarget_L) indicating the target position of the left eye. In this embodiment, the instruction information generation unit 1401 determines the adjustment direction based on which of four divided areas obtained by dividing the eye image horizontally and vertically based on the target position the user's eye is located in. The instruction information generation unit 1401 manages the relationship between the divided areas and the adjustment direction by assigning identification numbers from 1 to 4 to each of the four divided areas.

[0099] FIG. 16 is a diagram showing the correspondence between divided areas and identification numbers on the image plane of an eye image for one eye. In the example of FIG. 16, the identification number for the divided area in the upper right corner with respect to the pixel coordinates of the target position is set to 1, and identification numbers 2, 3, and 4 are assigned to each divided area clockwise from the target position. Note that when the target position information matches the position of the user's eye, it is managed as the identification number "5" indicating that they match. Although FIG. 16 shows the left eye, for the right eye, identification numbers 1 to 5 are similarly assigned to each divided area and target position with respect to the pixel coordinates of the target position of the right eye as the center, and the position of the user's right eye is managed.

[0100] Next, the instruction information generation unit 1401 acquires the direction of pixel coordinates (xeye_R, yeye_R) indicating the position of the user's right eye, based on the pixel coordinates (xtarget_R, ytarget_R) of the target position of the right eye, in the same way as for the left eye. The method of acquiring and managing the direction is the same as for the left eye.

[0101] Finally, the instruction information generation unit 503 selects instruction information for the left eye and the right eye from the instruction information candidates stored in the HDD 405 based on the acquired left eye direction and right eye direction. The instruction information candidates include information on whether adjustment is necessary and instruction information according to the adjustment direction. The instruction information candidates are managed by associating an identification number for identifying the direction with the instruction information.

[0102] FIG. 17 is a diagram showing examples of instruction information candidates. FIG. 17(a) shows instruction information candidates 1701 for the right eye, and FIG. 17(b) shows instruction information candidates 1711 for the left eye. As shown in FIG. 17(a), an identification number indicating a direction 1702 and instruction information for the right eye 1703 are associated with the instruction information candidates 1701 for the right eye. For example, instruction information 1704 saying "Lift your right side up" is associated with "1" in the upper right region and "4" in the upper left region based on the target position of the right eye. Instruction information 1705 saying "Lower your right side" is associated with "2" in the lower right region and "3" in the lower left region based on the target position of the right eye. Instruction information 1706 saying "Do not move your right side" is associated with identification number "5" when the target position for the right eye matches the position of the user's right eye.

[0103] Similarly, as shown in Fig. 17(b), a direction 1712 and left eye instruction information 1713 are associated with candidate instruction information for the left eye 1711. For example, instruction information 1714 saying "Lift your left side up" is associated with "1" in the upper right region and "4" in the upper left region based on the target position of the left eye. Instruction information 1715 saying "Lower your left side down" is associated with "2" in the lower right region and "3" in the lower left region based on the target position of the left eye. Instruction information 1716 saying "Do not move your left side" is associated with identification number "5" when the target position for the left eye matches the position of the user's left eye.

[0104] The instruction information generation unit 1401 outputs each of the left and right instruction information selected from the instruction information candidates to the display unit 304. Note that in this embodiment, the relationship between the user's eye position and the target position is divided into four regions. However, the number of regions is not limited to this, and the region may be divided into more regions to increase the number of guide information options. Furthermore, the region may be divided into two, such as upper and lower, or left and right, and corresponding guide information options may be prepared. Furthermore, the method of determining the user's eye position with respect to the target position is not limited to the above-described method. Any method may be used as long as it can determine the positional relationship between the target position and the user's eye position. Furthermore, the method of expressing the instruction information and the content of the message are not limited to the above-described example. Furthermore, the instruction information to be selected may be determined by a combination of a number indicating the left eye position and a number indicating the right eye position. For example, if the direction of the user's right eye position with respect to the target position for the right eye and the direction of the user's left eye position with respect to the target position for the left eye match, it is highly likely that adjustment in the left-right direction (horizontal direction) is necessary. For example, if the direction for the right eye is "3" and the direction for the left eye is "3," it can be estimated that both eyes are positioned to the lower left of the target position. This means that the HMD 101 is worn to the upper right of the user's actual eye position. Therefore, instruction information for both eyes, such as "Please move to the lower left," may be displayed. Such combination conditions and instruction information may be stored in association with each other in the HDD 405.

[0105] In S1502, the display control unit 504 outputs display data including the instruction information acquired from the instruction information generation unit 1401 and the guide image acquired from the guide image generation unit 503 to the HMD 101. The HMD 101 displays the display data on the display panel 203.

[0106] As described above, the image processing device 100B of the third embodiment generates instruction information for the user to adjust the position of the HMD 101 together with the guide image and displays the instruction information together with the guide image. This allows the user to easily understand the adjustment method and efficiently adjust the wearing position of the HMD 101.

[0107] Although the example of the instruction information is shown in which a text message or an arrow is displayed on the guide image, the instruction information is not limited to this. For example, the instruction may be given by voice. In this case, the CPU 401 of the image processing device 100 may generate voice data in which the text message generated by the instruction information generation unit 1401 is read aloud, and output the voice data from the speaker of the HMD 101.

[0108] Although preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. For example, not only all combinations of the first to third embodiments but also a combination of the first and third embodiments may be used. In other words, instruction information may be displayed together with a guide image without correcting the target position. It is clear that a person skilled in the art can conceive of various other modified or altered examples within the scope of the disclosed technical ideas, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0109] <Other embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0110] The disclosure of the above-described embodiment includes the following configurations.

[0111] (Configuration 1) an acquisition means for acquiring eye position information indicating the position of the eyes of a user wearing a head-mounted display device; a display control means for causing the display device to display guide information for the user to align the eyes together with the eye position information; An image processing device comprising:

[0112] (Configuration 2) the display device includes an imaging means for imaging the user's eye, 2. The image processing device according to configuration 1, wherein the acquisition means acquires the eye position information based on an image captured by the imaging means.

[0113] (Configuration 3) 3. The image processing device according to configuration 2, wherein the guide information includes a target display that is displayed at a target position that serves as an index for aligning the user's eyes.

[0114] (Configuration 4) 4. The image processing device according to configuration 3, further comprising storage means for storing in advance a specific position in the image captured by the imaging means as the target position.

[0115] (Configuration 5) 4. The image processing device according to configuration 3, wherein the display control means displays a superimposed image in which the target display is superimposed on the captured image captured by the imaging means.

[0116] (Configuration 6) 6. The image processing device according to any one of configurations 3 to 5, wherein the target position corresponds to a part of the display device.

[0117] (Configuration 7) 7. The image processing device according to configuration 6, wherein the imaging means captures, as the captured image, an image including the part of the display device and the user's eyes within an imaging range.

[0118] (Configuration 8) 8. The image processing device of claim 6, wherein the part of the display device is an eyepiece.

[0119] (Configuration 9) a correction unit that corrects the target position in accordance with characteristic information that indicates characteristics of the eyes of each of the users; 9. The image processing device according to any one of configurations 3 to 8, wherein the display control means displays the target display at the target position corrected by the correction means.

[0120] (Configuration 10) 10. The image processing device according to configuration 9, wherein the feature information is acquired based on the captured image captured by the imaging means.

[0121] (Configuration 11) 11. The image processing device according to configuration 9 or 10, wherein the feature information is the distance between the left eye and the right eye.

[0122] (Configuration 12) 12. The image processing device according to any one of configurations 9 to 11, wherein the correction means corrects the target positions equally on the left and right.

[0123] (Configuration 13) 13. The image processing device according to any one of configurations 1 to 12, wherein the guide information includes instruction information for the user to adjust the mounting position of the display device.

[0124] (Configuration 14) 14. The image processing device according to configuration 13, wherein the instruction information is generated for each of the left eye and the right eye.

[0125] (Configuration 15) 15. The image processing device according to claim 13, wherein the instruction information includes information instructing the user to adjust the mounting position of the display device.

[0126] (Configuration 16) 16. The image processing device according to any one of configurations 13 to 15, wherein the instruction information is a text message.

[0127] (Configuration 17) 17. The image processing device according to any one of configurations 1 to 16, wherein the display control means displays the guide information and the eye position information on the same screen.

[0128] (Configuration 18) 18. The image processing device according to configuration 17, wherein the display control means displays the screen at a display size that allows the entire screen to be viewed by the user while maintaining the user's posture.

[0129] (Configuration 19) 19. The image processing device according to configuration 18, wherein the display size is determined based on a human effective visual field.

[0130] (Configuration 20) an acquisition step of acquiring eye position information indicating the position of the eyes of a user wearing the head-mounted display device; a display control step of displaying guide information for the user to align their eyes on the display device together with the eye position information; An image processing method comprising:

[0131] (Configuration 21) A program for a computer to execute an image processing method, The image processing method includes: an acquisition step of acquiring eye position information indicating the position of the eyes of a user wearing the head-mounted display device; a display control step of displaying guide information for the user to align their eyes on the display device together with the eye position information; Programs including.

Claims

1. an acquisition means for acquiring eye position information indicating the position of the eyes of a user wearing a head-mounted display device; a display control means for causing the display device to display guide information for the user to align the eyes together with the eye position information; An image processing device comprising:

2. the display device includes an imaging means for imaging the user's eye, The image processing apparatus according to claim 1 , wherein the acquisition means acquires the eye position information based on an image captured by the imaging means.

3. The image processing device according to claim 2 , wherein the guide information includes a target display that is displayed at a target position that serves as an index for aligning the user's eyes.

4. 4. The image processing apparatus according to claim 3, further comprising a storage unit that stores in advance a specific position in the image captured by the imaging unit as the target position.

5. 5. The image processing apparatus according to claim 4, wherein the display control means displays a superimposed image in which the target display is superimposed on the image captured by the imaging means.

6. 4. The image processing device according to claim 3, wherein the target position corresponds to a part of the display device.

7. 7. The image processing device according to claim 6, wherein the imaging means captures, as the captured image, an image including the part of the display device and the user's eyes within an imaging range.

8. 7. The image processing apparatus according to claim 6, wherein the part of the display device is an eyepiece.

9. a correction unit that corrects the target position in accordance with characteristic information that indicates characteristics of the eyes of each of the users; 4. The image processing apparatus according to claim 3, wherein the display control means displays the target display at the target position corrected by the correction means.

10. 10. The image processing apparatus according to claim 9, wherein the feature information is acquired based on an image captured by the imaging means.

11. 11. The image processing device according to claim 10, wherein the feature information is a distance between the left eye and the right eye.

12. 10. The image processing apparatus according to claim 9, wherein the correcting means corrects the target positions equally on the left and right.

13. 13. The image processing device according to claim 1, wherein the guide information includes instruction information for the user to adjust the mounting position of the display device.

14. The image processing device according to claim 13 , wherein the instruction information is generated for each of the left eye and the right eye.

15. 14. The image processing device according to claim 13, wherein the instruction information includes information instructing the user to adjust the mounting position of the display device.

16. 14. The image processing device according to claim 13, wherein the instruction information is a text message.

17. 2. The image processing apparatus according to claim 1, wherein the display control means displays the guide information and the eye position information on the same screen.

18. 18. The image processing device according to claim 17, wherein the display control means displays the screen at a display size that allows the user to view the entire screen while maintaining the same posture.

19. 19. The image processing device according to claim 18, wherein the display size is determined based on a human effective visual field.

20. an acquisition step of acquiring eye position information indicating the position of the eyes of a user wearing the head-mounted display device; a display control step of displaying guide information for the user to align their eyes on the display device together with the eye position information; An image processing method comprising:

21. A program for a computer to execute an image processing method, The image processing method includes: an acquisition step of acquiring eye position information indicating the position of the eyes of a user wearing the head-mounted display device; a display control step of displaying guide information for the user to align their eyes on the display device together with the eye position information; Programs including.

Images