Information processing device and adjustment screen display method

Abstract

According to the present invention, an image generation device generates an adjustment screen 300 for causing a user who wears a head-mounted display to adjust the distance between lenses of the head-mounted display, and displays the adjustment screen 300 on the head-mounted display. On the adjustment screen 300, lens images (e.g., left lens image 304a, right lens image 304b) indicating lenses of the head-mounted display are arranged, and pupil images (e.g., left eye image 306a, right eye image 306b) indicating pupils of the user are further arranged thereon on the basis of a result of eye tracking.

Description

Technical Field

[0001] The present invention relates to data processing technology, and more particularly to an information processing apparatus and an adjustment screen display method.

Background Art

[0002] An image display system in which a user wearing a head-mounted display can view a target space from a free viewpoint has become widespread. For example, there is known an electronic content that realizes virtual reality (VR) by using a virtual three-dimensional space as a display target and displaying an image corresponding to the user's line-of-sight direction on the head-mounted display. By using a head-mounted display, it is possible to enhance the sense of immersion in a video or improve the operability of an application such as a game. In addition, a walk-through system has been developed in which a user wearing a head-mounted display can virtually walk around in a space displayed as a video by physically moving.

Summary of the Invention

Problems to be Solved by the Invention

[0003] When the distance between the left-eye lens and the right-eye lens provided in the head-mounted display (hereinafter also referred to as the "inter-lens distance") is not appropriately set, the display of the head-mounted display may appear blurred to the user. Therefore, the user needs to appropriately set the inter-lens distance of the head-mounted display.

[0004] The present invention has been made in view of such problems, and one object thereof is to provide a technique for assisting in setting the inter-lens distance of a head-mounted display.

Means for Solving the Problems

[0005] To solve the above problems, an information processing device according to one aspect of the present invention comprises an adjustment screen generation unit that generates an adjustment screen for a user wearing a head-mounted display to adjust the distance between the lenses of the head-mounted display, and a display control unit that displays the adjustment screen on the head-mounted display. The adjustment screen generation unit places lens images representing the lenses of the head-mounted display on the adjustment screen, and further places pupil images representing the user's pupils on the adjustment screen based on the results of eye tracking.

[0006] Another aspect of the present invention is a method for displaying an adjustment screen. This method involves a computer performing the steps of generating an adjustment screen for a user wearing a head-mounted display to adjust the interlens distance of the head-mounted display, and displaying the adjustment screen on the head-mounted display. The adjustment screen includes lens images representing the lenses of the head-mounted display, and further includes pupil images representing the user's pupils based on the results of eye tracking.

[0007] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0008] According to the present invention, it is possible to assist in setting the inter-lens distance of a head-mounted display. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the appearance of the head-mounted display in the embodiment. [Figure 2] This figure shows an example configuration of the image display system in the embodiment. [Figure 3] This is a diagram illustrating an example of the image world that an image generation device displays on a head-mounted display. [Figure 4]This is a diagram showing the internal circuit configuration of the image generation device. [Figure 5] This diagram shows the internal circuit configuration of a head-mounted display. [Figure 6] This is a block diagram showing the functional blocks of an image generation device. [Figure 7] This figure shows an example of an adjustment screen. [Figure 8] This is a flowchart showing the operation of the image generation device. [Figure 9] This figure shows an example of displacement detection by the displacement detection unit. [Figure 10] This figure schematically shows the adjustment screen corresponding to Figure 9. [Figure 11] This figure shows an example of displacement detection by the displacement detection unit. [Figure 12] This figure schematically shows the adjustment screen corresponding to Figure 11. [Figure 13] This figure shows an example of an adjustment screen for when the head-mounted display is tilted significantly. [Figure 14] This is a diagram showing the adjustment screen for modified examples. [Figure 15] Figures 15(a), 15(b), 15(c), and 15(d) show examples of feedback objects in the adjustment screen. [Modes for carrying out the invention]

[0010] This embodiment relates to an image display system that displays application images on a head-mounted display worn on the user's head. The head-mounted display is also called a VR headset. Figure 1 shows an example of the appearance of the head-mounted display 100 of this embodiment. The head-mounted display 100 comprises an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that wraps around the user's head when worn, securing the device in place.

[0011] The output mechanism 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and has a display panel inside that faces the eyes when worn. The display panel of the head-mounted display 100 in this embodiment is opaque. In other words, the head-mounted display 100 in this embodiment is a light-opaque head-mounted display.

[0012] The housing 108 further includes eyepieces (left lens 114 and right lens 116, described later) positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, thereby expanding the user's field of view. The head-mounted display 100 may also be equipped with speakers or earphones positioned to correspond to the user's ears when worn. The head-mounted display 100 also incorporates motion sensors to detect the translational and rotational movements of the user's head, as well as its position and orientation at each moment in time.

[0013] Furthermore, the head-mounted display 100 is equipped with a stereo camera 110 on the front of the housing 108. The stereo camera 110 captures video of the surrounding real space with a field of view corresponding to the user's gaze. By displaying the captured images immediately, it is possible to achieve so-called video see-through, where the user can see exactly what is happening in the real space in the direction they are facing. Furthermore, by drawing virtual objects on top of the images of real objects captured in the images, augmented reality (AR) can be realized. There is no limit to the number of cameras that the image display system 10 can have; the head-mounted display 100 may have one camera or three or more cameras.

[0014] Furthermore, the head-mounted display 100 is equipped with an adjustment dial 112 on the top of the housing 108. The adjustment dial 112 is a component for adjusting the distance between the lenses of the head-mounted display 100. The user can lengthen or shorten the distance between the lenses of the head-mounted display 100 by turning the adjustment dial 112.

[0015] FIG. 2 shows a configuration example of the image display system 10 of the embodiment. The image display system 10 includes a head-mounted display 100, an image generation device 200, and a controller 140. The head-mounted display 100 is connected to the image generation device 200 by wireless communication. The image generation device 200 may be further connected to a server (not shown) via a network. In that case, the server may provide data of an online application such as a game in which a plurality of users can participate via the network to the image generation device 200.

[0016] The image generation device 200 is an information processing device that identifies the position of the viewpoint and the direction of the line of sight based on the position and posture of the head of the user wearing the head-mounted display 100, and generates a display image so as to have a corresponding field of view and outputs it to the head-mounted display 100. The image generation device 200 may be a stationary game machine, a PC, or a tablet terminal. The image generation device 200 can execute various applications related to VR and AR. In the embodiment, the image generation device 200 progresses an electronic game (hereinafter also referred to as a "VR game") and generates a display image of a virtual world that is the stage of the game, and displays the display image on the head-mounted display 100. [[ID=?]]

[0017] Note that the image generation device 200 may generate a moving image for viewing or information provision regardless of whether it is a virtual world or the real world, and display the moving image on the head-mounted display 100. Further, the image generation device 200 may display a panoramic image with a wide angle of view centered on the user's viewpoint on the head-mounted display 100, thereby giving the user a deep sense of immersion in the displayed world.

[0018] It seems there is a missing ID in the original text for the line starting with "

[0017] ". I've left it as is in the translation for now.The controller 140 is an input device (e.g., a game controller) held in the user's hand, into which user operations are input. User operations include operations to control image generation in the image generation device 200 and operations to control image display in the head-mounted display 100. The controller 140 is connected to the image generation device 200 by wireless communication and transmits data indicating user operations to the image generation device 200. As a variation, either or both of the head-mounted display 100 and the controller 140 may be connected to the image generation device 200 by wired communication via a signal cable or the like.

[0019] Figure 3 illustrates an example of an image world displayed on the head-mounted display 100 by the image generation device 200. In this example, a state is created in which the user 12 is in a virtual room. As shown in the figure, objects such as walls, floors, windows, tables, and objects on the tables are placed in the world coordinate system that defines the virtual space. The image generation device 200 defines a view screen 14 in the world coordinate system according to the position and direction of the user 12's viewpoint, and renders the display image by representing the images of the objects on it.

[0020] The image generation device 200 acquires the position and direction of the user 12's viewpoint (hereinafter, these may be collectively referred to as "viewpoint") from the head-mounted display 100 at a predetermined rate, and changes the position and direction of the view screen 14 accordingly. This allows the image to be displayed on the head-mounted display 100 within the field of view corresponding to the user's viewpoint. Furthermore, the image generation device 200 can generate a stereo image with parallax and display the stereo image in the left and right areas of the display panel of the head-mounted display 100, allowing the user 12 to experience the virtual space in 3D. This allows the user 12 to experience virtual reality as if they were actually in a room in the displayed world.

[0021] Figure 4 shows the internal circuit configuration of the image generation device 200. The image generation device 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0022] The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and also receives data input from the controller 140. The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory.

[0023] The CPU 222 controls the entire image generation device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., VR game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. Either the CPU 222 or the GPU 224, or both, can also be referred to as processors. The main memory 226 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing.

[0024] Figure 5 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 120, main memory 122, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, an eye-tracking sensor 136, and a stereo camera 110.

[0025] The CPU 120 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the image generation device 200 to the display unit 124 and the audio output unit 126. The main memory 122 stores the programs and data necessary for processing by the CPU 120.

[0026] The display unit 124 includes a display panel such as an LCD panel or an organic EL panel, and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 achieves stereoscopic vision by displaying a pair of stereo images on a display panel for the left eye located in front of the user's left eye and a display panel for the right eye located in front of the user's right eye.

[0027] The display unit 124 further includes a pair of lenses positioned between the display panel and the user's eyes when the head-mounted display 100 is worn, thereby expanding the user's field of view. The pair of lenses includes a left lens 114 and a right lens 116. The left lens 114 is positioned between the display panel for the left eye and the user's left eye, and the right lens 116 is positioned between the display panel for the right eye and the user's right eye. An adjustment dial 112 is mechanically or electrically connected to the left lens 114 and the right lens 116 to adjust the distance between the lenses. The distance between the lenses is, for example, the distance from the center of the left lens 114 to the center of the right lens 116.

[0028] The audio output unit 126 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, allowing the user to hear audio. The communication unit 132 is an interface for sending and receiving data with the image generation device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®.

[0029] The motion sensor 134 includes a gyroscope and an accelerometer to acquire the angular velocity and acceleration of the head-mounted display 100. The eye-tracking sensor 136 is a known sensor for eye tracking. Eye tracking, also known as gaze measurement, is a technique that detects the position, movement, and gaze direction of the user's pupils (also known as eyeballs). For example, the eye-tracking sensor 136 detects the position and movement of the user's pupils using infrared light or the like.

[0030] As shown in Figure 1, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints, with a field of view corresponding to the user's viewpoint. The images of the user's surrounding space captured by the stereo camera 110 will hereafter be referred to as "camera images." Camera images can be said to be images that capture the real space in the direction of the user's line of sight (typically in front of the user), and can also be said to be images that capture objects that exist in the direction of the user's line of sight.

[0031] The data transmitted from the head-mounted display 100 to the image generation device 200 via the communication unit 132 includes the following: (1) Measurement values ​​from motion sensor 134. (2) Measurement values ​​from the eye-tracking sensor 136. (3) Data of images (camera images) captured by the stereo camera 110. (4) The amount of rotation and the angle of rotation of the adjustment dial 112.

[0032] The features of the image display system 10 of this embodiment will now be described. The image display system 10 provides an adjustment screen, which is a user interface for a user wearing the head-mounted display 100 to adjust the distance between the lenses of the head-mounted display 100. The adjustment screen displays lens images showing the left lens 114 and the right lens 116 of the head-mounted display 100, based on the orientation of the head-mounted display 100. The adjustment screen also displays pupil images showing the user's pupils (left eye and right eye), based on the results of eye tracking. This helps to facilitate the user's adjustment of the distance between the lenses of the head-mounted display 100.

[0033] Figure 6 is a block diagram showing the functional blocks of the image generation device. The image generation device 200 performs various information processing tasks, such as managing the progress of the VR game and communicating with the server. However, the following description will mainly focus on the functional blocks related to adjusting the inter-lens distance of the head-mounted display 100.

[0034] The multiple functional blocks shown in Figure 6 can be implemented in hardware using the configuration shown in Figure 4, such as the CPU 222, GPU 224, main memory 226, and storage unit 234, and in software using a computer program that implements the functions of the multiple functional blocks. Therefore, it will be understood by those skilled in the art that these functional blocks can be implemented in various ways using hardware alone, software alone, or a combination thereof, and are not limited to any one of these.

[0035] The image generation device 200 includes a data processing unit 250 and a data storage unit 252. The data storage unit 252 corresponds to the storage unit 234 in Figure 4 and stores data that is referenced or updated by the data processing unit 250. For example, the data storage unit 252 stores image data for each element arranged on the adjustment screen, which will be described later in relation to Figure 7, etc.

[0036] The data processing unit 250 performs various data processing operations. The data processing unit 250 transmits and receives data to and from the head-mounted display 100 and the controller 140 via the communication unit 232, output unit 236, and input unit 238 shown in Figure 4. For example, the data processing unit 250 acquires camera images and sensor data transmitted from the head-mounted display 100 and data related to user operations transmitted from the controller 140.

[0037] The data processing unit 250 includes a system unit 260, an App execution unit 262, and a display control unit 264. The functions of the multiple functional blocks included in the data processing unit 250 may be implemented in a computer program. The processor of the image generation device 200 (e.g., CPU 222 and GPU 224) may perform the functions of the multiple functional blocks by reading the computer program stored in the storage of the image generation device 200 (e.g., storage unit 234) into the main memory 226 and executing it.

[0038] The App execution unit 262 reads data related to the application selected by the user (a VR game in this embodiment) from the data storage unit 252 and executes the application selected by the user. Based on the camera image acquired by the system unit 260, the position and orientation of the head-mounted display 100 acquired by the system unit 260, and the user's gaze direction measured by the system unit 260, the App execution unit 262 generates a VR image showing the result of executing the VR game. The VR image includes an image for the left eye and an image for the right eye.

[0039] The display control unit 264 transmits various VR image data generated by the App execution unit 262 to the head-mounted display 100, causing the VR images to be displayed on the display unit 124 of the head-mounted display 100. The display unit 124 of the head-mounted display 100 displays the left-eye image on the left-eye display panel and the right-eye image on the right-eye display panel.

[0040] The system unit 260 performs system processing related to the head-mounted display 100. The system unit 260 provides common services to multiple applications for the head-mounted display 100 (e.g., multiple VR games). Common services include, for example, providing camera images, providing position and orientation information of the head-mounted display 100, and providing eye-tracking results. The system unit 260 also performs processing related to the basic settings of the head-mounted display 100, and in this embodiment, it performs processing to assist in adjusting the inter-lens distance.

[0041] The system unit 260 includes a lens distance acquisition unit 272, a gaze measurement unit 276, a misalignment detection unit 278, and an adjustment screen generation unit 280.

[0042] The lens-to-lens distance acquisition unit 272 acquires the lens-to-lens distance of the head-mounted display 100 based on the amount or angle of rotation of the adjustment dial 112 transmitted from the head-mounted display 100.

[0043] The gaze measurement unit 276 uses known eye-tracking technology to detect the position, movement, and gaze direction of the user wearing the head-mounted display 100, based on the detection values ​​of the eye-tracking sensor 136 of the head-mounted display 100.

[0044] The misalignment detection unit 278 detects the magnitude of the misalignment between the value detected by the eye-tracking sensor 136 of the head-mounted display 100 and the position of the user's pupils detected by the gaze measurement unit 276.

[0045] The adjustment screen generation unit 280 generates data for an adjustment screen that allows the user to adjust the inter-lens distance of the head-mounted display 100. As will be described later in relation to Figure 7, the adjustment screen generation unit 280 places images representing the lenses of the head-mounted display 100 on the adjustment screen. In addition, the adjustment screen generation unit 280 places images representing the user's pupils on the adjustment screen based on the results of eye tracking by the gaze measurement unit 276.

[0046] The adjustment screen generation unit 280 outputs the generated adjustment screen data to the display control unit 264. The display control unit 264 transmits the adjustment screen data generated by the adjustment screen generation unit 280 to the head-mounted display 100, causing the adjustment screen to be displayed on the display unit 124 of the head-mounted display 100.

[0047] Figure 7 shows an example of the adjustment screen 300. The adjustment screen 300 is a user interface that makes the user wearing the head-mounted display 100 appear as if they are looking at their own reflection in a mirror. The adjustment screen generation unit 280 places a left eye image 306a, representing the user's left eye, at a position on the adjustment screen 300 corresponding to the position of the user's left eye detected by the gaze measurement unit 276. The adjustment screen generation unit 280 places a right eye image 306b, representing the user's right eye, at a position on the adjustment screen 300 corresponding to the position of the user's right eye detected by the gaze measurement unit 276. Hereinafter, the left eye image 306a and the right eye image 306b will be collectively referred to as pupil images.

[0048] The adjustment screen generation unit 280 places an HMD image 302 representing the head-mounted display 100 on the adjustment screen 300. The HMD image 302 includes a left lens image 304a representing the left lens 114 and a right lens image 304b representing the right lens 116. The left lens image 304a and the right lens image 304b may be images that appear as if the portions corresponding to the left lens 114 and right lens 116 in the HMD image 302 have been cut out. When the position or orientation of the HMD image 302 changes, the positions of the left lens image 304a and the right lens image 304b also change. Note that the positions of each element on the adjustment screen 300 are reversed left and right to make them appear as if they are reflected in a mirror. Hereafter, when referring to the left lens image 304a and the right lens image 304b collectively, they will simply be called the lens image.

[0049] When the user changes the distance between the left lens 114 and the right lens 116 using the adjustment dial 112, the adjustment screen generation unit 280 updates the adjustment screen 300 to widen or narrow the gap between the left lens image 304a and the right lens image 304b. The inter-lens distance indicator 308 is a pair of objects that indicate the magnitude of the inter-lens distance. When the inter-lens distance is changed, the adjustment screen generation unit 280 updates the adjustment screen 300 to widen or narrow the gap between the inter-lens distance indicator 308 in conjunction with the left lens image 304a and the right lens image 304b.

[0050] In this embodiment, the best position for the left lens 114 and the right lens 116 is where the center of the left lens 114 coincides with the center of the user's left eye, and the center of the right lens 116 coincides with the center of the user's right eye. The adjustment screen 300 is configured to prompt the user to move the left lens 114 and the right lens 116 closer to the best position.

[0051] Specifically, the size of the left lens image 304a is designed such that, if the difference between the center of the left lens image 304a and the center of the left eye image 306a is within a predetermined threshold, the entire left eye image 306a fits within the circle of the left lens image 304a. In other words, if the above difference exceeds the threshold, the size of the left lens image 304a is designed such that at least a portion of the left eye image 306a extends beyond the circle of the left lens image 304a (and is hidden behind the HMD image 302 on the screen).

[0052] Similarly, the size of the right lens image 304b is designed such that the entire right eye image 306b fits within the circle of the right lens image 304b, provided that the deviation between the center of the right lens image 304b and the center of the right eye image 306b is within a predetermined threshold. In other words, the size of the right lens image 304b is designed such that if the above deviation exceeds the threshold, at least a portion of the right eye image 306b extends beyond the circle of the right lens image 304b (is hidden behind the HMD image 302 on the screen). The above threshold for deviation may be determined by the developer's knowledge or by experiments using the image display system 10. In this embodiment, the threshold is ±3 millimeters for both the left and right eyes.

[0053] Furthermore, the adjustment screen 300 is set to a normal range 312 (shown by a dotted line in Figure 7) that indicates the range of the correct position of the HMD image 302. The normal range 312 can also be said to be the range of the correct wearing position of the head-mounted display 100. The adjustment screen generation unit 280 may set the normal range 312 based on the positions of the left eye image 306a and the right eye image 306b. The size of the normal range 312 may be designed so that if the deviation between the center of the lens image and the center of the pupil image exceeds a predetermined threshold (e.g., ±3 millimeters), a part of the HMD image 302 deviates from the normal range 312. Alternatively, the size of the normal range 312 may be designed so that if the deviation between the alignment of the left and right lens images and the alignment of the left and right pupil images exceeds a predetermined threshold (e.g., 3 degrees), a part of the HMD image 302 deviates from the normal range 312.

[0054] Furthermore, the adjustment screen generation unit 280 may determine that the position of the user's left pupil is within the appropriate range if the center of the user's left pupil detected by the gaze measurement unit 276 is located within a circle with a radius of approximately 3 millimeters from the center of the left lens 114 of the head-mounted display 100. Similarly, the adjustment screen generation unit 280 may determine that the position of the user's right pupil is within the appropriate range if the center of the user's right pupil detected by the gaze measurement unit 276 is located within a circle with a radius of approximately 3 millimeters from the center of the right lens 116 of the head-mounted display 100.

[0055] The adjustment screen generation unit 280 places the correct / incorrect examples 310 on the adjustment screen 300. The upper part of the correct / incorrect examples 310 displays an image showing an example of the correct positional relationship between the left lens image 304a, the right lens image 304b, the left eye image 306a, and the right eye image 306b. The lower part of the correct / incorrect examples 310 displays an image showing an example of the incorrect positional relationship between the left lens image 304a, the right lens image 304b, the left eye image 306a, and the right eye image 306b. The lower part of the correct / incorrect examples 310 shows an example where the distance between the lenses is too wide.

[0056] On the adjustment screen 300, the adjustment is complete when the left eye image 306a is within the circle of the left lens image 304a and the right eye image 306b is within the circle of the right lens image 304b. Once the adjustment is complete, the user selects (presses) the exit button 314. Alternatively, the exit button 314 may be hidden on the initial adjustment screen 300, and the adjustment screen generation unit 280 may display the exit button 314 when the position and orientation of the head-mounted display 100 and the position of the user's pupils have been correctly adjusted.

[0057] The operation of the image generation device 200 with the above configuration will now be explained. Figure 8 is a flowchart showing the operation of the image generation device 200. Figure 8 shows the operation of the lens distance adjustment support process that is executed when a user wearing the head-mounted display 100 uses the controller 140 to select a menu for adjusting the lens distance from among the multiple setting menus of the head-mounted display 100 provided by the image generation device 200.

[0058] The lens-to-lens distance acquisition unit 272 of the image generation device 200 acquires the lens-to-lens distance of the head-mounted display 100 based on the amount of rotation and the rotation angle of the adjustment dial 112 of the head-mounted display 100 (S10).

[0059] The gaze measurement unit 276 of the image generation device 200 detects the position, movement, and gaze direction of the user's pupils when wearing the head-mounted display 100, based on the values ​​measured by the eye-tracking sensor 136 of the head-mounted display 100 (S11). In S11, the misalignment detection unit 278 of the image generation device 200 detects the misalignment between the position of the lenses of the head-mounted display 100 and the position of the user's pupils, specifically detecting the magnitude of the misalignment between the alignment of the left lens 114 and the right lens 116 of the head-mounted display 100 and the alignment of the user's left and right pupils.

[0060] The adjustment screen generation unit 280 of the image generation device 200 generates adjustment screen data (S12) based on the inter-lens distance of the head-mounted display 100 acquired in S10, the position of the user's pupils measured in S11, and the discrepancy between the position of the lenses of the head-mounted display 100 and the position of the user's pupils detected in S12. The display control unit 264 of the image generation device 200 displays the adjustment screen on the head-mounted display 100 (S13).

[0061] The user, while viewing the adjustment screen 300 displayed on the head-mounted display 100, adjusts the position and orientation (or fit) of the head-mounted display 100 so that the left eye image 306a is within the circle of the left lens image 304a and the right eye image 306b is within the circle of the right lens image 304b, and also turns the adjustment dial 112 on the head-mounted display 100. Once the user has finished adjusting the distance between the lenses, they select the exit button 314 on the adjustment screen 300.

[0062] When the exit button 314 on the adjustment screen 300 is selected (Y in S14), the display control unit 264 terminates the display of the adjustment screen 300, and the image generation device 200 terminates the inter-lens distance adjustment support process. If the exit button 314 is not selected (N in S14), the process returns to S10. As the processes in S10 to S13 are repeated, the adjustment screen generation unit 280 sequentially updates the display content of the adjustment screen 300 in accordance with changes in the position and orientation of the head-mounted display 100, changes in the inter-lens distance, etc.

[0063] For example, if the position or orientation of the head-mounted display 100 changes, the misalignment detection unit 278 detects the misalignment between the position of the lenses of the head-mounted display 100 after the change and the position of the user's pupils. In other words, it detects the positional relationship between the position of the lenses of the head-mounted display 100 after the change and the position of the user's pupils. The adjustment screen generation unit 280 generates a new adjustment screen 300 by changing the positions of the left lens image 304a and the right lens image 304b according to the misalignment (positional relationship) between the position of the lenses and the pupils, which is sequentially detected by the misalignment detection unit 278. Furthermore, if the user performs an operation to change the position of the lenses (i.e., rotates the adjustment dial 112), the adjustment screen generation unit 280 generates a new adjustment screen 300 by changing the positions of the left lens image 304a and the right lens image 304b.

[0064] Figure 9 shows an example of misalignment detection by the misalignment detection unit 278. In this embodiment, the misalignment detection unit 278 maps the position of the user's left eye (specifically, the center position of the pupil) detected by the gaze measurement unit 276 to the left lens region 320a corresponding to the left lens 114. The misalignment detection unit 278 also maps the position of the user's right eye detected by the gaze measurement unit 276 to the right lens region 320b corresponding to the right lens 116. The misalignment detection unit 278 sets an LSD (Lens Separation Distance) line 322 connecting the center of the left lens region 320a and the center of the right lens region 320b, and an IPD (Inter Pupillary Distance) line 324 connecting the center of the user's left pupil and the center of the user's right pupil. In the example in Figure 9, the user's left and right pupils are detected below their actual positions.

[0065] Figure 10 schematically shows the adjustment screen 300 corresponding to Figure 9. In the adjustment screen 300, the positions of the left eye image 306a and the right eye image 306b are fixed. That is, in the adjustment screen 300, the position and tilt change only in the HMD image 302 (including the left lens image 304a and the right lens image 304b). Therefore, in the adjustment screen 300 of Figure 9, it is displayed that the HMD image 302 is shifted upward from its original position. Note that in the adjustment screen 300 of Figure 9, the LSD line 322 and IPD line 324 are not displayed.

[0066] Figure 11 shows an example of misalignment detection by the misalignment detection unit 278. In the example in Figure 11, the user's left and right pupils are detected as tilted upward to the right.

[0067] Figure 12 schematically shows the adjustment screen 300 corresponding to Figure 11. In the adjustment screen 300 of Figure 12, the HMD image 302 is displayed tilted upwards to the left.

[0068] Figure 13 shows an example of the adjustment screen 300 when the head-mounted display 100 is tilted significantly. The adjustment screen generation unit 280 of the image generation device 200 displays content on the adjustment screen 300 indicating the misalignment when the magnitude of the misalignment between the position of the lenses of the head-mounted display 100 and the position of the user's pupils exceeds a predetermined threshold.

[0069] Specifically, the misalignment detection unit 278 detects the angle between the LSD line 322 and the IPD line 324 as the magnitude of the misalignment between the position of the lens of the head-mounted display 100 and the position of the user's pupil. As shown in Figure 13, if the angle between the LSD line 322 and the IPD line 324 exceeds a predetermined threshold, the adjustment screen generation unit 280 further displays the LSD line 322 and the IPD line 324 on the adjustment screen 300 as content suggesting misalignment. That is, the adjustment screen generation unit 280 visualizes the LSD line 322 and the IPD line 324 set by the misalignment detection unit 278 on the adjustment screen 300. The above threshold for misalignment may be determined by the developer's knowledge or by experiments using the image display system 10. In this embodiment, the threshold is 3 degrees.

[0070] According to the image generation device 200 of the embodiment, by providing the user wearing the head-mounted display 100 with an adjustment screen 300 including lens images and pupil images, it is possible to assist in setting the appropriate inter-lens distance of the head-mounted display 100. Furthermore, if the user changes the lens position of the head-mounted display 100, the position of the lens image on the adjustment screen 300 can be changed to effectively assist in setting the appropriate inter-lens distance.

[0071] Furthermore, according to the image generation device 200 of the embodiment, if the posture of the head-mounted display 100 changes and the positional relationship between the lenses of the head-mounted display 100 and the user's pupils changes, the position of the lens image on the adjustment screen 300 is changed. This helps to support appropriate adjustment of the posture of the head-mounted display 100. In addition, if the discrepancy between the lens position of the head-mounted display 100 and the position of the user's pupils becomes large, the adjustment screen 300 displays content suggesting that the discrepancy is large, thereby prompting the user to correct the discrepancy between the lens position and the pupil position. Furthermore, by placing correct and incorrect examples 310 on the adjustment screen 300, it is possible to effectively support the setting of an appropriate inter-lens distance and the appropriate adjustment of the posture of the head-mounted display 100.

[0072] The present invention has been described above based on examples. These examples are illustrative, and it will be understood by those skilled in the art that various modifications are possible for each component or combination of each processing process, and that such modifications also fall within the scope of the present invention.

[0073] A modified example will be described. Although not mentioned in the above embodiment, the adjustment screen generation unit 280 of the image generation device 200 may set content on the adjustment screen 300 that suggests the inter-lens distance in the head-mounted display 100 is appropriate if the position of the user's pupils relative to the position of the lenses of the head-mounted display 100 remains within an appropriate range for a predetermined time threshold or longer. The above time threshold may be determined by the developer's knowledge or by experimentation using the image display system 10. Hereinafter, it will be set to 1.3 seconds.

[0074] The adjustment screen generation unit 280 may determine that the position of the user's pupils is within the appropriate range relative to the lens position if the magnitude of the deviation between the position of the lenses of the head-mounted display 100 and the position of the user's pupils, as sequentially detected by the deviation detection unit 278, is less than or equal to a predetermined threshold (e.g., ±3 millimeters). Alternatively, the adjustment screen generation unit 280 may determine that the position of the user's pupils is within the appropriate range if the center of the user's pupils, as detected by the gaze measurement unit 276, is located within a circle with a radius of approximately 3 millimeters from the center of the lenses of the head-mounted display 100. The adjustment screen generation unit 280 may perform this determination for each of the user's left and right pupils.

[0075] Figure 14 shows the adjustment screen for a modified example. In the example showing the correct positional relationship, located in the upper section of the correct / incorrect example 310, the contour of the lens image 330 is set to a manner that suggests the inter-lens distance of the head-mounted display 100 is appropriate. The manner that suggests the inter-lens distance is appropriate may be, for example, the same contour as the left lens image 304a and right lens image 304b in Figure 15(d) described later, or it may be a special color such as blue. Furthermore, the manner that suggests the inter-lens distance is appropriate may be the same as the second manner among the multiple manners of the feedback object 334 described later, which suggests the inter-lens distance of the head-mounted display 100 is appropriate. In the example showing the incorrect positional relationship, located in the lower section of the correct / incorrect example 310, the contour of the lens image 332 is set to a normal manner.

[0076] Figures 15(a)-(d) show examples of how feedback objects are displayed on the adjustment screen. When the adjustment screen generation unit 280 detects that the positions of both the user's left and right pupils are within the appropriate range, it starts measuring the time that the user's left and right pupils are within the appropriate range. Hereinafter, when it is stated that the user's pupil position is within the appropriate range, it means that both the user's left and right pupils are within the appropriate range. Also, when it is stated that the user's pupil position has deviated from the appropriate range, it means that at least one of the user's left or right pupils has deviated from the appropriate range.

[0077] As shown in Figure 15(a), the adjustment screen generation unit 280 does not provide feedback to the user for 0.3 seconds after detecting that the user's pupil position is within the correct range; in other words, it does not display the feedback object described later on the adjustment screen 300. Here, a 0.3-second delay is provided as a mitigation measure to prevent the rendering process of the feedback object 334 (described later) from becoming hectic and hindering user understanding when the user's pupil position is at the very edge of the correct range, causing frequent fluctuations between correct and deviated states.

[0078] As shown in Figure 15(b), the adjustment screen generation unit 280 detects that the user's pupil position is within the correct range, and if the user's pupil position remains within the correct range 0.3 seconds later, it starts drawing the feedback object 334. The feedback object 334 is drawn along the contours of the left lens image 304a and the right lens image 304b, and is set to a first form that is more emphasized than the contours of the left lens image 304a and the right lens image 304b. For example, if the contours of the left lens image 304a and the right lens image 304b are thin, low-luminance gray lines, the first form of the feedback object 334 may be a thick, high-luminance white line.

[0079] As shown in Figure 15(c), the adjustment screen generation unit 280 draws the feedback object 334 so that it completes one revolution around the contours of the left lens image 304a and the right lens image 304b in one second. If the adjustment screen generation unit 280 detects that the user's pupil position has deviated from the correct range before the feedback object 334 completes one revolution around the contours of the left lens image 304a and the right lens image 304b, it erases the feedback object 334 from the adjustment screen 300 and returns the adjustment screen 300 to the state shown in Figure 15(a). At this time, the time that the user's pupil was within the correct range, which had been measured up to that point, is also reset.

[0080] As shown in Figure 15(d), the adjustment screen generation unit 280 sets the feedback object 334 to a second state when the feedback object 334 completes one circuit around the contours of the left lens image 304a and the right lens image 304b, or in other words, when the user's pupil position remains within the appropriate range for 1.3 seconds or longer. The second state is a more emphasized state than the first state, indicating that the inter-lens distance of the head-mounted display 100 is appropriate. For example, if the first state of the feedback object 334 is a thick white line, the second state may be a very thick, high-brightness blue line.

[0081] If the adjustment screen generation unit 280 detects that the user's pupil position has deviated from the appropriate range while the feedback object 334 is displayed in the second mode, it erases the feedback object 334 from the adjustment screen 300 and returns the adjustment screen 300 to the state shown in Figure 15(a). At this time, the time that the user's pupils have been within the appropriate range, which has been measured up to that point, is also reset.

[0082] When the user confirms that the feedback object 334 on the adjustment screen 300 is displayed in the second mode, they select (press) the finish button 314 on the adjustment screen 300 to finish adjusting the inter-lens distance. According to this modified version, by providing the user with visual feedback (feedback object 334) when adjusting the inter-lens distance, it is possible to help the user intuitively and correctly determine whether or not the inter-lens distance is appropriate.

[0083] Let's describe another variation. If the user closes their eyes, for example, the gaze measurement unit 276 of the image generation device 200 may not detect the position of at least one of the user's left and right eyes. If the position of the user's left eye is not detected, the adjustment screen generation unit 280 may place only the right eye image 306b on the adjustment screen 300, and if the position of the user's right eye is not detected, it may place only the left eye image 306a on the adjustment screen 300. Also, if the positions of both the user's left and right eyes are not detected, the adjustment screen generation unit 280 may not place either the left eye image 306a or the right eye image 306b on the adjustment screen 300. If the adjustment screen generation unit 280 does not place at least one of the left eye image 306a or the right eye image 306b on the adjustment screen 300, it may display advice to the user regarding the adjustment of the inter-lens distance on the adjustment screen 300. This advice may be, for example, "If the screen is clearly visible even though the eyes are not displayed, please press the OK button."

[0084] Further variations will be described. At least some of the functions implemented in the image generation device 200 in the above embodiment may be implemented in the head-mounted display 100, or in a server connected to the image generation device 200 via a network. For example, the head-mounted display 100 may have a function to generate data for various screens and images based on camera images and sensor measurements. The server may also have a function to generate data for various screens and images based on camera images and sensor measurements, and the head-mounted display 100 may display screens and images generated by the server.

[0085] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present disclosure. The new embodiments resulting from such combinations will possess the combined effects of the respective embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof. [Industrial applicability]

[0086] This invention can be applied to devices and systems that assist in adjusting the distance between the lenses of a head-mounted display. [Explanation of Symbols]

[0087] 10 Image display system, 100 Head-mounted display, 200 Image generation device, 264 Display control unit, 272 Inter-lens distance acquisition unit, 276 Eye-line measurement unit, 278 Shift detection unit, 280 Adjustment screen generation unit.

Claims

1. An adjustment screen generation unit generates an adjustment screen for a user wearing a head-mounted display to adjust the distance between the lenses of the head-mounted display, A display control unit that displays the adjustment screen on the head-mounted display, Equipped with, The adjustment screen generation unit places a lens image indicating the position of the lenses of the head-mounted display on the adjustment screen, and further places a pupil image indicating the position of the user's pupils based on the eye-tracking results on the adjustment screen. The adjustment screen generation unit changes the position of the lens image on the adjustment screen when the position of the lens is changed. Information processing device.

2. The adjustment screen generation unit, when the magnitude of the discrepancy between the position of the lenses of the head-mounted display and the position of the user's pupils exceeds a predetermined threshold, sets content on the adjustment screen that suggests there is a discrepancy between the position of the lenses and the position of the pupils. The information processing apparatus according to claim 1.

3. The adjustment screen generation unit further places on the adjustment screen at least one of an example showing the correct position of the lens image and the pupil image, or an example showing an incorrect position. The information processing apparatus according to claim 1.

4. The adjustment screen generation unit sets content on the adjustment screen indicating that the distance between the lenses of the head-mounted display is appropriate when the position of the user's pupils relative to the position of the lenses of the head-mounted display remains within an appropriate range for a predetermined period of time or longer. The information processing apparatus according to claim 1.

5. A step of generating an adjustment screen for a user wearing a head-mounted display to adjust the distance between the lenses of the head-mounted display, The steps include displaying the adjustment screen on the head-mounted display, The computer executes this, The adjustment screen includes a lens image showing the position of the lenses of the head-mounted display, and further includes a pupil image showing the position of the user's pupils based on the eye-tracking results. The above generation step involves changing the position of the lens image on the adjustment screen if the position of the lens is changed. Adjustment screen display method.

6. A function to generate an adjustment screen for a user wearing a head-mounted display to adjust the distance between the lenses of the head-mounted display, A function to display the aforementioned adjustment screen on the head-mounted display, To make this a reality on a computer, The function that generates the image places a lens image indicating the position of the lenses of the head-mounted display on the adjustment screen, and further places a pupil image indicating the position of the user's pupils on the adjustment screen based on the eye-tracking results. The function that generates the lens changes the position of the lens image on the adjustment screen when the position of the lens is changed. Computer program.

Images