Control device, control method, and program

Abstract

To complete automatic IPD adjustment quickly even when a disturbance occurs.SOLUTION: A control device for controlling an operation of a head-mounted display device having a left side display unit and a right side display unit includes a control part for determining driving target positions of the left side display unit and the right side display unit on the basis of each pupil detection result of the left eye and the right eye of a user wearing the display device when a start trigger related to position adjustment of the left side display unit and the right side display unit is acquired, and for driving the left side display unit and the right side display unit toward each of the determined driving target position. When either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, the control part determines the driving target positions of both the left side display unit and the right side display unit on the basis of the pupil detection result of the other eye that can be detected.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] Conventionally, mechanisms have been developed for HMD (Head Mounted Display) that automatically adjust the positions of the left and right displays and the optical axis of the eyepiece optical system for observing the displays to match the interpupillary distance of the user. Such adjustment mechanisms are also called automatic IPD adjustment mechanisms.

[0003] For example, Patent Document 1 discloses a method for storing an adjustment record relating to a user's features and the user's interpupillary distance, and for driving a display to a position according to the corresponding adjustment record when the user's feature information is detected. Patent Document 2 discloses a method for irradiating a user's eye with infrared rays, receiving the reflected light with an infrared receiving element, and realizing automatic IPD adjustment based on the output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2018-518707 [Patent Document 2] Japanese Patent Application Publication No. 6-276459 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, when there is no adjustment record, it is necessary to drive the display unit to a position that matches the user's IPD and store the position as an adjustment record in order to obtain a new adjustment record. If the user manually sets and registers the display unit position, there is a hassle associated with the manual operation. On the other hand, if it is to be automatically controlled, it is necessary to observe the eye position, which differs for each user, by some means. However, during this observation, there is a possibility of disturbance such as ghosting or eye closure, and when a disturbance occurs, it takes time to complete the drive.

[0006] In addition, the technology disclosed in Patent Document 2 also controls the display unit based on the output of the infrared receiving element, so if an external disturbance affects the output of the infrared receiving element, it will take time to complete driving.

[0007] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a control device that makes it possible to quickly complete automatic IPD adjustment even when a disturbance occurs. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, according to one aspect of the present invention, there is provided a control device for controlling the operation of a head-mounted display device having a left display unit and a right display unit, the control device including: a control unit that, when a start trigger for adjusting the position of the left display unit and the right display unit is obtained, determines drive target positions of the left display unit and the right display unit based on pupil detection results of the left eye and right eye of a user wearing the display device, and drives the left display unit and the right display unit toward the determined drive target positions, and when it is not possible to detect either the pupil of the right eye or the pupil of the left eye of the user, determines the drive target positions of both the left display unit and the right display unit based on the pupil detection result of the other eye that is detected. Effect of the Invention

[0009] According to the present invention, it is possible to provide a control device that makes it possible to quickly complete automatic IPD adjustment even when a disturbance occurs. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an external view of an HMD according to a first and second embodiment. [Diagram 2] FIG. 2 is an explanatory diagram for explaining an example of a functional configuration of a control box. [Diagram 3] FIG. 2 is a diagram showing a drive mechanism / movable range of the display unit in the first and second embodiments. [Figure 4] FIG. 2 is an enlarged view of an eyepiece portion of the display unit according to the first and second embodiments. [Diagram 5] 2 is a flowchart of the first embodiment. [Figure 6] 5A to 5C are diagrams illustrating an example of driving the display unit according to the first embodiment. [Figure 7] 13A to 13C are diagrams illustrating an example of driving a display unit according to the second embodiment. [Figure 8] 4A to 4C are conceptual diagrams of images captured by a camera for observing an eyeball in the first and second embodiments. [Figure 9] 11 is a flowchart of a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] (First embodiment) Hereinafter, an HMD 100 according to a first embodiment of the present invention will be described with reference to Figs. 1 to 5 and 8. Fig. 1 is an external view of the HMD 100 of the first embodiment. The HMD 100 is an example of a head-mounted display device. As shown in Fig. 1, the HMD 100 comprises an HMD main body 101, a connecting part 102, and a wearing part 103. The HMD main body 101 is connected to the wearing part 103, which is worn on the user's head like a headband, via the connecting part 102 in order to be fixed in front of the user's eyes.

[0013] Moreover, the HMD 100 according to this embodiment is connected to a control box 302, which is a device that controls the position, display, etc. of the HMD main body 101, via a connection cable 303. Hereinafter, an example of the functional configuration of the control box 302 will be described with reference to FIG.

[0014] 2 is an explanatory diagram showing an example of the functional configuration of the control box 302. The control box 302 is a control device that performs various controls of the HMD 100. The control box 302 includes, for example, a storage unit 310, a communication unit 320, and a control unit 330, as shown in FIG.

[0015] The storage unit 310 has a function of storing various information for the operation of the control box 302. For example, the storage unit 310 holds various information such as Th (first predetermined value), Th2 (second predetermined value), and Rth (third predetermined value) described later. The storage unit 310 can be realized by, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0016] The communication unit 320 performs various communications with the HMD 100. The communication unit 320 can be connected to the Internet, for example, via a wired LAN (Local Area Network), a wireless LAN, Wi-Fi (registered trademark), or the like. The control box 302 may be connected to the HMD main body 101 via a wireless device, and in this form, the user can operate the HMD main body 101 without being bothered by a connection cable 303.

[0017] The control unit 330 functions as an arithmetic processing unit or a control device, and controls the overall operation or a part of the operation of each component of the control box 302 based on various programs recorded in the storage unit 310 etc. The control unit 330 can be realized by a processor such as a CPU (Central Processing Unit) or an MCU (Micro Controller Unit).

[0018] When operation information relating to position adjustment of the left display unit 105 and the right display unit 104, which will be described later, is obtained, the control unit 330 determines drive target positions of the left display unit 105 and the right display unit 104. Here, the control unit 330 determines the drive target positions of the left display unit 105 and the right display unit 104 based on the pupil detection results of the left eye and the right eye of the user wearing the HMD 100.

[0019] Next, the control unit 330 drives the left display unit 105 and the right display unit 104 toward the determined drive target positions.

[0020] It is possible that the user's pupils cannot be detected due to blinking, ghosting, or other factors. In other words, the pupil detection result may include a result that the pupils could not be detected. A method for controlling the drive of the display units 104 and 105 in such a case where the pupils cannot be detected is desired.

[0021] Therefore, when the control unit 330 is unable to detect either the pupil of the user's right eye or the pupil of the left eye, it determines the drive target positions of both the left display unit 105 and the right display unit 104 based on the pupil detection result of the other eye that is detected.

[0022] For example, assume that the pupil of the user's right eye can be detected but the pupil of the left eye cannot be detected. In this case, the control unit 330 determines the drive target positions of both the left display unit 105 and the right display unit 104 using information on the pupil of the user's right eye.

[0023] A specific example will be described later regarding driving the left display unit 105 and the right display unit 104. In the following, the left display unit 105 and the right display unit 104 may be collectively referred to as display units 104, 105.

[0024] Furthermore, the control unit 330 may control, for example, the operation of a stepping motor 109 (described later) when driving the display units 104 and 105.

[0025] An example of the functional configuration of the control box 302 according to this embodiment has been described above. Note that each function of the control unit 330 mounted on the control box 302 may be implemented as an application on a general PC.

[0026] Furthermore, some or all of the functional configuration of the control box 302 may be mounted on the HMD 100. Specifically, various functions of the control box 302, such as a determination means and a drive means, may be mounted on a CPU mounted on a board in the HMD main body 101. In this case, the user can experience XR using only the HMD main body 101, reducing the complexity of handling the control box 302 and the connection cable 303 during the experience.

[0027] 3(a) and (b) are external views seen from the side of a user wearing the HMD 100. The right-side display unit 104 includes a display panel 108 and an eyepiece (not shown) which is a display optical system, and can be driven to change its position in the direction of the arrow shown in FIG. 3 (horizontal direction).

[0028] 3, the left display unit 105 is configured symmetrically to the right display unit 104 with respect to this center line, and its position can be changed in the horizontal direction in the same way. Note that the right and left sides here refer to the right eye side and the left eye side. Also, when the right eye side is one side, the left eye side corresponds to the other side.

[0029] Fig. 3(a) shows the right display unit 104 and the left display unit 105 when both are driven to the outermost positions, and Fig. 3(b) shows the opposite, when driven to the innermost positions. Fig. 3(c) shows the configuration around the right display unit 104. The right display unit 104, which includes an eyepiece optical system and a display panel 108, can be driven left and right by a stepping motor 109, and its absolute position can be measured by an encoder 110.

[0030] Furthermore, the display units 104 and 105 can be driven by a stepping motor 109 (automatic position adjustment), but the user can also manually adjust the position by applying force to the display units 104 and 105 to move them.

[0031] If the distance from the center line to the optical axis center of the right-side display unit 104 is denoted as Xr, and the distance to the optical axis center of the left-side display unit 105 is denoted as Xl, the movable ranges of Xr and Xl are expressed by the following formulas (1) and (2), respectively.

[0032] X_min≦Xr≦X_max (1) X_min≦Xl≦X_max (2) The smaller X_min and the larger X_max, the wider the IPD adjustment range and the more diverse the users can be accommodated, but the larger the space required for the drive mechanism, the larger the HMD body 101. Furthermore, since the distribution of interpupillary distances of adults is determined to a certain extent, for example, in this embodiment, it is assumed that users in the range of 51 to 77 mm are covered, and X_min=25.5 mm and X_max=38.5 mm are set. However, the range of interpupillary distances to be covered and the set values ​​of X_min and X_max are not limited to the above example.

[0033] 4 is an enlarged view of the eyepieces of right display unit 104 and left display unit 105. As shown in the figure, an infrared emitting diode (hereinafter referred to as IRED) 106 or a camera 107 for observing the user's eye may be placed at the 1 to 12 o'clock positions of the clock. Camera 107 is sensitive to at least the wavelengths in the infrared region emitted by IRED 106.

[0034] In the example shown in FIG. 4, in the right display unit 104, the camera 107 is arranged only at the 8 o'clock position, and the IREDs 106 are arranged at the other positions of 1 to 7 o'clock and 9 to 12 o'clock. However, the arrangement of the cameras 107 and the IREDs 106 is not limited to this example. In the example shown in the figure, each eyepiece is equipped with one camera 107 and eleven IREDs 106, but the number of cameras 107 and IREDs 106 mounted on the eyepieces is not limited to this example. For example, two or more cameras 107 may be mounted on the eyepieces.

[0035] The camera 107 and IRED 106 also serve as a mechanism for detecting the direction of the user's gaze. The IRED 106 illuminates the user's eye, and the camera 107 captures the light reflected by the cornea (Purkinje image) and the user's pupil, and the gaze direction is determined from the positional relationship. This is a well-known technique, so a detailed description will be omitted here.

[0036] In the automatic IPD adjustment, the control unit 330 determines the drive target positions of the display units 104, 105 from the image acquired by the camera 107. Specifically, the control unit 330 determines the drive target positions using the following first to third information, and moves the left and right display units 104, 105. The first information here is information indicating where the current pupil is shown in the captured image. The second information is information indicating the expected pupil position in the captured image if the IPD is adjusted correctly. The third information is information indicating the correlation (sensitivity) between the drive amount of the display units 104, 105 and the pupil position in the captured image.

[0037] The left display unit 105 is configured symmetrically to the right display unit 104, and thus has a configuration in which the camera 107 is disposed at the 4 o'clock position, and the IREDs 106 are disposed at the 1 to 3 o'clock and 5 to 12 o'clock positions, as shown in Fig. 4. However, similar to the right display unit 104, the positions and the number of the cameras 107 and the IREDs 106 are not limited to the above example.

[0038] Next, a flow chart of the first embodiment will be described with reference to FIG. 5. Since a human face is almost symmetrical, it is possible to control the right display unit 104 and the left display unit 105 so that they are always driven to positions symmetrical with respect to the center line in FIG. 3. However, since a face is not completely symmetrical, and the HMD 100 is not necessarily worn so that the center line in FIG. 3 is aligned with the user's median line, it is preferable to control the left and right separately, and this embodiment also controls the left and right separately. The process in FIG. 5 is an adjustment flow for one of the eyes, and in reality, each process in FIG. 5 is performed almost simultaneously for each of the left and right eyes, thereby controlling the left and right separately.

[0039] The first step S401 in the flow of Fig. 5 is a start step of the automatic IPD adjustment. When the user wears the HMD 100 and completes preparations to start use, the automatic IPD adjustment starts in S401. The start trigger for the automatic IPD adjustment may be some kind of action from the user to the HMD main body 101, such as a button operation. The start trigger may also be a form that detects the completion of the user's wearing from the output of the binocular camera 107 or a gyro sensor built into the HMD main body 101. When some kind of start trigger is obtained and the automatic IPD adjustment starts, the process proceeds to S402.

[0040] In the automatic IPD adjustment, control unit 330 detects the pupil position from the image captured by camera 107, and from there determines and drives target positions for display units 104 and 105. Therefore, if the pupil cannot be detected in the image captured by camera 107, automatic IPD adjustment basically cannot be performed, and some ingenuity is required to continue the adjustment.

[0041] S402 is a process of resetting an internal variable E that counts up the number of errors that occur when an error occurs when the pupil cannot be detected in an image captured by the camera 107. Immediately after the start, E=0 is set and the process proceeds to S403.

[0042] S403 is a process of detecting the pupil from the image captured by the camera 107. The process of detecting the pupil may be performed by the control unit 330 using the image captured by the camera 107, or may be performed by the HMD 100 side (for example, the camera 107).

[0043] FIG. 8 is a schematic example of an image of the area around the eye captured by the camera 107. As shown in FIG. 8, the pupil is characterized by being darker than other parts and having a circular or elliptical shape. Therefore, by utilizing this characteristic, it is possible to detect the position of the pupil in the image. On the other hand, for a user who wears glasses in particular, ghosts may appear in the captured image due to the relative positions of the IRED 106, the camera 107, and the eyeball and glasses, making it difficult to identify the position of the pupil. In addition, if the timing of a blink is captured by the camera 107, the pupil will not be captured.

[0044] In these cases, pupil detection is NG. As described above, if the pupil position can be specified (detected), the process proceeds to S404, and if not, the process proceeds to S411. S404 to S410 in the case where the pupil position can be specified (detected) will be described first.

[0045] As already described, in S404, the control unit 330 calculates the necessary drive amount (hereinafter, also referred to as the target drive amount). The target drive amount can be calculated using the position of the pupil in the captured image detected in S403, the expected pupil position in the captured image when the IPD is correctly adjusted, and the correlation (sensitivity) between the drive amount of the display units 104 and 105 and the pupil position in the captured image. Ideally, it is desirable to drive to the true IPD adjustment position in one detection, but in reality, due to various error factors, multiple drives may be required. For example, when the drive distance is large, there is a large error in the sensitivity, which is the correlation between the drive amount of the display units 104 and 105 and the pupil position in the captured image, so that it may not be possible to drive to the appropriate position in one drive.

[0046] Also, the drive target value may not completely converge to the true IPD position due to factors such as noise of the camera 107 and backlash in the display unit drive system. Therefore, if the target drive amount becomes less than Th (less than the first predetermined value), it may be considered that the IPD adjustment is completed since the position is sufficiently close to the true IPD position. Here, the drive target value is an example of the drive target position. Also, the target drive amount indicates the difference between the above-mentioned drive target position (drive target value) and the position before driving. Therefore, the drive amount here is an absolute value. Also, Th is an example of a first predetermined value. S404 is a step for making this judgment, and if the target drive amount is equal to or greater than Th (equal to or greater than the first predetermined value), the control unit 330 proceeds to S405 to actually drive the display units 104 and 105.

[0047] S405 is a step for determining whether or not the drive target value falls within the movable range Xr_min to Xr_max of the display units 104 and 105. When the center line in Fig. 3 is misaligned from the user's midline and worn, for example, there is a possibility that a position outside the movable range becomes the drive target value. Therefore, if the drive target value is outside the movable range of the display units 104 and 105 (i.e., when it is outside the movable range), the process proceeds to S409. Then, the control unit 330 drives the display units 104 and 105 to the end of the movable range on the drive target value side (drive target position side). And error processing is performed in S410.

[0048] If it is within the movable range, the process proceeds to S406, and the control unit 330 actually drives the display units 104 and 105 toward the drive target value. After the drive is completed, the process returns to S403 again, and S403 to S406 are repeated until the target drive amount converges to less than Th.

[0049] If the target drive amount converges to less than Th in S404, the drive is completed and the process proceeds to S407 for error checking. If, for example, after the drive for automatic IPD adjustment is completed for both the left and right sides, the difference between the left and right stop positions is large, it is suspected that the center line in Fig. 3 is misaligned from the user's midline and worn. In such a case, the stop positions of the display units 104 and 105 on the left and right are likely to be at the very edge of the movable range. Therefore, if the wearing becomes further misaligned for some reason while the user is using the HMD100, there is a risk that the true IPD position will not fall within the movable range of the display units 104 and 105.

[0050] Therefore, for example, when the final stop position of the right display unit 104 is Xr = Xrf and the stop position of the left display unit 105 is Xl = Xlf, if |Xrf - Xlf| ≥ Th2 is satisfied, the process proceeds to S410. In S410, the control unit 330 displays an error screen prompting the user to re-wear the HMD100, and ends the automatic IPD adjustment in S408. If |Xrf - Xlf| < Th2 in S407, the automatic IPD adjustment is ended in S408 without displaying an error. Note that Th2 is an example of a second predetermined value determined in advance.

[0051] On the other hand, even when the drive target value goes out of the movable range in S405, it is similarly suspected that the HMD 100 is misaligned and worn. Therefore, it is driven to the end of the movable range as close as possible to the true IPD position and an error display is performed. That is, when the drive target value is Xr > Xr_max, it is driven to Xr = Xr_max, and when Xr < Xr_min, it is driven to Xr = Xr_min at S409, an error display is performed at S410, and the automatic IPD adjustment is terminated at S408.

[0052] In this way, S403 to S406 are repeated until the target drive amount converges within the range of ±Th. When it converges, an error check is performed at S407 to complete the automatic IPD adjustment, which is the normal flow when pupil detection is stably possible at S403.

[0053] FIG. 6(a) shows a driving example of the typical right display unit 104 when pupil detection is stably possible. It moves from the initial position Xr = A to Xr = B by the first drive based on pupil detection, then to Xr = C by the second drive, and after the third pupil detection, it confirms the convergence of the target drive amount and completes the automatic IPD adjustment.

[0054] Next, the flow of S411 to S418 when pupil detection cannot be performed at S403 will be described.

[0055] S411 is a count-up process of the internal variable E that counts up when pupil detection fails. When pupil detection fails at S403 and E is counted up at S411, the process proceeds to S412.

[0056] S412 is a judgment for the counted-up internal variable E. In this embodiment, if an error occurs once, the automatic IPD adjustment is attempted to continue in the processes from S414 onwards, which will be described later. However, if an error occurs twice, it is considered difficult to complete the automatic IPD adjustment due to reasons such as the user having thick eyelids that make it difficult to detect the pupil. Therefore, if E=2, the automatic IPD adjustment is abandoned, and an error is displayed in S413 to prompt the user to adjust the IPD manually, after which the automatic IPD adjustment is terminated. Note that, although an example is shown here in which the automatic IPD adjustment is abandoned when the judgment is made that E=2, the judgment value (predetermined number of times) may be set to a different number of times, such as E=3.

[0057] If E=2 is not reached in S412, proceed to S414. S414 to S415 are steps in which, since the pupil of the eye for which IPD adjustment is being attempted could not be detected, the display units 104 and 105 are driven using the drive target value of the opposite eye, which is performed almost simultaneously. That is, it is assumed that a human face is almost symmetrical, and further that the user wears the HMD 100 by aligning the center line in FIG. 3 with the user's own midline. According to this assumption, it is predicted that the final stop positions Xrf and Xlf of the left and right display units 104 and 105 will be Xrf≒Xlf.

[0058] However, since there may be cases where the pupil of the opposite eye cannot be detected at the same time, the control unit 330 determines whether or not there is a pupil detection result for the opposite eye in S414. If there is a pupil detection result for the opposite eye, in S415, the control unit 330 drives the right display unit 104 (or the left display unit 105) to a symmetrical position of the drive target value for the opposite eye.

[0059] Fig. 6(b) shows an example of driving the right display unit 104 when pupil detection is not possible at the initial position. For example, assume that pupil detection is possible in the left display unit 105 and that the left display unit 105 is being driven with a drive target value of position Xl = D. In this case, the control unit 330 may drive the right display unit 104 with a drive target value of position Xr = D (a symmetrical position to the drive target value of the left display unit 105) as shown in Fig. 6(b).

[0060] Next, a flow of steps S416 to S418 will be described when the pupils of both eyes cannot be detected in step S414. As already described, possible causes of failure to detect the pupils include ghosting and closed eyes. Since ghosting occurs due to the positional relationship between IRED 106, camera 107, and the eyeballs (and possibly glasses), improvement is expected by driving display units 104 and 105 from the position at which the ghosting occurred. Also, closed eyes will improve over time. Therefore, the aim is to make pupil detection possible by driving display units 104 and 105 to change their positions and by detecting the pupils again after the time has elapsed due to the position change.

[0061] In driving, since it is not possible to set a plausible drive target value, the control unit 330 drives the display units 104, 105 for both eyes to a position of Xr=Xl=Xa, which is a preset fixed value, in S417. The fixed value Xa may be anywhere within the movable range of the display units 104, 105, but since there is statistical data that shows that the average interpupillary distance of humans is 63 mm, for example, it may be set to Xa=31.5. This makes it possible to drive in a direction that is probabilistically more likely to approach the user's true IPD position. The fixed value Xa may be set by the user, or may be set automatically by referring to past data, etc.

[0062] Even more rarely, it may be possible that pupil detection is not possible on either the left or right side at the position Xr=Xl=Xa. In this case, from the viewpoint of driving to a different position, a process of driving is performed by an amount of a preset fixed value Xb. The value of Xb may be added to Xa (driving toward the outside of the face) or subtracted from it (driving toward the inside of the face), but it is sufficient that the position (Xa+Xb) is set within the movable range of the display units 104 and 105. Note that the fixed value Xb here is an example of another fixed value.

[0063] By returning to S403 after the above processing of S415 to S418, even if the first pupil detection is unsuccessful, the possibility of a second pupil detection being possible increases, and the automatic IPD adjustment can be completed quickly even when a disturbance occurs.

[0064] Second Embodiment Hereinafter, an HMD 100 according to a second embodiment of the present invention will be described with reference to Figures 7(a), 7(b), and 9. As the hardware configuration is similar to that of the first embodiment, a description thereof will be omitted.

[0065] The second embodiment is an example of overlap control in which pupil detection is performed not only when the display units 104 and 105 are stopped but also during driving, and the obtained drive target value is updated while controlling. An example of overlap control is shown in FIG. 7(a). First, the control unit 330 performs pupil detection at the initial position Xr=A, and drives the display unit to the drive target value Xr=B obtained therefrom. Furthermore, the control unit 330 performs pupil detection without stopping when it reaches Xr=A', updates the drive target value to the drive target value Xr=C obtained therefrom, and continues driving the display unit. That is, the control unit 330 may perform control characterized by canceling drive 1 in the figure midway and seamlessly switching to drive 2. By performing such overlap control, it may be possible to drive more smoothly to the true IPD position.

[0066] On the other hand, as shown in FIG. 7(b), there may be a case where pupil detection is NG even during overlap control. FIG. 7(b) shows a case where pupil detection is successful at the initial position of Xr=A, and driving is started with Xr=B as the driving target value (driving 3 in the figure), but the second pupil detection at the position of Xr=A' in overlap control is NG. In this case, the control unit 330 may perform driving 4 in the figure, such as updating the driving target value to the symmetrical position Xr=D of the driving target value of the opposite eye, as in S414 shown in FIG. 5 of the first embodiment. Alternatively, the control unit 330 may perform driving 5 in the figure, which drives toward Xr=B without updating the driving target value.

[0067] As described above, an example of a flowchart when pupil detection fails during overlap control is shown in FIG. 9. When pupil detection fails during overlap control in S701 of FIG. 9, the control unit 330 evaluates the reliability of the immediately preceding pupil detection result in S702. Although various concepts and calculation methods can be considered for the reliability of the pupil detection result, for example, since an image as shown in FIG. 8 is acquired by the camera 107, a case where the edge of the pupil in the image can be detected more clearly can be considered to have a high reliability. For example, it is conceivable to count the number of pixels of the edge of the pupil (the boundary between the pupil and the iris) detected while scanning the image from top to bottom, and use this count value as the reliability. When the count value is high, it is considered that the area where the pupil is partially hidden by the eyelid or the like is small, and thus a highly reliable pupil detection has been performed. Let the reliability of the immediately preceding pupil detection result calculated in this way be R (here, the larger R is, the higher the reliability), and a preset predetermined value Rth. When R < Rth, the process proceeds to S703, and when R ≧ Rth, the process proceeds to S705. Note that the predetermined value Rth here is an example of a third predetermined value.

[0068] S703 is a step of checking whether the pupil of the opposite eye has been detected. When the reliability of the immediately preceding pupil detection result is low in S702, the control unit 330 does not adopt the immediately preceding drive target value and wants to set the drive target value based on the information of the opposite eye. However, if the opposite eye also fails in pupil detection in S703, such control cannot be performed. Therefore, when the pupil of the opposite eye has been detected in S703, the control unit 330 proceeds to S704 and drives the display unit to the symmetric position of the drive target value of the opposite eye.

[0069] On the other hand, when the reliability of the immediately preceding pupil detection result is high in S702 or when the opposite eye also fails in pupil detection in S703, the process proceeds to S705. Then, the control unit 330 drives the display unit with the drive target value based on the pupil detection result before update without updating the drive target value, and performs the next pupil detection in S706.

[0070] Both the first and second embodiments are characterized in that even if pupil detection of one eye is NG, the pupil detection result of the opposite eye is used. If pupil detection of both the left and right eyes is NG at the same time, the information of the opposite eye cannot be used, so it is desirable to take measures to avoid such a situation. Therefore, the following describes measures to avoid the case where pupil detection of both the left and right eyes is NG at the same time.

[0071] First, one of the possible causes of pupil detection failure is blinking, but due to the structure of the human nervous system, the left and right eyes basically blink at the same time. In other words, when photographing the left and right eyes with camera 107, it is possible to avoid a situation in which the left and right blinking timings are photographed at the same time by intentionally shifting the exposure timing of the camera photographing the right eye and the camera photographing the left eye.

[0072] Ghosting is also considered to be one of the causes of pupil detection NG, and ghosting occurs due to the positional relationship between the IRED 106, the camera 107, and the eyeball (and possibly glasses). Assume that the human face is completely symmetrical, the HMD 100 is worn with the center line of FIG. 3 aligned with the midline of the user, and the automatic IPD adjustment start position is also symmetrical. In this case, the left and right display units 104 and 105 during automatic IPD adjustment are always driven while taking completely symmetrical positions. In such a case, if a condition occurs in which a ghost appears in one eye, it is assumed that a ghost will also appear in the opposite eye. Therefore, it is desirable for the control unit 330 to control the left and right display units 104 and 105 not to exist in symmetrical positions at the same time while the display units are being driven. For example, if automatic IPD adjustment is performed first for only one eye, and adjustment of the opposite eye is started after the adjustment is completed or after a short time has elapsed, the possibility of both the left and right being affected by ghosting at the same time can be reduced.

[0073] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0074] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0075] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0076] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0077] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0078] [Configuration 1] A control device for controlling an operation of a head-mounted display device having a left display unit and a right display unit, a control unit that, when a start trigger for adjusting the positions of the left and right display units is obtained, determines drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and drives the left and right display units toward the determined drive target positions; Equipped with The control unit is when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, determining drive target positions of both the left display unit and the right display unit based on a pupil detection result of the other eye that can be detected; Control device.

[0079] [Configuration 2] The control unit is when both the pupil of the right eye and the pupil of the left eye of the user are detected, a drive target position of the right display unit is determined based on a position of the pupil of the right eye of the user, and a drive target position of the left display unit is determined based on a position of the pupil of the left eye of the user. 2. The control device according to configuration 1.

[0080] [Configuration 3] when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, a symmetrical position of the drive target position of the display unit on the other side, which is determined based on the position of the pupil of the other eye that can be detected, with respect to a center line of the face of the user, is determined as the drive target position of the display unit on the one side where the pupil cannot be detected. The control device according to configuration 2.

[0081] [Configuration 4] The control unit is repeating a process of determining drive target positions of the left display unit and the right display unit and driving the left display unit and the right display unit until a target drive amount indicating a difference between the drive target position and a position before driving becomes less than a first predetermined value; 4. The control device according to configuration 2 or 3.

[0082] [Configuration 5] The control unit is when the target drive amount is equal to or greater than the first predetermined value and the drive target position is within a movable range, drive the left display unit or the right display unit, for which the drive target position has been determined, toward the drive target position; The control device according to configuration 4.

[0083] [Configuration 6] The control unit is when the target drive amount is equal to or greater than the first predetermined value and the drive target position is outside the movable range, driving the left display unit or the right display unit, for which the drive target position has been determined, toward an end of the movable range on the drive target position side; The control device according to configuration 5.

[0084] [Configuration 7] The control unit is when the difference between the stop position of the left display unit and the stop position of the right display unit when the target drive amount of both the right display unit and the left display unit becomes less than the first predetermined value is equal to or greater than a second predetermined value, performing control to display an error screen on at least one of the left display unit or the right display unit; A control device according to any one of configurations 4 to 6.

[0085] [Configuration 8] The control unit is driving the left and right display units toward positions corresponding to preset fixed values ​​when neither the right eye pupil nor the left eye pupil of the user is detected; The control device according to any one of configurations 2 to 7.

[0086] [Configuration 9] The control unit is when neither the pupil of the right eye nor the pupil of the left eye of the user can be detected, and when the left display unit and the right display unit are present at a position corresponding to the fixed value, driving the left display unit and the right display unit toward a position obtained by adding or subtracting another preset fixed value to or from the fixed value; 9. The control device according to configuration 8.

[0087] [Configuration 10] The control unit is When at least one of the pupils of the right eye and the left eye of the user cannot be detected a predetermined number of times, control is performed to display an error screen on at least one of the left display unit and the right display unit, the error screen prompting a manual adjustment. A control device according to any one of configurations 3 to 9.

[0088] [Configuration 11] the pupil detection result is obtained while the left display unit or the right display unit is being driven, The control unit is updating the drive target position based on a pupil detection result of the left eye or the right eye of the user while driving the left display unit or the right display unit, and driving the left display unit or the right display unit toward the updated drive target position; 3. The control device according to configuration 1 or 2.

[0089] [Configuration 12] The control unit is When the drive target position of one of the left display unit and the right display unit cannot be updated, the one of the display units is driven based on a reliability of a pupil detection result before the update on the one side. The control device according to configuration 11.

[0090] [Configuration 13] The control unit is When the drive target position of one of the left display unit and the right display unit cannot be updated, and the reliability of the pupil detection result before the update on the one side is equal to or higher than a third predetermined value, the one display unit is driven toward the drive target position before the update. 13. The control device according to configuration 12.

[0091] [Configuration 14] The control unit is when the drive target position of one of the left display unit and the right display unit cannot be updated, and when the reliability of the pupil detection result before the update on the one side is less than the third predetermined value and when the pupil of the eye on the other side has been detected, driving the one display unit toward a symmetrical position of the drive target position of the other display unit; 14. The control device according to configuration 13.

[0092] [Configuration 15] The control unit is when the drive target position of one of the left display unit and the right display unit cannot be updated, and when the reliability of the pupil detection result before the update on the one side is less than the third predetermined value and when the pupil of the eye on the other side cannot be detected, driving the one of the display units toward the drive target position before the update on the one side. 15. The control device according to configuration 14.

[0093] [Configuration 16] the pupil detection result is acquired by photographing with a camera included in the display unit; The reliability is based on the number of pixels of the edge of the pupil included in the image acquired by photographing the camera. A control device according to any one of configurations 12 to 15.

[0094] [Configuration 17] The control unit is performing control to shift the exposure timing of a camera included in the right display unit and a camera included in the left display unit; 17. The control device according to configuration 16.

[0095] [method] A method for controlling an operation of a head-mounted display device having a left display unit and a right display unit, comprising: when a start trigger for adjusting the positions of the left and right display units is obtained, determining drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and further driving the left and right display units toward the determined drive target positions; Including, Determining drive target positions of the left display unit and the right display unit includes: determining drive target positions of both the left and right display units based on a pupil detection result of the other eye that has been detected when either the right or left pupil of the user cannot be detected; The computer-implemented control method further comprises:

[0096] [program] Computer, A control device for controlling an operation of a head-mounted display device having a left display unit and a right display unit, a control unit that, when a start trigger for adjusting the positions of the left and right display units is obtained, determines drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and drives the left and right display units toward the determined drive target positions; Equipped with The control unit is when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, determining drive target positions of both the left display unit and the right display unit based on a pupil detection result of the other eye that can be detected; A program that functions as a control device.

Claims

1. A control device for controlling an operation of a head-mounted display device having a left display unit and a right display unit, a control unit that, when a start trigger for adjusting the positions of the left and right display units is obtained, determines drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and drives the left and right display units toward the determined drive target positions; Equipped with The control unit is when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, determining drive target positions of both the left display unit and the right display unit based on a pupil detection result of the other eye that can be detected; Control device.

2. The control unit is when both the pupil of the right eye and the pupil of the left eye of the user are detected, a drive target position of the right display unit is determined based on a position of the pupil of the right eye of the user, and a drive target position of the left display unit is determined based on a position of the pupil of the left eye of the user. The control device according to claim 1 .

3. when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, a symmetrical position of the drive target position of the display unit on the other side, which is determined based on the position of the pupil of the other eye that can be detected, with respect to a center line of the face of the user, is determined as the drive target position of the display unit on the one side where the pupil cannot be detected. The control device according to claim 2.

4. The control unit is repeating a process of determining drive target positions of the left display unit and the right display unit and driving the left display unit and the right display unit until a target drive amount indicating a difference between the drive target position and a position before driving becomes less than a first predetermined value; The control device according to claim 2.

5. The control unit is when the target drive amount is equal to or greater than the first predetermined value and the drive target position is within a movable range, driving the left display unit or the right display unit, for which the drive target position has been determined, toward the drive target position; The control device according to claim 4.

6. The control unit is when the target drive amount is equal to or greater than the first predetermined value and the drive target position is outside the movable range, driving the left display unit or the right display unit, for which the drive target position has been determined, toward an end of the movable range on the drive target position side; The control device according to claim 5.

7. The control unit is when a difference between a stop position of the left display unit and a stop position of the right display unit when the target drive amount of both the right display unit and the left display unit becomes less than the first predetermined value is equal to or greater than a second predetermined value, performing control to display an error screen on at least one of the left display unit or the right display unit; The control device according to claim 4.

8. The control unit is driving the left and right display units toward positions corresponding to preset fixed values ​​when neither the right eye pupil nor the left eye pupil of the user is detected; The control device according to claim 2.

9. The control unit is when neither the pupil of the right eye nor the pupil of the left eye of the user can be detected, and when the left display unit and the right display unit are present at a position corresponding to the fixed value, driving the left display unit and the right display unit toward a position obtained by adding or subtracting another preset fixed value to or from the fixed value; The control device according to claim 8.

10. The control unit is When at least one of the pupils of the right eye and the left eye of the user cannot be detected a predetermined number of times, control is performed to display an error screen on at least one of the left display unit and the right display unit, the error screen prompting a manual adjustment. The control device according to claim 3.

11. the pupil detection result is obtained while the left display unit or the right display unit is being driven, The control unit is updating the drive target position based on a pupil detection result of the left eye or the right eye of the user while driving the left display unit or the right display unit, and driving the left display unit or the right display unit toward the updated drive target position; The control device according to claim 1 .

12. The control unit is When the drive target position of one of the left display unit and the right display unit cannot be updated, the one of the display units is driven based on a reliability of a pupil detection result before the update on the one side. The control device according to claim 11.

13. The control unit is when the drive target position of one of the left display unit and the right display unit cannot be updated, and when the reliability of the pupil detection result before the update on the one side is equal to or higher than a third predetermined value, driving the one of the display units toward the drive target position before the update. The control device according to claim 12.

14. The control unit is when the drive target position of one of the left display unit and the right display unit cannot be updated, and when the reliability of the pupil detection result before the update on the one side is less than the third predetermined value and when the pupil of the eye on the other side has been detected, driving the one display unit toward a symmetrical position of the drive target position of the other display unit; The control device according to claim 13.

15. The control unit is when the drive target position of one of the left display unit and the right display unit cannot be updated, and when the reliability of the pupil detection result before the update on the one side is less than the third predetermined value and when the pupil of the eye on the other side cannot be detected, driving the one of the display units toward the drive target position before the update on the one side. The control device according to claim 14.

16. the pupil detection result is acquired by photographing with a camera included in the display unit; The reliability is based on the number of pixels of the edge of the pupil included in the image acquired by photographing the camera. A control device according to any one of claims 12 to 15.

17. The control unit is performing control to shift the exposure timing of a camera included in the right display unit and a camera included in the left display unit; The control device according to claim 16.

18. A method for controlling an operation of a head-mounted display device having a left display unit and a right display unit, comprising: when a start trigger for adjusting the positions of the left and right display units is obtained, determining drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and further driving the left and right display units toward the determined drive target positions; Including, Determining drive target positions of the left display unit and the right display unit includes: determining drive target positions of both the left and right display units based on a pupil detection result of the other eye that has been detected when either the right or left pupil of the user cannot be detected; The computer-implemented control method further comprises:

19. Computer, A control device for controlling an operation of a head-mounted display device having a left display unit and a right display unit, a control unit that, when a start trigger for adjusting the positions of the left and right display units is obtained, determines drive target positions of the left and right display units based on pupil detection results of the left and right eyes of a user wearing the display device, and drives the left and right display units toward the determined drive target positions; Equipped with The control unit is when either the pupil of the right eye or the pupil of the left eye of the user cannot be detected, determining drive target positions of both the left display unit and the right display unit based on a pupil detection result of the other eye that can be detected; A program that functions as a control device.

Images