Control device, control method, and program

Abstract

To complete automatic IPD adjustment quickly even when a disturbance occurs.SOLUTION: A control device for controlling an operation of a head-mounted display device includes: an estimation unit for estimating a factor for not detecting the pupil of a user when the pupil of the user wearing the display device cannot be detected by a pupil detection part of a display unit of the display device; and an operation control unit for controlling an operation for driving the display unit of the display device on the basis of the factor estimated by the estimation unit.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] Conventionally, mechanisms have been developed for HMD (Head Mounted Display) that automatically adjust the positions of the left and right displays and the optical axis of the eyepiece optical system for observing the displays to match the interpupillary distance of the user. Such adjustment mechanisms are also called automatic IPD adjustment mechanisms.

[0003] For example, Patent Document 1 discloses a method for storing adjustment records relating to a user's features and the user's interpupillary distance, and for driving a display to a position according to the corresponding adjustment record when the user's feature information is detected. Patent Document 2 discloses a method for irradiating a user's eye with infrared rays, receiving the reflected light with an infrared receiving element, and realizing automatic IPD adjustment based on the output. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2018-518707 [Patent Document 2] Japanese Patent Application Publication No. 6-276459 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology disclosed in the above-mentioned Patent Document 1, when there is no adjustment record, in order to obtain a new adjustment record, it is necessary to drive the display unit to a position that matches the user's IPD and store that position as an adjustment record. If the user manually sets and registers the display unit position, there is a hassle associated with manual operations. On the other hand, if it is to be automatically controlled, it is necessary to observe the eye position, which differs for each user, by some means, but it does not take into consideration that this observation may be affected by external disturbances such as ghosts and closing of the eyes. In addition, the technology disclosed in Patent Document 2 also controls the display unit based on the output received by the infrared receiving element, but like Patent Document 1, it does not take into consideration that the output of the infrared receiving element may be affected by external disturbances.

[0006] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide a control device that makes it possible to quickly complete automatic IPD adjustment even when a disturbance occurs. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, according to one aspect of the present invention, there is provided a control device that controls the operation of a head-mounted display device, comprising: an estimation unit that estimates a factor why the pupil of a user wearing the display device cannot be detected when a pupil detection unit of a display unit of the display device cannot detect the pupil of the user; and an operation control unit that controls the operation of driving the display unit of the display device based on the factor estimated by the estimation unit. Effect of the Invention

[0008] According to the present invention, it is possible to provide a control device that makes it possible to quickly complete automatic IPD adjustment even when a disturbance occurs. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an external view of an HMD according to a first and second embodiment. [Diagram 2]FIG. 2 is a diagram illustrating an example of a functional configuration of a control box according to the first and second embodiments. [Diagram 3] FIG. 2 is a diagram showing a driving mechanism / driving range of the display unit according to the first and second embodiments. [Figure 4] FIG. 2 is an enlarged view of an eyepiece portion of the display unit according to the first and second embodiments. [Diagram 5] 2 is a flowchart of the first embodiment. [Figure 6] 3A and 3B are conceptual diagrams of images captured by a camera for eyeball observation in the first and second embodiments. [Figure 7] 4A to 4C are diagrams illustrating an example of driving the display unit according to the first and second embodiments. [Figure 8] 11 is a flowchart of a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0011] (First embodiment) Hereinafter, an HMD 100 according to a first embodiment of the present invention will be described with reference to Figs. 1 to 7. Fig. 1 is an external view of the HMD 100 of the first embodiment. The HMD 100 is an example of a head-mounted display device. As shown in Fig. 1, the HMD 100 comprises an HMD main body 101, a connecting part 102, and a wearing part 103. In order to fix the HMD main body 101 in front of the user's eyes, the HMD main body 101 is connected via the connecting part 102 to the wearing part 103, which is worn on the user's head like a headband.

[0012] Moreover, the HMD 100 according to this embodiment is connected to a control box 302, which is a device that controls the position and the like of the HMD main body 101, via a connection cable 303. Hereinafter, an example of the functional configuration of the control box 302 will be described with reference to FIG.

[0013] 2 is an explanatory diagram showing an example of the functional configuration of the control box 302. The control box 302 is a control device that performs various controls of the HMD 100. The control box 302 includes, for example, a storage unit 310, a communication unit 320, and a control unit 330, as shown in FIG.

[0014] The storage unit 310 has a function of storing various information for the operation of the control box 302. For example, the storage unit 310 holds various information such as an IPD average value and T_blink, which will be described later. The storage unit 310 can be realized by, for example, a read only memory (ROM) or a random access memory (RAM).

[0015] The communication unit 320 performs various communications with the HMD 100. It is communicatively connected to the HMD main body 101 via a connection cable 303. The communication unit 320 can be connected to the Internet, for example, by a wired LAN (Local Area Network), a wireless LAN, Wi-Fi (registered trademark), or the like. Note that the control box 302 may be connected to the HMD main body 101 by a wireless device, and in this form, the user can operate the HMD main body 101 without being bothered by the connection cable 303.

[0016] The control unit 330 functions as an arithmetic processing unit or a control device, and controls the overall operation or a part of the operation of each component of the control box 302 based on various programs recorded in the storage unit 310 etc. The control unit 330 can be realized by a processor such as a CPU (Central Processing Unit) or an MCU (Micro Controller Unit).

[0017] 2, the control unit 330 includes an estimation unit 331 and an operation control unit 335. When the pupil of a user wearing the HMD 100 cannot be detected by a pupil detection unit (camera 107 described later) of the display units 104 and 105 of the HMD 100, the estimation unit 331 estimates the cause of the failure to detect the pupil of the user.

[0018] For example, the estimation unit 331 determines whether the cause of the failure to detect the user's pupil is a ghost or closing of the eyes. A specific example of the estimation process will be described later.

[0019] The operation control unit 335 controls the operation of driving the display units 104 and 105 included in the HMD 100, based on the cause of why the user's pupil could not be detected, which is estimated by the estimation unit 331. The operation control unit 335 may control, for example, the operation of a stepping motor 109 (described later) when driving the display units 104 and 105.

[0020] An example of the functional configuration of the control box 302 according to this embodiment has been described above. Note that each function of the control unit 330 mounted on the control box 302 may be implemented as an application on a general PC.

[0021] Furthermore, some or all of the functional configuration of the control box 302 may be mounted on the HMD 100. Specifically, various functions of the control box 302, such as the determination means and control means, may be mounted on a CPU mounted on a board in the HMD main body 101. In this case, the user can experience XR using only the HMD main body 101, reducing the complexity of handling the control box 302 and connection cable 303 during the experience.

[0022] 3(a) and 3(b) are external views seen from the side of a user wearing the HMD 100. The right-side display unit 104 includes a display panel 108 (described later) and an eyepiece (not shown) which is a display optical system, and can be driven to change its position in the horizontal direction, as shown in FIG.

[0023] 3, the left display unit 105 is configured symmetrically to the right display unit 104 with respect to this center line, and its position can be changed in the horizontal direction in the same way. Note that the right and left sides here refer to the right eye side and the left eye side.

[0024] 3(a) is a diagram showing the HMD main body 101 when both the right display unit 104 and the left display unit 105 are driven to the outermost positions, and FIG. 3(b) is a diagram showing the HMD main body 101 when they are driven to the innermost positions. FIG. 3(c) is a configuration diagram showing the periphery of the right display unit 104. The right display unit 104, which includes an eyepiece optical system and a display panel 108, can be driven left and right by a stepping motor 109, and its absolute position can be measured by an encoder 110. In addition to being driven by the stepping motor 109, the right display unit 104 can also be manually adjusted in position by the user himself by applying force to the display unit when it is not driven by the stepping motor 109.

[0025] If the distance from the center line to the optical axis center of the right-side display unit 104 is denoted as Xr, and the distance to the optical axis center of the left-side display unit 105 is denoted as Xl, the movable ranges of Xr and Xl are expressed by the following formulas (1) and (2), respectively.

[0026] X_min≦Xr≦X_max (1) X_min≦Xl≦X_max (2) The smaller X_min and the larger X_max, the wider the IPD adjustment range and the more diverse the users can be accommodated, but the larger the space required for the drive mechanism, the larger the HMD body 101. Furthermore, since the distribution of interpupillary distances of adults is determined to a certain extent, for example, in this embodiment, it is assumed that users in the range of 51 to 77 mm are covered, and X_min=25.5 mm and X_max=38.5 mm are set. However, the range of interpupillary distances to be covered and the set values ​​of X_min and X_max are not limited to the above example.

[0027] 4 is an example of an enlarged view of the eyepieces of right display unit 104 and left display unit 105. In the eyepieces, infrared light emitting diodes (hereinafter referred to as IREDs) 106 or cameras 107 for observing the user's eyes may be disposed at each of the 1 to 12 o'clock positions on the clock as shown in the figure. Camera 107 is an example of a pupil detection unit, and is sensitive to at least the wavelengths in the infrared region emitted by IRED 106. IRED 106 is an example of a light emitting unit.

[0028] In the example shown in Fig. 4, in the right display unit 104, the camera 107 is arranged only at the 8 o'clock position, and the IREDs 106 are arranged at the other positions of 1 to 7 o'clock and 9 to 12 o'clock. However, the arrangement of the cameras 107 and the IREDs 106 is not limited to this example. In addition, in the example shown in Fig. 4, each eyepiece is equipped with one camera 107 and eleven IREDs 106, but the number of cameras 107 and IREDs 106 mounted on the eyepieces is not limited to this example. For example, two or more cameras 107 may be mounted on the eyepieces.

[0029] The camera 107 and IRED 106 also serve as a mechanism for detecting the direction of the user's gaze. The IRED 106 illuminates the user's eye, and the camera 107 captures the light reflected by the cornea (Purkinje image) and the user's pupil, and the gaze direction is determined from their positional relationship. This is a well-known technique, so details will be omitted here.

[0030] In automatic IPD adjustment, a drive target position of the display unit is determined from an image acquired by camera 107. Specifically, control unit 330 calculates a required drive amount using the following first to third information, and moves left and right display units 104, 105. The first information is information indicating where the current pupil is shown in the captured image. The second information is information indicating the expected pupil position in the captured image if the IPD is adjusted correctly. The third information is information indicating the correlation (sensitivity) between the drive amount of display units 104, 105 and the pupil position in the captured image.

[0031] The left display unit 105 is configured symmetrically to the right display unit 104, and thus has a configuration in which the camera 107 is disposed at the 4 o'clock position, and the IREDs 106 are disposed at the 1 to 3 o'clock and 5 to 12 o'clock positions, as shown in Fig. 4. However, similar to the right display unit 104, the positions and the number of the cameras 107 and the IREDs 106 are not limited to the above example.

[0032] Next, a flowchart of the first embodiment will be described with reference to Fig. 5. Basically, the flowchart of Fig. 5 is executed independently for the left and right display units 104 and 105, but the left and right pupil detection results are mutually used to speed up automatic IPD adjustment only in processing that utilizes the left-right symmetry of the human face and the HMD 100. Details of this will be described later.

[0033] 5 is a start step of the automatic IPD adjustment. When the user wears the HMD 100 and completes preparations to start use, the automatic IPD adjustment starts in S401. The automatic IPD adjustment may start when the user applies some action to the HMD main body 101, such as operating a button (not shown). The automatic IPD adjustment may also start when the completion of wearing by the user is detected as a start trigger from the output of the binocular camera 107 or a gyro sensor (not shown) built into the HMD main body 101. When the automatic IPD adjustment starts, the process proceeds to S402.

[0034] In S402, the eyeball observation camera 107 captures an image of the area around the user's eyeballs, and the image capture time is stored. The image capture time is used to calculate the duration of eye closure when eye closure is detected, as described below, so it does not need to be a precise time, and may be, for example, a counter value internally used for timing purposes. After capturing an image of the area around the eyeballs in S402, the process proceeds to S403.

[0035] S403 is a process for detecting the pupil from the captured image of the user's eyeball surroundings acquired in S402. FIG. 6(a) is an example of a schematic diagram of an image of the eyeball surroundings captured by the camera 107. As shown in FIG. 6(a), the pupil is darker than other parts and has a circular or elliptical shape, and this is used to detect the position of the pupil in the image. On the other hand, for a user wearing glasses in particular, a ghost may appear in the captured image due to the positional relationship between the IRED 106, the camera 107, and the eyeball and the glasses, making it difficult to identify the position of the pupil. FIG. 6(b) shows an image captured by the camera 107 when a ghost appears. Also, if the camera 107 captures an image at the timing of a blink, the pupil will not be captured, and in these cases, pupil detection may not be possible. As described above, if the pupil position can be identified, the process proceeds to S404, and if not, the process proceeds to S408. S404 to S407 in the case where the pupil position can be identified first will be described.

[0036] In S404, as already described, the control unit 330 calculates the necessary drive amount. The necessary drive amount may be calculated using the position of the pupil in the captured image detected in S403, the expected pupil position in the captured image when the IPD is adjusted correctly, and the correlation (sensitivity) between the drive amount of the display unit and the pupil position in the captured image. Ideally, it is desirable to drive to the true IPD adjustment position in one detection, but in reality, due to various error factors, multiple drives may be required. For example, when the drive distance is large, the error in the sensitivity, which is the correlation between the drive amount of the display unit and the pupil position in the captured image, becomes large, so that it may not be possible to drive to the appropriate position in one drive. In addition, a situation is assumed in which the drive target value does not completely converge to the true IPD position due to factors such as noise of the camera 107 and backlash of the display unit drive system. Therefore, if the necessary drive amount is within the range of ±Th (Th is a predetermined value), it may be considered that the IPD adjustment is completed as it has been driven to a position sufficiently close to the true IPD position.

[0037] S404 to S407 are the steps for carrying out this series of controls, where the drive amount is calculated in S404, convergence is determined in S405, and the display unit is actually driven the drive distance calculated in S404 in S406. If the convergence determination in S405 determines that the required drive amount is within ±Th (in other words, the drive distance is less than a predetermined value), it is determined that the display unit has been driven to a position nearly close to the true IPD position, and the automatic IPD adjustment ends in S407.

[0038] On the other hand, if the pupil cannot be detected in S403, the estimation unit 331 determines in S408 whether the cause (also called a factor) of the non-detection of the pupil is closing of the eyes. For example, when the eyes are closed, the image captured in S402 shows the eyelids instead of the pupil. Since the eyelids have a higher reflectivity than the pupil, a method of determining whether the eyes are closed based on the brightness of the captured image can be considered. Another method is to detect the edge of the eyelids every time the camera 107 captures an image of the periphery of the eyeball, and determine whether the eyes are closed based on the time-series changes. If closing of the eyes is detected in S408, the process proceeds to S409, and if not, the process proceeds to S411.

[0039] If it is determined in S408 that the cause of pupil detection is closed eyes, it is possible that the user has closed their eyes by blinking. Since the duration of a human blink is generally about 100 to 150 ms, it is expected that pupil detection will basically be possible if you wait as is.

[0040] Therefore, when it is determined that the eyes are closed in S408, the basic flow is to loop the photographing of the periphery of the eyeball in S402 and the pupil detection process in S403, and as soon as the eyes are no longer closed, the pupil is detected in S403 and the loop is terminated. The operation control unit 335 may stop the display units 104 and 105 and wait until the eyes are no longer closed while the user's eyes are closed and the process is looped. Also, the operation control unit 335 may drive the display units 104 and 105 toward the position of the average IPD value of a person that is considered to be closer to the IPD position of more users, for example, during the time until the eyes are closed and returned to normal. At this time, the operation control unit 335 may drive the left and right display units 104 and 105 at a predetermined speed. The average IPD value is an example of a reference position. The average IPD value is considered to be about Xr=Xl=31.5 mm, and in this embodiment, this position is defined as the average IPD value. In this case, the left and right display units 104, 105 can be moved in advance to positions predicted to be close to the user's IPD position before the eye closure is released, thereby aiming to shorten the time required to complete automatic IPD adjustment.

[0041] On the other hand, a situation may be considered in which the judgment of eyes being closed is erroneously detected due to some factor, and the judgment of eyes being closed continues. For example, when an unspecified number of people use the HMD 100, they may use a sanitary cloth eye mask to prevent the adhesion of sebum, sweat, makeup, etc., but it is conceivable that the position of the eye mask may shift and affect the image captured by the camera 107. In preparation for such a situation, in S409, the control unit 330 calculates the duration after the occurrence of eye closure and compares it with a preset T_blink. T_blink is an example of a first specified value. The duration of eyes being closed is determined by using the difference between the shooting time information acquired together with the image in S402, and T_blink may be set to a value with a margin for the duration of a typical human blink. For example, when T_blink=400 ms is set, if the state of eyes being closed continues for more than 400 ms, it can be determined in S409 that some abnormality has occurred.

[0042] If the duration of eye closure is equal to or less than T_blink in S409 (ie, equal to or less than the first specified value), the process returns to S402 and the camera 107 attempts to detect the pupil again, but if the duration of eye closure is greater than T_blink, the process proceeds to S410.

[0043] S410 is a step of notifying the user of an error screen. For example, in S410, the operation control unit 335 may display an error screen on the left and right display units 104 and 105. The error screen of S410 may be displayed by the left and right display units 104 and 105 in the form of a message, an icon, or the like, to the effect that the automatic IPD adjustment could not be completed.

[0044] Additionally, the left and right display units 104 and 105 may display possible guidance for the user to improve the error under the control of the operation control unit 335. The guidance here may be, for example, guidance to prompt the user to check that the center line of the HMD 100 is aligned with the bridge of the user's nose and that there is no significant misalignment when using the above-mentioned eye mask. After notifying the user of the error in S410, the process proceeds to S407, where the automatic IPD adjustment ends.

[0045] As described above, S408 to S410 are processes when closing of the eyes is detected in S408. However, if it is determined in S408 that the reason for non-detection of the pupil is not closing of the eyes, the process proceeds to S411. If the reason for non-detection of the pupil is not closing of the eyes even though the pupil is not detected, it is highly likely that the cause is a ghost of the IRED 106. In particular, for a user wearing glasses, a ghost may appear in the image captured by the camera 107, making it difficult to identify the position of the pupil. Since the ghost occurs due to the positional relationship between the IRED 106, the camera 107, and the eyeball and the glasses, it may be possible to improve the situation by moving the left and right display units 104 and 105. Therefore, the operation control unit 335 performs various controls while changing the positions of the left and right display units 104 and 105, and looping the process of searching for a place where the pupil can be detected in the order of S402, S403, S408, S411, and S412.

[0046] In this case, it is possible that pupil detection will be attempted at all pre-determined positions, but the pupil will not be detected at any of them. Therefore, in S412, pupil detection is performed in S402 and S403 while changing the positions of the left and right display units 104 and 105 to predetermined positions (which may exist in multiple positions). If the pupil cannot be detected at any of the positions, this is determined in S411, and after an error screen is displayed by the display units 104 and 105 in S410, the control unit 330 ends the automatic IPD adjustment in S407.

[0047] Regarding the positions to which the left and right display units 104, 105 are moved in S412, a first example is when the operation control unit 335 moves the display units 104, 105 to the position of the average human IPD value described above. In this case, the left and right display units 104, 105 can be moved to a position predicted to be close to the user's IPD position, which can shorten the time required to complete the automatic IPD adjustment.

[0048] As a second example, a method is given in which a predetermined amount of movement is determined in advance, and the operation control unit 335 drives the left and right display units 104, 105 by a predetermined amount of movement relative to the position where the pupil could not be detected. For example, if the position X_error where the pupil could not be detected initially is outside the position Xr=Xl=31.5 mm, the position is shifted inward by a predetermined amount of movement to search for a point where the pupil can be detected. Then, an operation of repeating this until Xl=Xr=X_min is reached is conceivable. If X_error is inside the position Xr=Xl=31.5 mm, the operation control unit 335 may control the position to be shifted outward by a predetermined amount of movement A to search for a point where the pupil can be detected.

[0049] As a third example, a method is given in which a number of points are determined in advance between X_min and X_max, and scanning is performed in order. For example, as shown in Fig. 7(a), the operation control unit 335 may move the display units 104 and 105 in order at the predetermined points L1 to L4 and R1 to R4, searching for a position where the pupil can be detected. In this case, L1=R1=X_min, L4=R4=X_max. In the example of Fig. 7(a), the operation control unit 335 moves the display units 104 and 105 in order from the outside to the inside, searching for a position where the pupil can be detected without being affected by ghosts.

[0050] Furthermore, because a human face is nearly symmetrical, and the HMD 100 is similarly symmetrical, if a ghost appears at position L1 in Fig. 7(a), there is a high possibility that a ghost will also appear at position R1. By utilizing this symmetry, if pupils cannot be detected on either side and the cause is not eye closure, performing the scan operation asymmetrically will likely enable faster discovery of a position where no ghost appears.

[0051] For example, consider a case where the operation control unit 335 starts scanning from the outermost R4 (=X_max) for the right display unit 104 and the innermost L1 (=X_min) for the left display unit 105, as shown in Fig. 7(b). If the right display unit 104 can detect the pupil at the initial R4 position, the left display unit 105 may also be more likely to be able to detect the pupil at L4 position (i.e., the symmetrical position of R4). Therefore, the operation control unit 335 can aim to shorten the time until the pupil can be detected by immediately moving the position of the left display unit 105 to L4 and attempting pupil detection.

[0052] Although not shown in Fig. 5, as an additional example utilizing symmetry, consider a case in Fig. 7(b) where only the right display unit 104 is able to detect the pupil at some position first, and the IPD position calculated from the pupil detection result is Xr=Ra. In this case, the operation control unit 335 may drive the left display unit 105 to Xl=La, which is a symmetrical position of Ra, and end the automatic IPD adjustment. In this way, more efficient pupil detection may be possible by aligning the position of one of the left and right display units 104, 105 with the position of the other.

[0053] Although several examples have been given so far regarding the method of driving the display units 104 and 105 in S412, a combination of these examples may be used. For example, control may be performed such that pupil detection is first attempted at the position of the average human IPD value, and if pupil detection at the average IPD value is NG, each position is scanned in the order of R4 to R1.

[0054] Second embodiment Hereinafter, a flowchart according to the second embodiment of the present invention will be described with reference to Fig. 8. In the second embodiment, the external view and the hardware configuration such as the driving mechanism are similar to those in the first embodiment, so the description will be omitted.

[0055] The flowchart of the second embodiment (FIG. 8) is also almost the same as the flowchart of the first embodiment (FIG. 5), and steps with the same last two digits in their step numbers, such as S404 and S704, basically have the same processing content. The difference is that the contents of the determination processing between S408 and S708, and the flow path of processing from S709 to S712 are different from S409 to S412. Since S701 to S707 are the same as S401 to S407, the processing from S708 onwards will be explained.

[0056] S708 is a step for determining whether the reason for non-detection of the pupil is a ghost, when the pupil is not detected in S703. For example, when a ghost is present, the ghost appears as a high-output area close to saturation in the image captured in S702. In other words, the estimation unit 331 may determine that a ghost is present when pixels having an output equal to or greater than a reference value (above the second specified value), which is an example of the second specified value, are distributed over a range of a predetermined number of pixels on the image captured by the camera 107.

[0057] Alternatively, since the IRED 106 is the source of the ghost, the estimation unit 331 may determine the ghost from the difference between the images captured by the camera 107 when the IRED 106 is turned on (when turned on) and when it is not turned on (when turned off).

[0058] Furthermore, since closed eyes and ghost images are major factors in causing non-detection of the pupil, a method can be considered in which an eye closure determination is made in the same manner as in S408, and if it is determined that the eyes are not closed, then the presence of a ghost image can be determined simply based on this.

[0059] If it is determined in S708 that the cause of the non-detection of the pupil is a ghost, the process proceeds to S711, and if it is determined that the cause is not a ghost, the process proceeds to S709.

[0060] S711 and S712 are processes performed when it is determined that a ghost exists, i.e., the same processes as S411 and S412. In S711, similar to S411, pupil detection is attempted at multiple positions, and if the pupil cannot be detected at any position, an error screen is displayed on the display units 104 and 105 in S710, and the automatic IPD adjustment ends.

[0061] On the other hand, if it is determined in S708 that the cause of non-detection of the pupil is not a ghost, then in S709, the pupil detection process is looped in the order of S702, S703, S708, and S709 for the duration of T_blink, as in S409. If the duration of eye closure exceeds T_blink, an error screen is displayed on the display units 104 and 105 in S710, and the automatic IPD adjustment ends in S707.

[0062] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0063] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-mentioned embodiments are supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0064] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0065] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0066] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0067] [Configuration 1] A control device for controlling an operation of a head-mounted display device, The control device includes: an estimation unit that estimates a reason why a pupil of a user wearing the display device cannot be detected when the pupil of the user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device; an operation control unit that controls an operation of driving a display unit included in the display device based on the factor estimated by the estimation unit; A control device comprising:

[0068] [Configuration 2] The factors include at least one of eye closure and ghosting, 2. The control device according to configuration 1.

[0069] [Configuration 3] The pupil detection unit includes a camera. the camera acquires an image including light emitted from a light-emitting unit of the display unit and reflected by the pupil, and detects the pupil of the user based on the image. The control device according to configuration 2.

[0070] [Configuration 4] The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is that the user has closed his / her eyes, based on the brightness included in the image obtained by the camera. The control device according to configuration 3.

[0071] [Configuration 5] The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is closing the eyes, based on a time-series change in an edge of the eyelid of the user included in the image obtained by the camera. 5. The control device according to configuration 3 or 4.

[0072] [Configuration 6] The operation control unit is stopping driving the display unit when the duration of the eye closure is equal to or less than a first specified value; When the duration of the eye closure exceeds the first specified value, control is performed so as to display an error screen on a display unit of the display device. A control device according to any one of configurations 3 to 5.

[0073] [Configuration 7] The operation control unit is When the duration of the eye closure is equal to or shorter than a first specified value, the position of the display unit is driven to a reference position that is held in advance; When the duration of the eye closure exceeds the first specified value, control is performed so as to display an error screen on a display unit of the display device. A control device according to any one of configurations 3 to 5.

[0074] [Configuration 8] The estimation unit is When the camera does not detect the pupil of the user, and when it is determined that the cause of the inability to detect the pupil of the user is not closing of the eyes, it is determined that the cause of the inability to detect the pupil of the user is a ghost. The control device according to any one of configurations 3 to 7.

[0075] [Configuration 9] The estimation unit is when the camera does not detect the pupil of the user, based on a distribution of pixels in the image obtained by the camera whose output is equal to or greater than a second specified value, determine whether or not the cause of the failure to detect the pupil of the user is a ghost. A control device according to any one of configurations 3 to 8.

[0076] [Configuration 10] The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is a ghost, based on a difference between an image obtained when the light-emitting unit is turned on and an image obtained when the light-emitting unit is turned off. A control device according to any one of configurations 3 to 8.

[0077] [Configuration 11] The operation control unit is When the estimation unit estimates that the cause of the failure to detect the pupil of the user is a ghost, the position of the display unit is driven to the reference position; The camera again detects the pupils of the user at the position of the display unit driven by the operation control unit. 8. The control device according to configuration 7.

[0078] [Configuration 12] The operation control unit is When the estimation unit estimates that the cause of the inability to detect the pupil of the user is a ghost, the estimation unit drives the display unit by a predetermined movement amount; The camera again detects the pupils of the user at the position of the display unit after being driven by the operation control unit. 11. The control device according to any one of configurations 3 to 10.

[0079] [Configuration 13] The operation control unit is When the estimation unit estimates that the cause of the failure to detect the pupil of the user is a ghost, the display unit is driven toward a predetermined position among a plurality of predetermined positions where the camera has not performed a process to detect the pupil of the user; The camera again detects the pupil of the user at the predetermined position of the display unit driven by the operation control unit. 11. The control device according to any one of configurations 3 to 10.

[0080] [Configuration 14] The operation control unit is driving the right eye side and the left eye side of the display unit in asymmetric directions; 14. The control device according to configuration 13.

[0081] [Configuration 15] The operation control unit is When the camera detects a pupil on either the right eye side or the left eye side of the display unit, the other display unit is driven so that the pupil is positioned symmetrically to the one of the display units. 15. The control device according to configuration 13 or 14.

[0082] [method] A method for controlling an operation of a head-mounted display device, comprising: When a pupil of a user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device, estimating a cause of the failure to detect the pupil of the user; an operation control unit that controls an operation of driving a display unit included in the display device based on the cause; A control method implemented by a computer, comprising:

[0083] [program] Computer, A control device for controlling an operation of a head-mounted display device, an estimation unit that estimates a reason why a pupil of a user wearing the display device cannot be detected when the pupil of the user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device; an operation control unit that controls an operation of driving a display unit included in the display device based on the factor estimated by the estimation unit; A program for causing the device to function as a control device.

Claims

1. A control device for controlling an operation of a head-mounted display device, an estimation unit that estimates a reason why a pupil of a user wearing the display device cannot be detected when the pupil of the user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device; an operation control unit that controls an operation of driving a display unit included in the display device based on the factor estimated by the estimation unit; A control device comprising:

2. The factors include at least one of eye closure and ghosting, The control device according to claim 1 .

3. The pupil detection unit includes a camera. the camera acquires an image including light emitted from a light-emitting unit of the display unit and reflected by the pupil, and detects the pupil of the user based on the image. The control device according to claim 2.

4. The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is that the user has closed his / her eyes, based on the brightness included in the image obtained by the camera. The control device according to claim 3.

5. The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is closing the eyes, based on a time-series change in an edge of the eyelid of the user included in the image obtained by the camera. The control device according to claim 3.

6. The operation control unit is When the duration of the eye closure is equal to or less than a first specified value, driving of the display unit is stopped; controlling the display device so as to display an error screen on a display unit of the display device when the duration of the eye closure exceeds the first specified value; The control device according to claim 3.

7. The operation control unit is When the duration of the eye closure is equal to or less than a first specified value, the position of the display unit is driven to a reference position that is held in advance; controlling the display device so as to display an error screen on a display unit of the display device when the duration of the eye closure exceeds the first specified value; The control device according to claim 3.

8. The estimation unit is When the camera does not detect the pupil of the user, and when it is determined that the cause of the inability to detect the pupil of the user is not closing of the eyes, it is determined that the cause of the inability to detect the pupil of the user is a ghost. The control device according to claim 3.

9. The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is a ghost, based on a distribution of pixels having an output equal to or greater than a second specified value, which are included in the image obtained by the camera; The control device according to claim 3.

10. The estimation unit is when the camera does not detect the pupil of the user, it is determined whether or not the cause of the failure to detect the pupil of the user is a ghost, based on a difference between an image obtained when the light-emitting unit is turned on and an image obtained when the light-emitting unit is turned off. The control device according to claim 3.

11. The operation control unit is When the estimation unit estimates that the cause of the failure to detect the pupil of the user is a ghost, the position of the display unit is driven to the reference position; The camera again detects the pupils of the user at the position of the display unit driven by the operation control unit. The control device according to claim 7.

12. The operation control unit is When the estimation unit estimates that the cause of the inability to detect the pupil of the user is a ghost, the estimation unit drives the display unit by a predetermined movement amount; The camera again detects the pupils of the user at the position of the display unit after being driven by the operation control unit. The control device according to claim 3.

13. The operation control unit is When the estimation unit estimates that the cause of the failure to detect the pupil of the user is a ghost, the display unit is driven toward a predetermined position among a plurality of predetermined positions where the camera has not performed a process to detect the pupil of the user; The camera again detects the pupil of the user at the predetermined position of the display unit driven by the operation control unit. The control device according to claim 3.

14. The operation control unit is driving the right eye side and the left eye side of the display unit in asymmetric directions; The control device according to claim 13.

15. The operation control unit is When the camera detects a pupil on either the right eye side or the left eye side of the display unit, the other display unit is driven so that the pupil is positioned symmetrically to the one of the display units. The control device according to claim 13 or 14.

16. A method for controlling an operation of a head-mounted display device, comprising: When a pupil of a user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device, a cause of the failure to detect the pupil of the user is estimated; an operation control unit that controls an operation of driving a display unit included in the display device based on the cause; A control method implemented by a computer, comprising:

17. Computer, A control device for controlling an operation of a head-mounted display device, an estimation unit that estimates a cause of failure to detect the pupil of a user wearing the display device when the pupil of the user wearing the display device cannot be detected by a pupil detection unit of a display unit of the display device; an operation control unit that controls an operation of driving a display unit included in the display device based on the factor estimated by the estimation unit; A program for causing the device to function as a control device.

Images