Display device, control method, and program

The HMD system addresses positioning challenges by using a combination of eyeball position detection and user reactions to adjust display elements, enhancing image clarity and detection accuracy.

JP2024163623APending Publication Date: 2024-11-22CANON KK

Patent Information

Application Number
JP2023079399
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing head-mounted displays (HMDs) face challenges in accurately positioning display images and members relative to the user's eyeballs due to variations in eyeball distances, leading to poor image quality and reduced detection accuracy of eyeball positions.

Method used

The HMD employs a system with display means, driving means, and acquisition means to adjust the position of display members and images based on eyeball positions, using first and second drives to align them accurately, and incorporates user reactions to refine the positioning.

Benefits of technology

This approach allows for precise placement of display images and members relative to the eyeballs, improving image quality and eyeball position detection accuracy by reducing misalignment and blurring.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163623000001_ABST
    Figure 2024163623000001_ABST
Patent Text Reader

Abstract

To provide a technique that can arrange display images, lenses, or displays at more appropriate positions.SOLUTION: A display device has display means that displays an image. The display device moves at least one of a first member of the display means and a display position of the image, when the positions of a user's eyeballs are in a predetermined state, executes first drive of moving at least one of the display position of the image and the first member to a first target position to change the state of the eyeball positions to a non-predetermined state, controls the display means to display at least one of an index indicating a position gazed by the user and an instruction to the user after the first drive, and consequently executes second drive of moving at least one of the display position of the image and the first member to a second target position based on the user's response.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a display device, a control method, and a program. [Background technology]

[0002] A head mounted display (hereinafter, referred to as "HMD") is equipped with display components (such as eyepieces and displays) to enable a user to see an image. In general, if there is a large difference between the position of the displayed image (or the display components) and the position of the user's eyeball, the image quality of the image viewed by the user will deteriorate.

[0003] In addition, the distance between the left and right eyeballs of a user varies from person to person. In order to prevent deterioration of image quality caused by this variation, it is necessary to adjust the position of the display image or the display member according to the distance between the left and right eyeballs. Patent Document 1 describes an HMD that includes a camera that acquires the eyeball position and adjusts the position of the display image and the display member according to the acquired eyeball position. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-232718 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to adjust the position of the display image or the display member with high accuracy, it is necessary to detect the position of the eyeball with high accuracy. In order to detect the position of the eyeball, it is necessary to display instructions or indicators to the user. However, since such display is performed before the position adjustment of the display member, the above-mentioned positional deviation may be large. In such a case, the visibility of the instructions or indicators may be poor, and the user may not be able to properly view the instructions or indicators, which may deteriorate the accuracy of detecting the position of the eyeball. As a result, there are cases where the display image or the display member cannot be positioned in an appropriate position.

[0006] Therefore, an object of the present invention is to provide a technique that enables a display image, a lens, a display, or the like to be positioned more appropriately with respect to the eyeball. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a composition comprising the steps of: A display means for displaying an image; a driving means for moving at least one of a first member of the display means and a display position of the image; An acquisition means for acquiring an eyeball position of a user; a control means for controlling the driving means to execute a first drive for moving at least one of the display position of the image and the first member to a first target position when the eyeball position is in a predetermined state, thereby causing the eyeball position to be in a state other than the predetermined state; and for controlling the display means to display at least one of an indicator indicating the position at which the user is gazing and an instruction to the user after the first drive, and then controlling the driving means to execute a second drive for moving at least one of the display position of the image and the first member to a second target position based on a reaction of the user. The display device is characterized by having:

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: A method for controlling a display device having a display means for displaying an image, comprising the steps of: a driving step of moving at least one of a first member of the display means and a display position of the image; An acquisition step of acquiring an eyeball position of a user; a first control step of causing the eyeball position to be in a predetermined state by controlling to execute a first drive in the drive step for moving at least one of the display position of the image and the first member to a first target position when the eyeball position is in the predetermined state, and a second control step of controlling the display means to display at least one of an index indicating a position where the user is gazing and an instruction to the user after the first drive, and then controlling the display means to execute a second drive in the drive step to move at least one of a display position of the image and the first member to a second target position based on a reaction of the user; The control method is characterized by having the following features. Effect of the Invention

[0009] According to the present invention, a display image, a lens, a display, or the like can be positioned at a more appropriate position relative to the eyeball. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating the configuration of an HMD according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating the configuration of an HMD according to the first embodiment. [Diagram 3] FIG. 1 is a diagram for explaining a problem that occurs in an HMD. [Figure 4] 4 is a flowchart of a position adjustment process according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing a state before a first drive according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing a state after a first driving according to the first embodiment. [Figure 7] 10 is a flowchart of a position adjustment process according to the second embodiment. [Figure 8]FIG. 11 is a diagram showing a state after a first driving according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings.

[0012] <Embodiment 1> First, an HMD 1 according to the first embodiment will be described. The structure of the HMD 1 will be described with reference to Figs. 1A, 1B, 2A, and 2B. Fig. 1A is an external view of the HMD 1. Fig. 1B is an external view of an HMD main body 11 included in the HMD 1. Fig. 2A is a diagram showing the configuration of the HMD 1 (display 121a, display 121b, eyepiece 122a, eyepiece 122b, eyeball camera 123a, and eyeball camera 123b). Fig. 2B is an image of the left eyeball 91a and the right eyeball 91b.

[0013] In the following, as shown in Figures 1A and 1B, the left-right direction as seen by the user is referred to as the "X direction," the up-down direction is referred to as the "Y direction," and the front-back direction (line of sight) is referred to as the "Z direction."

[0014] The HMD 1 is a display device worn on the head of a user. As shown in Fig. 1A, the HMD 1 has an HMD main body 11 and a wearing mechanism 13. The HMD main body 11 is held by the wearing mechanism 13 and is placed in front of the user's eyes. The wearing mechanism 13 is a headband-like member that is fixed to the user's head.

[0015] As shown in FIG. 1B, the HMD main body 11 displays an image corresponding to the left eyeball of the user (left The HMD main body 11 has a display unit 12a that displays an image corresponding to the user's right eye (an image for the right eye), and a display unit 12b that displays an image corresponding to the user's right eye (an image for the right eye). The display units 12a and 12b each have a display member. The HMD main body 11 also has a rotation motor 14a as a drive unit that drives the display unit 12a in the X direction, and a rotation motor 14b as a drive unit that drives the display unit 12b in the X direction.

[0016] 2A is a diagram showing display members included in display unit 12a and display unit 12b. Display unit 12a on the left side has display 121a and eyepiece 122a. Display unit 12b on the right side has display 121b and eyepiece 122b. In the following, display 121a and eyepiece 122a are collectively referred to as "display member 120a." Display 121b and eyepiece 122b are collectively referred to as "display member 120b."

[0017] Display 121a and display 121b each display a display image. A user views the display image through eyepiece 122a and eyepiece 122b. Left display unit 12a and right display unit 12b each have a plurality of IRED elements 124 that project infrared light toward the user's eyeball. Left display unit 12a has an eyeball camera 123a (imaging unit) that observes the left eyeball. Right display unit 12b has an eyeball camera 123b (imaging unit) that observes the right eyeball.

[0018] 2B shows an image of the user's left eyeball 91a captured by eyeball camera 123a and an image of the user's right eyeball 91b captured by eyeball camera 123b. Eyeball camera 123a observes the outline of left pupil 911a and reflected infrared light to obtain the two-dimensional or three-dimensional relative position of eyeball 91a with respect to display unit 12a. Eyeball camera 123b, like eyeball camera 123a, also obtains the two-dimensional or three-dimensional relative position of eyeball 91b with respect to display unit 12b.

[0019] In the above configuration, the HMD 1 drives the rotation motor 14a and the rotation motor 14b according to the positions of the eyeballs 91a and 91b to move the display units 12a and 12b in the X direction, respectively. This allows the display member 120a to be accurately positioned in front of the left eyeball 91a, and the display member 120b to be accurately positioned in front of the right eyeball 91b. When the display members 120a and 120b are driven by the rotation motors 14a and 14b, the display image also moves in the X direction at the same time.

[0020] Here, the above-mentioned problems with the HMD 1 will be described with reference to Figures 3A to 3D. Figure 3A is a diagram of the display members (display members 120a and 120b) and the eyeballs (eyeballs 91a and 91b) viewed from the Y direction. Figure 3B shows a display image IM1a displayed on the display 121a and a display image IM1b displayed on the display 121b.

[0021] FIG. 3C shows a perceived image IM2a and a perceived image IM2b. The perceived image IM2a is an image recognized by the user's left eyeball 91a through the eyepiece 122a. The perceived image IM2b is an image recognized by the user's right eyeball 91b through the eyepiece 122b. Since the user views the display image IM1a through the eyepiece 122a, the user recognizes that the perceived image IM2a is displayed at a position farther away from the eyeball 91a than the display 121a, as shown in FIG. 3A. Similarly, since the user views the display image IM1b through the eyepiece 122b, the user recognizes that the perceived image IM2b is displayed at a position farther away from the eyeball 91a than the display 121a, as shown in FIG. 3A.

[0022] 3D shows an image IM3 that the user recognizes when the user simultaneously sees the displayed image IM1a and the displayed image IM1b through the eyeball 91a and the eyeball 91b. In other words, the image IM3 is the image that the user recognizes when the perceived image IM2a and the perceived image IM2b are synthesized in the user's brain.

[0023] 3A, display member 120a (display 121a and eyepiece 122a) is disposed in front of left eyeball 91a in the Z direction with no misalignment in the X direction. On the other hand, display member 120b (display 121b and eyepiece 122b) is disposed in front of right eyeball 91b in the Z direction with a large misalignment in the X direction.

[0024] When the user's left eyeball 91a views the display image IM1a displayed on the display 121a through the eyepiece 122a, the image is perceived as a perceived image IM2a. When the user's right eyeball 91b views the display image IM1b displayed on the display 121a through the eyepiece 122b, the image is perceived as a perceived image IM2b.

[0025] In the HMD 1, the center in the X direction of the display member 120a (display 121a and eyepiece lens 122a) and the center in the X direction of the display image IM1a (perceived image IM2a) always coincide with each other. Therefore, when the display member 120a is driven in the X direction, the display image IM1a (perceived image IM2a) is also driven in the X direction at the same time. Similarly, the center in the X direction of the display member 120b and the center in the X direction of the display image IM1b (perceived image IM2b) always coincide with each other. Therefore, when the display member 120b is driven in the X direction, the display image IM1b (perceived image IM2b) is also driven in the X direction at the same time.

[0026] Also, eyeball camera 123a can detect the position of pupil 911a. Eyeball camera 123a can detect the position of pupil 911b. In Fig. 3A, in order to accurately detect the user's eyeball position 92a and eyeball position 92b, eyeball angles θ1a and θ1b are required in addition to the positions of pupil 911a and pupil 911b detected by eyeball camera 123a and eyeball camera 123b.

[0027] 3B, the gaze index D1 is displayed in the center of the display image IM1a and the display image IM1b, and an instruction D2 is displayed to request the user to gaze at the gaze index D1. Note that "the user gazes at the gaze index D1" means that the user continues to look at the range of the gaze index D1 for a certain period of time or more.

[0028] When the user is gazing at the central gaze index D1, the HMD1 can assume that the eyeball angle θ1a and the eyeball angle θ1b are known. For example, the position of the eyeball 91a is approximately fixed by the goggles of the HMD1, so the distance DS1 between the perceived image IM2a and the position of the pupil 911a is approximately fixed to a specific value. Then, by referring to the captured image of the eyeball 91a, the distance DS2 between the position of the gaze point P1, which is the position of the gaze index D1, and the position of the pupil 911a can also be calculated. Then, since tan θ1a=DS2 / DS1 holds, the eyeball angle θ1a can be calculated according to the distance DS1 and the distance DS2.

[0029] Also, the size of the human eyeballs is roughly the same for adults. Therefore, the HMD 1 can detect the eyeball position 92a with high accuracy based on the general size of the eyeball, the position of the pupil 911a, and the eyeball angle θ1a. The HMD 1 can detect the eyeball position 92b with high accuracy based on the general size of the eyeball, the position of the pupil 911b, and the eyeball angle θ1b.

[0030] At this time, as shown in FIG. 3A, the display image IM1a and the display member 120a (the display 121a and the eyepiece 122a) are not shifted in the X direction with respect to the left eyeball 91a. Therefore, the user can view an image that does not impair the image quality of the displayed image IM1a, such as a perceived image IM2a shown in FIG.

[0031] On the other hand, if the display image IM1 and the display member 120b (the display 121b and the eyepiece 122b) are misaligned in the X direction with respect to the right eyeball 91b, the display performance of the image to be viewed will be degraded. There are various phenomena that can cause the degradation of the display performance, but in this case, image blurring will be described as an example. Specifically, as in the perceived image IM2b shown in FIG. 3B, the displayed gaze index D1 and instruction D2 appear blurred, resulting in degraded visibility.

[0032] In addition, humans generally have difficulty in simultaneously directing the left and right visual axes A1a and A1b outward from the state in which the left and right visual axes A1a and A1b are parallel to each other. For this reason, when the distance L1 between the eyeballs 91a and 91b is smaller than the center distance (distance between the centers) L2 between the two display members (and the display positions of the two display images), the eyeballs 91a and 91b may not be able to simultaneously gaze at the left and right gaze indicators D1. For example, when the user's dominant eye is the right eyeball 91b, only the right eyeball 91b gazes at the gaze indicator D1, and the left eyeball 91a may be directed in a direction parallel to the visual axis A1b of the right eyeball 91a, and may not be able to gaze at the gaze indicator D1 (see FIG. 3A).

[0033] 3A and 3C, the gaze point of the eyeball 91a in the perceived image IM2a and the gaze point of the eyeball 91b in the perceived image IM2a are represented as gaze points P1. In this example, the left eyeball 91a gazes at the gaze index D1, but the right eyeball 91b does not gaze at the gaze index D1. Because the left eyeball 91a does not gaze at the gaze index D1, the HMD erroneously detects the eyeball angle θ1a of the left eyeball 91a, and the detection accuracy of the eyeball position 92a decreases.

[0034] In addition, when the positions of the gaze points P1 are different between the eyeballs 91a and 91b, the user recognizes that the two images are misaligned and overlapped as shown in Fig. 3D. Furthermore, the perceived image IM2b itself viewed by the right eyeball 91b is also blurred due to a decrease in display performance.

[0035] As described above, when attempting to appropriately position the display image and the display member relative to the user's eyeball, there is a possibility that the visibility of the image may deteriorate and the accuracy of detecting the position of the eyeball may deteriorate.

[0036] Hereinafter, the process of adjusting the positions of the display members 120a and 120b in the HMD 1 (the process of adjusting the display positions of the display images IM1a and IM1b) will be described with reference to the flowchart in Fig. 4. In the following, the process of the flowchart in Fig. 4 is realized by the control unit included in the HMD 1 executing a program. Here, the control unit is realized by a processor or the like.

[0037] 5A to 5D show a state before the start of the process of the flowchart in FIG. 4. The positional relationship between the display members (display members 120a and 120b) and the eyeballs (eyeballs 91a and 91b) shown in FIG. 5A is the same as the positional relationship shown in FIG. 3A. At this point, as shown in FIG. 5B, the display image IM1a displayed on the display 121a and the display image IM1b displayed on the display 121b do not include the gaze index D1 and the instruction D2. As a result, the user recognizes an image that does not include the gaze index D1 and the instruction D2, as shown in FIG. 5C and FIG. 5D.

[0038] In step S1002, the control unit detects the eye 91a by the eye camera 123a. Then, eyeball camera 123b detects eyeball 91b. The control unit then detects eyeball position 92a and eyeball position 92b. Note that because gaze index D1 is not displayed, eyeball angle θ1a and eyeball angle θ1b are unknown. For this reason, the detection accuracy of eyeball position 92a and eyeball position 92b is low.

[0039] In step S1003, the control unit determines whether or not the eyeballs 91a and 91b are in a predetermined state. The predetermined state is, for example, one of the following three states, the first state, the second state, and the third state. Determining these states does not require high accuracy in detecting the eyeball positions, and can be achieved even if the detection accuracy is not sufficiently high.

[0040] The first state is a state in which eyeball 91a is located within the angle of view of eyeball camera 123a, and eyeball 91b is located within the angle of view of eyeball camera 123b. In other words, the first state can also be said to be a state in which eyeball 91a is captured in the captured image acquired by eyeball camera 123a, and eyeball 91b is captured in the captured image acquired by eyeball camera 123b.

[0041] The second state is a state in which the eyeball position 92a is displaced in the X direction by a predetermined amount or more from the display position of the display image IM1a, or a state in which the eyeball position 92b is displaced in the X direction by a predetermined amount or more from the display position of the display image IM1b. The second state may be a state in which the eyeball position 92a is displaced in the X direction by a predetermined amount or more from the position of the display member 120a, or a state in which the eyeball position 92b is displaced in the X direction by a predetermined amount or more from the position of the display member 120b.

[0042] The third state is a state in which the center distance L1 between the eyeballs 91a and 91b is shorter than the center distance L2 between the display members 120a and 120b. The third state may be a state in which the center distance L1 between the eyeballs 91a and 91b is shorter than the center distance L2 between the display images IM1a and IM1b.

[0043] If it is determined in step S1003 that the eyeballs 91a and 91b are in the predetermined state, the process proceeds to step S1004. If it is determined that the eyeballs 91a and 91b are not in the predetermined state, the process proceeds to step S1006.

[0044] In step S1004, the control unit calculates (determines) first target positions in the X direction of each of display members 120a and 120b in order to put eyeballs 91a and 91b in a state other than the predetermined state. Here, the control unit calculates the first target positions based on eyeball positions 92a and 92b detected in step S1002. The "first target position" is a position to which the center position of the object (display member or display image) is moved in order to eliminate the above-mentioned predetermined state.

[0045] For example, when the display member 120a and the display member 120b are moved to the first target position, the first target position is calculated so that the center distance between the display member 120a and the display member 120b is smaller than the center distance between the eyeball position 92a and the eyeball position 92b. Alternatively, the first target position is calculated so that the amount of deviation in the X direction of the display member 120a from the eyeball 91a and the amount of deviation in the X direction of the display member 120b from the eyeball 91b are smaller than a predetermined value.

[0046] In step S1005, the control unit performs a first drive to put the eyeballs 91a and 91b in a state other than the predetermined state. The first drive is a process of moving the display member 120a (display image IM1a) to a first target position by rotating the rotary motor 14a, and moving the display member 120b (display image IM1b) to the first target position by rotating the rotary motor 14b.

[0047] In step S1006, the control unit displays a display image IM1a on the display 121a and a display image IM1b on the display 121b, as shown in FIG. 6B. Each of the display images IM1a and IM1b includes a gaze index D1 and an instruction D2 for gazing at the gaze index D1. When the user gazes at the gaze index D1, the eyeball angle θ1a and the eyeball angle θ1b can be known as described above. Therefore, the control unit can detect the eyeball position 92a and the eyeball position 92b with high accuracy based on the eyeball angle θ1a and the eyeball angle θ1b.

[0048] In step S1005, the control unit drives the display member 120a and the display member 120b. Therefore, the amount of misalignment in the X direction between the left eyeball 91a and the display member 120a shown in FIG. 6A and the amount of misalignment in the X direction between the right eyeball 91b and the display member 120b shown in FIG. 3A are smaller than the amount of misalignment in the X direction between the right eyeball 91b and the display member 120b shown in FIG. 3A. As a result, the blurring of the gaze index D1 and the indication D2 is suppressed in the perceived images IM2a and IM2b visually recognized by the user shown in FIG. 6C. Also, at the time of step S1006, the distance L1 between the eyeball 91a and the eyeball 91b is larger than the center distance L2 between the display member 120a and the display member 120b. Therefore, the visual axis A1a of the eyeball 91a and the visual axis A1b of the eyeball 91b are directed inward rather than parallel. The eyeballs 91a and 91b can easily gaze at the gaze index D1. In addition, because the eyeballs 91a and 91b can gaze at the same position in the image, the user can recognize the image without blurring, as shown in FIG. 6D.

[0049] In step S1007, the control unit calculates (determines) second target positions in the X direction of the display members 120a and 120b. Here, the control unit calculates the second target positions based on the user's reaction when the gaze index D1 and the instruction D2 are displayed in step S1006. In the first embodiment, the user's reaction is the action of the user gazing at the gaze index D1. The "second target position" is a position calculated using the position of the eyeball in the X direction as a target.

[0050] In step S1007, the eyeballs 91a and 91b are properly gazing at the gaze index D1 as shown in FIG. 6A and FIG. 6C, so that the HMD1 can correctly grasp the eyeball angle θ1a and the eyeball angle θ1b. Therefore, in step S1007, the eyeball position 92a and the eyeball position 92b can be detected with higher accuracy than when the first target position is calculated in step S1004. In addition, a state in which the eyeballs 91a and 91b cannot gaze at the gaze index D1 as shown in FIG. 3A and FIG. 3C is unlikely to occur, and the accuracy of the detection of the eyeball position 92a and the eyeball position 92b is unlikely to decrease. Therefore, the second target position can be properly calculated.

[0051] In step S1008, the control unit performs a second drive. In the second drive, similarly to the first drive, the control unit rotates the rotary motor 14a to move the display member 120a to the second target position, and rotates the rotary motor 14b to move the display member 120b to the second target position.

[0052] In step S1009, the control unit determines whether the amount of misalignment in the X direction of the display member 120a (display image IM1a) relative to the left eyeball 91a and the amount of misalignment in the X direction of the display member 120b (display image IM1b) relative to the right eyeball 91b are within the target range. If it is determined that the two misalignment amounts are within the target range, the process proceeds to step S1010. On the other hand, if it is determined that at least one of the two misalignment amounts is outside the target range, the process returns to step S1007. In this case, the processes of steps S1007 and S1008 are executed again.

[0053] When the process proceeds to step S1010, the display member 120a and the display member 120b The position adjustment of the display image IM1a and the display image IM1b has also been completed.

[0054] In step S1010, the control unit displays the gaze indicator D1 on each of the display 121a and the display 121b.

[0055] In step S1011, the control unit acquires eyeball information of the user. Recent HMDs have many functions that use the user's gaze information, and the user's eyeball information may be required to calculate the gaze information. The eyeball information may be, for example, information on the diameter of the eyeball or information on the radius of the cornea. In step S1011, the control unit acquires eyeball information of eyeball 91a and eyeball information of eyeball 91b based on images captured by eyeball camera 123a and eyeball camera 123b of eyeball 91a and eyeball 91b.

[0056] The control unit repeatedly displays the gaze index D1 in step S1010 while changing the display position of the gaze index D1 and acquires eyeball information in step S1011. At the start of the process in step S1010, the display member 120a is arranged with little misalignment in the X direction relative to the eyeball 91a, and the display member 120b is arranged with little misalignment in the X direction relative to the eyeball 91b. This prevents degradation of display performance, and the user can view the gaze index D1 without blurring.

[0057] The operation of the HMD 1 will be described below. When the positional deviation in the X direction between the eyeball position 92a and the display member 120a or the positional deviation in the X direction between the eyeball position 92b and the display member 120b is large, the visibility of the gaze index D1 and the instruction D2 is low when the gaze index D1 and the instruction D2 are displayed (see FIG. 3D). In addition, since either the eyeball 91a or the eyeball 91a cannot gaze appropriately at the gaze index D1, the position detection accuracy of the eyeball position 92a or the eyeball position 92b may be deteriorated.

[0058] In contrast, before the start of the flowchart of Fig. 4 according to the first embodiment (before the first drive), the display image IM1a and the display image IM1b do not include the gaze index D1 and the instruction D2 as shown in Fig. 5B. Therefore, as shown in Fig. 5D, the image recognized by the user has better visibility than the image shown in Fig. 3D.

[0059] After the first drive is performed in step S1005, as shown in FIG. 6A, the X-direction deviation between the eyeball position 92a and the position of the display member 120a and the X-direction deviation between the eyeball position 92b and the position of the display member 120b are reduced. This improves the visibility of the gaze index D1 and the instruction D2. Since the gaze index D1 and the instruction D2 are displayed from this point, the image recognized by the user shown in FIG. 6D has better visibility than the image shown in FIG. 3D.

[0060] In addition, since the gaze index D1 can be gazed at appropriately, it is possible to suppress a decrease in the accuracy of detection of the eyeball positions 92a and 92b. Then, in the second drive, the display members (the display members 120a and 120b) and the display images (the display images IM1a and IM1b) can be arranged at more appropriate positions.

[0061] In the HMD1, the center of the display member 120a in the X direction always coincides with the center of the display image IM1a in the X direction, and the center of the display member 120b in the X direction always coincides with the center of the display image IM1b in the X direction. The display position of the display image IM1a also moves together with the display member 120a, and the display position of the display image IM1b also moves together with the display member 120b. However, only one of the display member 120a and the display image IM1a may move, and only one of the display member 120b and the display image IM1b may move. A method of driving only the display position of the display image IM1a (display image IM1b) For example, a method of changing the position where an image is displayed on a display may be used.

[0062] Note that an image including both the gaze index D1 and the instruction D2 is displayed in the HMD 1. However, an image including only one of the gaze index D1 and the instruction D2 may be displayed.

[0063] 3A, the display member 120a (display image IM1a) is misaligned in the X direction with respect to the eyeball position 92a, or the display member 120b (display image IM1b) is misaligned in the X direction with respect to the eyeball position 92a. Also, the center distance between the left and right eyeballs is smaller than the center distance between the left and right display members (display positions of the left and right display images). In such a state, the visibility of the image deteriorates and the detection accuracy of the eyeball position deteriorates.

[0064] For this reason, it is preferable that the above state is set as a predetermined state, and the first drive is performed when this state is reached. Alternatively, the state in which the eyeballs are positioned within the angles of view of eyeball camera 123a and eyeball camera 123b may be set as a predetermined state, and the first drive may be performed without fail.

[0065] In the first drive of the HMD1, the first target position of the display member (display image) is calculated based on the eyeball position. Although a method of setting the average user's eyeball position as the first target position without using the eyeball position is also conceivable, this manual method increases the possibility that the position of the display member (display image) will be inappropriate after the first drive. For this reason, in the first drive, it is preferable to calculate the first target position of the display member (display image) based on the eyeball position.

[0066] Furthermore, if the center-to-center distance between the display members 120a and 120b (between the display images IM1a and IM1b) is greater than the center-to-center distance between the eyeball positions 92a and 92b, the possibility of not being able to gaze at the index properly increases, as described above. For this reason, it is preferable to calculate a "first target position that allows the center-to-center distance between the display members 120a and 120b (between the display images IM1a and IM1b) to be smaller than the center-to-center distance between the eyeball positions 92a and 92b."

[0067] In the HMD1, in the second drive, the second target position is calculated based on the user's reaction acquired when at least the display image IM1a and the display image IM1b are displayed. The user's reaction is the user's action of gazing at the gaze indicator D1. This can improve the accuracy of position detection of the eyeball position 92a and the eyeball position 92b, and in the second drive, the display member 120a and the display member 120b can be more appropriately positioned with respect to the eyeball 91a and the eyeball 91b.

[0068] From the above, in the second drive, it is preferable to calculate the second target position based on the user's reaction obtained when at least the display image IM1a and the display image IM1b are displayed. Furthermore, it is preferable that the display image includes the gaze index D1, and the user's reaction is the user's action of gazing at the gaze index D1.

[0069] The HMD1 measures the eyeball information of the user after performing the second drive as in step S1010 and step S1011. When measuring the eyeball information, the HMD1 displays the indexes multiple times to have the user gaze at the indexes. At this time, since it is after the second drive, the display member 120a is arranged with little misalignment with the eyeball 91a, and the display member 120b is arranged with little misalignment with the eyeball 91b. Therefore, even in the gaze index D1 displayed when measuring the eyeball information, the deterioration of display performance is suppressed and it can be viewed without blurring. For the above reasons, it is preferable to measure the eyeball information of the user after the second drive.

[0070] In the above description, in the first driving and the second driving, the display member 120a and the display Although the display member 120b is driven simultaneously, only one of the display member 120a and the display member 120b may be driven. For example, when only the display member 120a is driven, the predetermined state is a state in which the eyeball 91a is disposed within the angle of view of the eyeball camera 123a, or a state in which the eyeball position 92a is shifted in the X direction by a predetermined amount or more from the position of the display member 120a. In addition, the case where the position of the display member 120a is adjusted is described as a case where the positions of both the display 121a and the eyepiece 122a are adjusted, but the position of either the display 121a or the eyepiece 122a may be adjusted. Similarly, when the position of the display member 120b is adjusted, the position of either the display 121b or the eyepiece 122b may be adjusted.

[0071] <Embodiment 2> Next, a description will be given of the HMD 2 according to the embodiment 2. The HMD 2 has a display member 120a (display 121a and eyepiece 122a) and a display member 120b (display 121b and eyepiece 122b) similar to the HMD 1. The description and structure of each member in the HMD 2 are the same as those in the HMD 1, and therefore will not be repeated.

[0072] The position adjustment of the display members 120a and 120b in the HMD 2 will be described with reference to Fig. 7. Steps S2002 to S2005 in Fig. 7 are the same as steps S1002 to S1005 in Fig. 4, and therefore will not be described.

[0073] At the end of step S2005, the first drive is completed as shown in Fig. 8A. In Fig. 8A, the amount of deviation in the X direction of the display member 120a (display image IM1a) relative to the eyeball 91a is smaller than the amount of deviation in the X direction of the display member 120a (display image IM1a) relative to the left eyeball 91a shown in Fig. 3A. In Fig. 8A, the amount of deviation in the X direction of the display member 120b (display image IM1b) relative to the eyeball 91b is also smaller than the amount of deviation in the X direction of the display member 120b (display image IM1b) relative to the right eyeball 91b shown in Fig. 3A.

[0074] In step S2006, the control unit displays a display image IM1a on the display 121a and a display image IM1b on the display 121b, as shown in FIG. 8B. The display images IM1a and IM1b each include an instruction D3 for requesting selection of information for the user wearing the HMD2. Blurring is suppressed in the display images IM1a and IM1b viewed by the user shown in FIG. 8C. In addition, the user can recognize images without misalignment as shown in FIG. 8D by simultaneously viewing the display images IM1a and IM1b with the eyeballs 91a and 91b.

[0075] In step S2007, the control unit accepts an operation by the user to select his / her own information (user information).

[0076] In step S2008, the control unit calculates a second target position when driving the display member 120a and the display member 120b in the X direction. Here, the control unit calculates the second target position based on the user's reaction when the instruction D3 is displayed in step S2006. The user's reaction according to the second embodiment is an operation of selecting user information. By selecting the user's information, the HMD 2 can refer to eyeball information (user's eyeball information measured in advance) associated with the user's information. In this way, the control unit can detect the positions of the eyeball position 92a and the eyeball position 92b with high accuracy by using the eyeball information measured in advance, without "displaying a gaze index to have the user gaze at it". Therefore, the control unit can appropriately calculate the second target position. Note that the eyeball information of multiple users is stored in a storage unit such as a memory after being measured in advance. Therefore, the control unit acquires eyeball information corresponding to the selected user information from the eyeball information of multiple users stored in the storage unit.

[0077] In step S2009, the control unit performs the second driving, similarly to step S1008.

[0078] In step S2010, the control unit determines whether or not the amount of misalignment in the X direction of the display member 120a (display image IM1a) relative to the eyeball 91a is within the target range, as in step S1009. The control unit also determines whether or not the amount of misalignment in the X direction of the display member 120b (display image IM1b) relative to the eyeball 91b is within the target range. If it is determined that the two misalignment amounts are within the target range, the process of this flowchart ends. If it is determined that at least one of the two misalignment amounts is outside the target range, the process returns to step S2008.

[0079] The following describes the operation of the HMD 2. In the second drive, the HMD 2 calculates a second target position based on the user's reaction obtained when the display image IM1a and the display image IM1b are displayed.

[0080] In the HMD2, unlike the HMD1, the user's reaction is the selection of the user's information. By using the user's eyeball information linked to the selected user's information, the detection accuracy of the eyeball position 92a and the eyeball position 92b can be improved. Therefore, in the second drive, the display members 120a and the display members 120b can be more appropriately positioned with respect to the eyeballs 91a and 91b. Also, in the second drive, the display images IM1a and IM1b (perceptual images IM2a and IM2b) can be more appropriately positioned with respect to the eyeballs 91a and 91b.

[0081] In each embodiment, the display member 120a and the display member 120b are a lens (optical element) and a display (retinal projection type), but are not limited to this. The display member 120a and the display member 120b may be a laser light source and a mirror (MEMS mirror) (retinal scanning type). In this case, the light emitted by the laser light source is reflected by the mirror and then reaches the retina of the user's eye. As a result, the display member 120a and the display member 120b draw (display) an image on the retina of the user's eye. Instead of a mirror, any optical element having optical power (for example, capable of changing the trajectory of light) may be used, for example, a prism or a diffractive optical element may be used.

[0082] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0083] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0084] Each functional unit in each of the above embodiments (variations) may or may not be an individual piece of hardware. Each of the functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0085] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0086] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A display means for displaying an image; a driving means for moving at least one of a first member of the display means and a display position of the image; An acquisition means for acquiring an eyeball position of a user; a control means for controlling the driving means to execute a first drive for moving at least one of the display position of the image and the first member to a first target position when the eyeball position is in a predetermined state, thereby causing the eyeball position to be in a state other than the predetermined state; and for controlling the display means to display at least one of an indicator indicating the position at which the user is gazing and an instruction to the user after the first drive, and then controlling the driving means to execute a second drive for moving at least one of the display position of the image and the first member to a second target position based on a reaction of the user. A display device comprising: (Configuration 2) The driving means moves a display position of the image, the first driving and the second driving are drivings for moving a display position of the image; 2. The display device according to configuration 1. (Configuration 3) The driving means moves the first member, The first drive and the second drive are drives that move the first member. 3. The display device according to configuration 1 or 2. (Configuration 4) the acquiring means acquires the eyeball position based on an image of the eyeball of the user captured by an imaging means; The predetermined state is a state in which the eyeball is disposed within the angle of view of the imaging means. 4. The display device according to any one of configurations 1 to 3. (Configuration 5) The predetermined state is a state in which the eyeball position is shifted by a predetermined amount or more in a predetermined direction with respect to at least one of the display position of the image and the first member. 4. The display device according to any one of configurations 1 to 3. (Configuration 6) the first member has a second member for the user's left eye and a third member for the user's right eye; The display means displays a first image for the user's right eye and a second image for the user's left eye. a first image and a second image; The predetermined state is a state in which the distance between the center of the right eyeball and the center of the left eyeball is smaller than at least one of the distance between the center of the first image and the center of the second image and the distance between the center of the second member and the center of the third member. 4. The display device according to any one of configurations 1 to 3. (Configuration 7) The display device described in any one of configurations 1 to 6, characterized in that, in the second drive, the control means determines the second target position based on a reaction of the user when at least one of the indicator and the instruction to the user is displayed. (Configuration 8) the control means controls the display means to display the indicator after the first driving; The user's reaction is an action of the user gazing at the indicator. 8. The display device according to any one of configurations 1 to 7. (Configuration 9) the control means controls the display means to display an instruction requesting selection of user information after the first driving; The user's response is a selection of the user information by the user. 8. The display device according to any one of configurations 1 to 7. (Configuration 10) The device further includes a storage means for storing eyeball information of a plurality of users, In the second driving, the control means 1) acquires eye information corresponding to the selected user information from the storage means, and 2) controls the drive means to move at least one of a display position of the image and the first member to the second target position based on the acquired eye information. 10. The display device according to configuration 9. (Configuration 11) Further comprising a measuring means for measuring eyeball information of the user after the second driving. 11. The display device according to any one of configurations 1 to 10. (Configuration 12) The control means controls the display means so as not to display any of an indicator indicating a position where the user is gazing and an instruction to the user before the first driving. 12. The display device according to any one of configurations 1 to 11. (Configuration 13) The display device is a head-mounted display that is worn on the user's head. 13. The display device according to any one of configurations 1 to 12. (method) A method for controlling a display device having a display means for displaying an image, comprising the steps of: a driving step of moving at least one of a first member of the display means and a display position of the image; An acquisition step of acquiring an eyeball position of a user; a first control step of causing the eyeball position to be in a predetermined state by controlling to execute a first drive in the drive step for moving at least one of the display position of the image and the first member to a first target position when the eyeball position is in the predetermined state, and a second control step of controlling the display means to display at least one of an index indicating a position where the user is gazing and an instruction to the user after the first drive, and then controlling the display means to execute a second drive in the drive step to move at least one of a display position of the image and the first member to a second target position based on a reaction of the user; A control method comprising the steps of: (program) A program for causing a computer to function as each of the means of the display device according to any one of configurations 1 to 13. [Explanation of symbols]

[0087] 1: HMD (head mounted display), 120a, 120b: display members, 121a, 121b: display, 122a, 122b: eyepieces, 123a, 123b: eyeball camera, 14a, 14b: Rotary motor

Claims

1. A display means for displaying an image; a driving means for moving at least one of a first member of the display means and a display position of the image; An acquisition means for acquiring an eyeball position of a user; 1) when the eyeball position is in a predetermined state, controlling the driving means to execute a first drive to move at least one of the display position of the image and the first member to a first target position, thereby causing the eyeball position to be in a state other than the predetermined state; 2) a control means for controlling the display means to display at least one of an indicator indicating the position at which the user is gazing and an instruction to the user after the first drive, and then controlling the driving means to execute a second drive to move at least one of the display position of the image and the first member to a second target position based on a reaction of the user; A display device comprising:

2. The driving means moves a display position of the image, the first driving and the second driving are drivings for moving a display position of the image; 2. The display device according to claim 1 .

3. The driving means moves the first member, The first drive and the second drive are drives that move the first member.

2. The display device according to claim 1 .

4. the acquiring means acquires the eyeball position based on an image of the eyeball of the user captured by an imaging means; The predetermined state is a state in which the eyeball is disposed within the angle of view of the imaging means.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

5. The predetermined state is a state in which the eyeball position is shifted by a predetermined amount or more in a predetermined direction with respect to at least one of the display position of the image and the first member.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

6. the first member includes a second member for the user's left eye and a third member for the user's right eye; the display means displays a first image for the user's right eye and a second image for the user's left eye; The predetermined state is a state in which a distance between a center of a right eyeball and a center of a left eyeball is smaller than at least one of a distance between a center of the first image and a center of the second image and a distance between a center of the second member and a center of the third member.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

7. The display device according to any one of claims 1 to 3, characterized in that, in the second drive, the control means determines the second target position based on a reaction of the user when at least one of the indicator and an instruction to the user is displayed.

8. the control means controls the display means to display the indicator after the first driving; The user's reaction is an action of the user gazing at the indicator.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

9. the control means controls the display means to display an instruction requesting selection of user information after the first driving; The user's response is a selection of the user information by the user.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

10. The device further includes a storage means for storing eyeball information of a plurality of users, In the second driving, the control means 1) acquires eye information corresponding to the selected user information from the storage means, and 2) controls the drive means to move at least one of a display position of the image and the first member to the second target position based on the acquired eye information.

10. The display device according to claim 9.

11. Further comprising a measuring means for measuring eyeball information of the user after the second driving.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

12. the control means controls the display means so as not to display any of an indicator indicating a position where the user is gazing and an instruction to the user before the first driving.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

13. The display device is a head-mounted display that is worn on the user's head.

4. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

14. A method for controlling a display device having a display means for displaying an image, comprising the steps of: a driving step of moving at least one of a first member of the display means and a display position of the image; An acquisition step of acquiring an eyeball position of a user; a first control step of causing the eyeball position to be in a predetermined state by controlling to execute a first drive in the drive step for moving at least one of a display position of the image and the first member to a first target position when the eyeball position is in the predetermined state; a second control step of controlling the display means to display at least one of an index indicating a position where the user is gazing and an instruction to the user after the first drive, and then controlling the display means to execute a second drive in the drive step to move at least one of a display position of the image and the first member to a second target position based on a reaction of the user; A control method comprising the steps of:

15. A program for causing a computer to function as each of the means of the display device according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Head-mounted image display apparatus

    JP2010232718A

Cited By

  • Display device and method for setting display device

    WO2026141513A1