Display apparatus and display method

Abstract

A display device capable of accurately detecting a relative position between a light projection system and an eyepiece optical system is provided. A display device according to the present technology includes: a light projection system including a light source; and an eyepiece optical system that guides the light projected from the light projection system to an eyeball, in which the eyepiece optical system includes: an optical element; the light projection system includes an optical element, and three or more feature points, a relative position of the feature points with respect to the optical element being fixed, and includes an acquisition unit that acquires at least two-dimensional position information of each of at least three feature points among the three or more feature points. According to the display device of the present technology, it is possible to provide a display device capable of accurately detecting the relative position between a light projection system and an eyepiece optical system.

Description

TECHNICAL FIELD

[0001] The technology according to the present disclosure (hereinafter also referred to as "the present technology") relates to a display device and a display method. BACKGROUND

[0002] Conventionally, a display device that displays an image by irradiating an eyeball of an observer with light (specifically, image light) is known (for example, see Patent Literature 1).

[0003] For example, in the display device described in Patent Literature 1, a relative position between a scanning unit (light projection system) and a deflection unit (eyepiece optical system) is detected from two-dimensional position information of each of two characteristic points provided in a contact lens, and the scanning unit is controlled on the basis of the detection result.

[0004] LIST OF CITATIONS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: WO 2009 / 066446 A SUMMARY

[0007] PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] However, in the conventional display device, there is room for improvement in accurately detecting a relative position between a light projection system and an eyepiece optical system.

[0009] Thus, a main object of the present technology is to provide a display device capable of accurately detecting a relative position between a light projection system and an eyepiece optical system.

[0010] SOLUTION TO PROBLEM

[0011] The present technology provides a display device including:

[0012] a light projection system including a light source; and

[0013] an eyepiece optical system that guides light projected from the light projection system to an eyeball,

[0014] wherein the eyepiece optical system includes:

[0015] an optical element; and

[0016] three or more characteristic points whose relative positions with respect to the optical element are fixed, and

[0017] an acquisition unit that acquires at least two-dimensional position information of each of at least three characteristic points among the three or more characteristic points.

[0018] The at least three feature points can not be on the same straight line.

[0019] The optical element can be provided in a contact lens attached to the eyeball. The at least three feature points can be provided in the contact lens.

[0020] The acquisition unit can acquire at least two-dimensional position information of each of the at least three feature points even when an orientation of the eyeball changes.

[0021] The optical element can be provided in a lens mounted on a spectacle frame.

[0022] All of the at least three feature points can be provided in the lens or the spectacle frame, or a part of the at least three feature points can be provided in the lens and the other part can be provided in the spectacle frame.

[0023] The at least three feature points and an optical center of the optical element can be located on the same plane.

[0024] The acquisition unit can include at least one camera.

[0025] The at least one camera can include a visible light camera and / or a TOF camera.

[0026] The at least one camera can include a plurality of cameras that constitute a stereo camera.

[0027] The light projection system can include the acquisition unit.

[0028] The acquisition unit can include an infrared light source, the camera can be sensitive to an infrared wavelength, and the at least three feature points can be made of a retroreflective material.

[0029] The light projection system can include a projection optical system arranged on an optical path of light from the light source, and an optical axis of the projection optical system and an optical axis of the camera can be coaxial.

[0030] The light projection system can include a projection optical system arranged on an optical path of light from the light source, and an optical axis of the projection optical system and an optical axis of the camera can be coaxial.

[0031] The projection optical system can include a movable deflection element and a focusing lens, the acquisition unit can acquire two-dimensional position information of each of the at least three feature points in a plane orthogonal to an optical axis direction of the camera, and distance information of the camera from each of the at least three feature points with respect to the optical axis direction of the camera, and the control unit can concurrently perform the following operations: controlling the movable deflection element based on the two-dimensional position information; and controlling the light source based on at least the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information.

[0032] At least two of the at least three feature points can be integrally provided.

[0033] The light projection system and the eyepiece optical system can be separated from each other.

[0034] The present technology also provides a display method including:

[0035] acquiring at least two-dimensional position information of each of at least three of three or more feature points, the relative positions of the feature points with respect to an optical element that guides light projected from a light projection system to an eyeball being fixed; and

[0036] controlling a part of the light projection system based on the result of the acquisition in the acquisition step.

[0037] The light projection system can include a light source, a movable deflection element, and a focusing lens, the step of acquiring at least two-dimensional position information can include a step of acquiring two-dimensional position information of each of the at least three feature points in a plane orthogonal to an optical axis direction of the camera, and a step of acquiring distance information of the camera from each of the at least three feature points with respect to the optical axis direction of the camera, and the control step can include a step of controlling the movable deflection element based on at least the two-dimensional position information, and a step of controlling the light source based on the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a diagram for explaining the principle of the present technology.

[0039] Figure 2 is a diagram for explaining a method of calculating a normal vector.

[0040] Figure 3 is a diagram for explaining a display device according to a first embodiment of the present technology.

[0041] Figure 4 A ~ Figure 4C is a diagram for explaining a configuration example 1 of an eyepiece optical system of a display device in Figure 3

[0042] Figure 5 A ~ Figure 5 C is a diagram for explaining a configuration example 2 of an eyepiece optical system of a display device in Figure 3

[0043] Figure 6 A ~ Figure 6 C is a diagram for explaining a configuration example 3 of an eyepiece optical system of a display device in Figure 3

[0044] Figure 7 A ~ Figure 7 E is a diagram for explaining a configuration example 4 of an eyepiece optical system of a display device in Figure 3

[0045] Figure 8 A ~ Figure 8 E is a diagram for explaining a configuration example 5 of an eyepiece optical system of a display device in Figure 3

[0046] Figure 9 is a diagram for explaining an incident angle dependency of a diffraction efficiency in a HOE.

[0047] Figure 10 is a block diagram illustrating a basic configuration example of control of a display device in Figure 3

[0048] Figure 11 is a flowchart illustrating a flow of overall control in the basic configuration example of Figure 10

[0049] Figure 12 is a block diagram illustrating a configuration example 1 of control of a display device in Figure 3

[0050] Figure 13 is a flowchart illustrating a flow of overall control in the configuration example 1 of Figure 12

[0051] Figure 14 is a flowchart illustrating a movable mirror control process of the configuration example 1 of Figure 12

[0052] Figure 15 is a flowchart illustrating a focusing lens control process of the configuration example 1 in Figure 12

[0053] Figure 16 is a flowchart illustrating​​​​​​​​​​​Figure 12 The flowchart of the projection image control processing in Example 1 is shown.

[0054] Figure 17 It is a diagrammatic explanation. Figure 3 Example 2 shows a block diagram illustrating the control structure of a display device.

[0055] Figure 18 It is a diagrammatic explanation. Figure 17 The flowchart of the overall control process in Example 2.

[0056] Figure 19 This is a diagram illustrating a display device according to a second embodiment of the present technology.

[0057] Figure 20 This is a flowchart illustrating the control flow of a display device according to a second embodiment of the present technology.

[0058] Figure 21 This is a diagram illustrating a display device according to a third embodiment of the present technology.

[0059] Figure 22 This is a diagram illustrating a display device according to a fourth embodiment of the present technology.

[0060] Figure 23 This is a diagram illustrating a display device according to a fifth embodiment of the present technology.

[0061] Figure 24 It is a diagram used to illustrate the principle of a stereo camera.

[0062] Figure 25 This is a diagram illustrating a display device according to a sixth embodiment of the present technology.

[0063] Figure 26 This is a diagram used to illustrate retroreflection.

[0064] Figure 27 This is a block diagram illustrating an example of the structure of a TOF camera.

[0065] Figure 28 This is a block diagram illustrating an example of the components of an event camera.

[0066] Figure 29 A and Figure 29 B is used for explanation. Figure 3 Examples of the configuration of the eyepiece optical system of the display device are shown in Figures 6 and 7. Detailed Implementation

[0067] Hereinafter, preferred embodiments of the present technology will be described in detail with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same function are denoted with the same reference numerals, and redundant description is omitted. The embodiments described below provide representative embodiments of the present technology, and the scope of the present technology should not be construed in a narrow sense by these embodiments. In this specification, even in a case where a display device and a display method according to the present technology are described as exhibiting a plurality of effects, the display device and the method according to the present technology only need to exhibit at least one effect. The effects described in this specification are merely examples and are not limiting, and there can be additional effects.

[0068] Further, the description will be provided in the following order.

[0069] 0. INTRODUCTION

[0070] 1. Display device according to first embodiment of the present technology

[0071] 2. Display device according to second embodiment of the present technology

[0072] 3. Display device according to third embodiment of the present technology

[0073] 4. Display device according to fourth embodiment of the present technology

[0074] 5. Display device according to fifth embodiment of the present technology

[0075] 6. Display device according to sixth embodiment of the present technology

[0076] 7. Effects of display device according to the present technology

[0077] 8. Variations of the present technology

[0078] <0. INTRODUCTION>

[0079] Conventionally, a retinal direct drawing device (display device) is known, which displays an image by projecting light (specifically, image light) from a light projection system and guiding the projected light to a pupil of an observer through an eyepiece optical system attached to the observer. In particular, in order to realize a retinal direct drawing device in which a positional relationship (relative position) between the light projection system and the eyepiece optical system is not fixed, it is necessary to recognize a state of the observer from the light projection system side. Such a retinal direct drawing device has a feature that the relative position between the eyepiece optical system and the light projection system changes in real time in accordance with movement of the observer, and thus it is desirable to mainly perform the following recognition and correction.

[0080] • Recognizing a position of an optical center of an optical element viewed from the light projection system

[0081] In the case of a Maxwellian optical system, the observer cannot see the image unless the optical axis of the projection passes through the pupil. Usually, it is desirable to identify the position (e.g., two-dimensional position information) of the optical center of the optical element from the light projection system side because the optical elements are arranged so that their optical centers overlap with the pupil of the observer.

[0082] • Correction of the angle (attitude) of the optical element with respect to the optical axis of the projection

[0083] The diffraction efficiency of a diffractive element (e.g., a holographic element (HOE)) as an optical element has an incident angle dependency, and when the incident angle of the projection light on the HOE changes, the diffraction efficiency changes independently for each of RGB. In order to correctly reproduce the image and the brightness that are desired to be displayed to the observer, it is desirable to correct the brightness of the original image in accordance with the incident angle.

[0084] Meanwhile, in the display device described in Patent Literature 1, the relative position between the scanning unit (light projection system) and the deflection unit (eyepiece optical system) is detected from two-dimensional position information of each of two characteristic points provided in the deflection unit (eyepiece optical system). However, in this display device, there is room for improvement in improving the detection accuracy of the relative position.

[0085] Thus, as a result of intensive studies, the inventors have developed a display device according to the present technology as a display device capable of accurately detecting the relative position between the light projection system and the eyepiece optical system (e.g., the relative position of the eyepiece optical system as seen from the light projection system).

[0086] In particular, in the display device according to the present technology, in the case where the optical element of the eyepiece optical system is provided in a contact lens worn on the eyeball of the observer, the contact lens and the optical element also move with the movement of the pupil. Thus, the position of the optical element can be identified by identifying the position of the pupil, and the angle (attitude) of the optical element can be identified by identifying the direction of the pupil (the direction of the line of sight). As a result, the relative position between the light projection system and the eyepiece optical system can be more accurately detected.

[0087] In addition, in the display device described in Patent Literature 1, it is considered that a configuration in which the scanning unit (light projection system) is mounted on the head of the observer is assumed, and it is extremely difficult to cope with a configuration in which the scanning unit (light projection system) and the deflection unit (eyepiece optical system) are spaced apart from each other by a long distance in space and the positional relationship dynamically changes, and the detection accuracy is required.

[0088] On the other hand, in the display device according to the present technology, a change in the relative position between the spatially separated (separate) light projection system and the eyepiece optical system can be accurately detected, and thus, such a configuration in which the positional relationship of the light projection system and the eyepiece optical system dynamically changes can be sufficiently dealt with.

[0089] Figure 1 is a diagram for explaining the principle of the present technology. As Figure 1 In the present technology, as illustrated in Figure 1 In the example of, the three feature points FP are provided in the eyeglass lens GL equipped with the optical element OE. At least one of the three feature points FP can be provided in the eyeglass frame GF, for example. These three feature points FP are recognized by the camera C mounted on the light projection system LPS spatially separated from the eyepiece optical system EOS. By measuring the following spatial coordinates and calculating the normal vector, the three-dimensional coordinates (x, y, z) of the optical center OC (optical center) of the optical element OE as seen from the light projection system LPS and the angle (θx, θy) of the optical element OE as seen from the light projection system LPS can be detected in real time. Note that θx represents the rotation angle (pitch) around the x axis extending to the left and right of the observer. θy represents the rotation angle (yaw) around the y axis extending vertically of the observer. Note that in the present specification and the drawings, unless otherwise specified, "x" and "X" are used with the same meaning, "y" and "Y" are used with the same meaning, and "z" and "Z" are used with the same meaning.

[0090] (Measurement of spatial coordinates)

[0091] For example, the three-dimensional coordinates (x, y, z) of each of the three feature points FP are acquired. If the positional relationship between the three feature points FP and the optical center OC of the optical element OE is known in advance, the three-dimensional coordinates (x, y, z) of the relative position of the optical center OC of the optical element OE as seen from the light projection system LPS can be calculated by interpolation or extrapolation calculation of the three points. The three-dimensional coordinates (x, y, z) of each feature point FP are acquired using the camera C. The main acquisition method is a stereo camera method, but a configuration using only a TOF camera or a configuration using a visible light camera and a TOF camera in combination can be considered. The stereo camera method includes a method using an infrared camera, an infrared light source, and a retroreflective material in addition to a visible light camera.

[0092] (Calculation of normal vector)

[0093] Figure 2is a diagram for explaining a method of calculating a normal vector. From the three-dimensional coordinates (x, y, z) of each of the three feature points FP acquired as described above, a normal vector of a plane including the three feature points FP is calculated using a vector cross product operation. This normal vector corresponds to the direction of the optical element OE, and by taking an inner product with an axis vector of a reference coordinate system set in the light projection system LPS, the angle (θx, θy) of the optical element OE as seen from the light projection system LPS can be calculated.

[0094] When the spatial coordinate vectors of the three feature points FP are P[0], P[1], and P[2], respectively, the normal vector N of the plane including the three feature points can be calculated using a vector cross product as in the following equation (1).

[0095] N = (P[2] - P[0]) x (P[1] - P[0]) · · · (1)

[0096] The angles θx, θy, and θz formed by the x-axis, y-axis, and z-axis with the normal vector N can be calculated using an inner product of vectors as in the following equations (2) to (4), where the direction vectors of the respective axes are Ex, Ey, and Ez.

[0097] N · Ex = |N||Ex|cos θx · · · (2)

[0098] N · Ey = |N||Ey|cos θy · · · (3)

[0099] N · Ez = |N||Ez|cos θz · · · (4)

[0100] From the above equations (2) to (4), θx, θy, and θz can be obtained as in the following equations (5) to (7).

[0101] θx = arccos(cos θx) · · · (5)

[0102] θy = arccos(cos θy) · · · (6)

[0103] θz = arccos(cos θz) · · · (7)

[0104] In order to perform the above arithmetic operations, three-dimensional coordinates of at least three feature points FP are acquired. In the case where four or more feature points FP are present, at least three feature points FP can be selected from among the four or more feature points FP, and the arithmetic operations can be performed.

[0105] <1. Display device according to a first embodiment of the present technology>

[0106] Figure 3 is a diagram illustrating a display device 1 according to a first embodiment of the present technology. As an example, the display device 1 is a retinal direct drawing display device that directly draws an image on a retina of a user who is an observer using light. The display device 1 includes a light projection system 10 and an eyepiece optical system 20 that guides light projected from the light projection system 100 to an eyeball EB. The light projection system 10 further includes an acquisition unit 300 including at least one camera 300a and a control unit 400 (see, for example, 10).

[0107] In the display device 1, as an example, the light projection system 10 and the eyepiece optical system 20 are separately (spatially separated) disposed. For example, the light projection system 10 can be placed on a table, a stand, or the like, can be hung on a wall, can be installed on a portable object such as a watch or a smartphone of the user, or can be attached to a moving body such as a car, a motorcycle, or a bicycle, on which the user rides. The eyepiece optical system 20 is disposed at or near the eyeball EB of the observer.

[0108] Image data is input to the control unit 400. The control unit 400 generates a modulation signal of each color in RGB according to the input image data, and outputs the modulation signal to a light source driving unit 100d (see, for example, 10) of the light projection system 10 described later. Figure 12 ).

[0109] Note that, in Figure 3 , the light projection system 10 includes 2 cameras 300a, but can include only 1 camera 300a or can include 3 or more cameras 300a. (This similarity applies to other drawings in which a plurality of cameras 300a are illustrated).

[0110] (light projection system)

[0111] As an example, the light projection system 10 includes an image light generating unit 100 and a projection optical system 200 that projects image light IL generated by the image light generating unit 100. As an example, the image light generating unit 100, the projection optical system 200, and the at least one camera 300a are integrally disposed in a housing H.

[0112] As an example, the image light generating unit 100 includes a light source 100a, a scanning mirror 100b, and a light source driving unit 100d (see, for example, 10). Figure 12 ​

[0113] The light source 100a includes, for example, a laser such as an edge emitting laser (LD) or a surface emitting laser (VCSEL). Here, the light source 100a includes, for example, a red laser, a green laser, and a blue laser. Note that, as the light source 100a, a light emitting diode (LED) can be used, for example. The light source 100a can include only a single-color laser or LED. Thus, the configuration of the light source 100a can be simplified.

[0114] The light source driving unit 100d generates a driving signal based on a modulation signal of each color in RGB input from the control unit 400, and applies the driving signal to a corresponding laser of the light source 100a to drive the laser. The light source 100a and the light source driving unit 100d are also collectively referred to as a "light source system". As an example, luminance information of a projected image is input from the control unit 400 to the light source driving unit 100d.

[0115] The scanning mirror 100b deflects and scans light emitted from the light source 100a. As the scanning mirror 100b, a MEMS mirror or the like that can be driven around two axes is used, for example. As a result, the number of components can be reduced and the size can be reduced. On the other hand, in a case where a mirror that can be driven around two axes is used as the scanning mirror 100b, a projected image can be distorted due to driving the mirror. To avoid this, two mirrors that can be driven around one axis, such as MEMS mirrors, can be used in combination, for example.

[0116] As an example, the projection optical system 200 includes a focusing lens 200c as a projection lens and a movable mirror 200a as a movable deflection element. The projection optical system 200 is controlled by the control unit 400. A projection window 200b is provided at a rear stage of the movable mirror 200a.

[0117] The focusing lens 200c is disposed on an optical path of the image light IL via the scanning mirror 100b. The focusing lens 200c is provided to be movable in an optical axis direction, and is driven in the optical axis direction by a focusing lens driving unit. The focusing lens driving unit is controlled by the control unit 400. In a case where an observer moves in a z-axis direction (a direction parallel to a projection optical axis), an image is not focused. Then, it is desirable to perform position adjustment (focus adjustment) of the focusing lens 200c in the optical axis direction in accordance with a position of the optical element OE in the z-axis direction, so that a focal point of the image light IL coincides with a retina of the observer.

[0118] The movable mirror 200a reflects the image light IL that has passed through the focusing lens 200c, and guides the image light IL to the projection window 200b. The image light IL that is reflected by the movable mirror 200a and that has passed through the projection window 200b is projected to the eyepiece optical system 20. The movable mirror 200a is disposed so as to be able to swing, for example, around two axes, and is driven by a movable mirror driving unit. As the movable mirror 200a, for example, a combination of two galvanomirrors that can swing around one axis, a gimbalmirror that can swing around two axes, or the like is used. The movable mirror driving unit is controlled by the control unit 400. By swinging the movable mirror 200a around two axes, the projection direction of the image light IL can be changed two-dimensionally. The light projection system 10 and the eyepiece optical system 20 are spatially separated, and the relative positional relationship dynamically changes in real time according to the orientations of the eyeball, the head, and the body of the observer. If the projection optical axis of the light projection system 10 (specifically, the optical axis of the projection optical system 200, hereinafter also simply referred to as the "projection optical axis") does not pass through the optical element OE, the observer cannot see the image. Therefore, it is preferable to control the angle (attitude) of the movable mirror 200a based on the two-dimensional coordinates (x, y) of the optical element OE, that is, the two-dimensional positional information in the plane orthogonal to the projection optical axis, so that the projection optical axis passes through the optical element OE. As a result, the observer can continuously view the image. Note that, as the movable deflection element, a mechanical deflection element can be used instead of the movable mirror.

[0119] (Eyepiece optical system)

[0120] The eyepiece optical system 20 includes the optical element OE and three or more characteristic points FP. In use of the display device 1 (when the image light IL is projected from the light projection system 10 to the eyeball EB), the eyepiece optical system 20 is located within the imaging field of view (within the angle of view) of the camera 300a.

[0121] The optical element OE is arranged so that the focal point is located near the pupil of the observer. The image light IL projected from the light projection system 10 is condensed by the optical element OE, passes through the pupil of the observer, and is directly depicted on the retina. As a result, the observer can recognize the image.

[0122] The optical element OE is provided in a contact lens CL worn on an eyeball EB of an observer. The contact lens CL can or can not be a component of the ocular optical system 20. The contact lens CL has an advantage of self-alignment in which the center of the contact lens CL moves to coincide with the center of the pupil of the eyeball EB. That is, the direction of the contact lens CL coincides with the direction of the line of sight, which is the direction of the eyeball EB. For example, in the case where the optical element OE is provided in glasses, it is possible that a positional deviation occurs between the glasses and the pupil. In this case, it is necessary to detect the positions of the glasses and the pupil and control taking account of the positional deviation, or to advance the adjustment of the glasses and the time variation to a level at which the amount of the positional deviation does not become a problem. By providing the optical element OE on the contact lens CL, it is not necessary to take account of such a positional deviation. Note that the frequency of the contact lens CL can be 0.

[0123] The optical element OE is, for example, a holographic element (HOE) and diffracts and condenses the image light IL projected from the light projection system 10 and guides the image light IL to the retina of the eyeball EB. When the angle (attitude) of the HOE with respect to the projection optical axis changes, the color of the image seen by the observer also changes. This is because the diffraction efficiency of the HOE has an incident angle dependency.

[0124] That is, the HOE has a characteristic in which the diffraction efficiency differs depending on the incident angle of light. Specifically, the diffraction efficiency of the HOE has an incident angle dependency such that light other than the image light IL projected from the light projection system 10 is transmitted without being diffracted (see the left side of the graph of FIG. 6). As a result, the observer can view the background image and the projected image in an overlapped manner like in AR. On the other hand, it is physically impossible to set the diffraction efficiency of the HOE to 1 (100%) only in a certain incident angle range (see the left side of the graph of FIG. 6), the HOE has a characteristic in which the diffraction efficiency always has a peak incident angle and decreases with the angular width from the peak incident angle (see the right side of the graph of FIG. 6). Furthermore, this characteristic is independent in RGB, and the peak angle and the half-width do not necessarily match. In order to optimize the color of the image desired to be displayed to the observer by the HOE having such a characteristic, it is necessary to identify the angle (θx, θy) of the HOE with respect to the projection optical axis and then generate a projected image taking account of the diffraction efficiency at that angle. Figure 9 Figure 9 Figure 9

[0125] (feature point)

[0126] ​​​Each feature point FP desirably has a light reflection characteristic different from that of the surrounding area, so as to be distinguished from the area in the captured image of the camera 300a. For example, each feature point FP can be made of a reflective material having a higher reflectance than the surrounding area, or can be made of a light-transmissive material or a light-absorbing material having a lower reflectance than the surrounding area. For example, each feature point FP can include an unevenness obtained by machining a part of the member (e.g., contact lenses, eyeglass lenses, eyeglass frames, etc.).

[0127] For example, a visible light camera can be used as the camera 300a, and a marker visually recognizable at a visible light wavelength can be employed as each feature point FP. By using a visible light camera, the marker can be recognized in a wide dynamic range. Further, for example, an infrared camera can be used as the camera 300a, and a marker (e.g., a dichroic film) visible at an infrared wavelength can be employed as each feature point FP.

[0128] Each feature point FP needs to have a fixed relative position with respect to the optical element OE. Then, in the display device 1, at least three feature points FP are provided on the contact lenses CL as a rigid body on which the optical element OE is provided. That is, each feature point FP and the optical element OE are integrated via the contact lenses CL.

[0129] Suppose that XYZ coordinates of the three feature points FP are P[0] to P[2] (see Figure 2 ), a coordinate of the optical center OC of the optical element OE is Hc, and position vectors of the three feature points FP and the optical center OC are P[0], P[1], P[2], and Hc, then the following equation (8) is established using a first combination of vectors.

[0130] Hc = a P[0] + b P[1] + c P[2] · · · (8)

[0131] Here, if the coordinates Po[0] to Po[2] of each feature point FP and the coordinate Hco of the optical center OC of the optical element OE in the initial state (at the time of manufacture) are known, the coefficients a to c can be obtained. Further, if each feature point FP can be arranged so that a + b + c = 1 holds, the above equation (8) always holds regardless of the position and attitude of the rigid body (e.g., the contact lenses CL). That is, the optical center Hc of the optical element OE can be obtained by calculation from the three-dimensional coordinates P[0] to P[2] of the three feature points FP recognized by the camera 300a. The condition a + b + c = 1 mentioned here means that the optical center Hc of the optical element exists in a plane specified by the three feature points FP having P[0] to P[2] as three-dimensional coordinates.

[0132] By the way, in order for the camera 300a to recognize the position information (x, y, z, θx, θy) about the 5 axes of the optical element OE, it is desirable that at least 3 feature points FP are always located at a position where the camera 300a can capture an image (however, it can be excluded that there is no need to recognize a case where the observer closes his / her eyes or blinks), that is, it is desirable that even if the direction of the eyeball EB changes, the acquisition unit 300 including the camera 300a can at least acquire the two-dimensional coordinates (x, y) of each of the at least 3 feature points FP. Ideally, it is desirable that the acquisition unit 300 including the camera 300a can at least acquire the two-dimensional coordinates (x, y) of each of the at least 3 feature points FP regardless of the direction of the eyeball EB.

[0133] Therefore, it is conceivable to design the arrangement, size, shape, color, and the like of the 3 feature points FP. In addition, for example, even if 4 or more feature points FP are arranged and a part of the feature points FP is shielded by the eyelid or the like, at least 3 feature points FP can be imaged by the camera 300a. In addition, the 3 feature points FP do not necessarily have to be physically separated (separated) from each other, but for example, can be integrally provided in a certain pattern (see Figure 29 A and Figure 29 B). In Figure 29 Example 6 of the eyepiece optical system 20 illustrated in Figure 29 Example 7 of the eyepiece optical system 20 illustrated in

[0134] In order to obtain the direction of the optical element OE using the position information of each of the 3 feature points FP, it is necessary to specify a plane by at least 3 feature points FP, and therefore at least 3 feature points FP among the 3 or more feature points FP do not need to be on the same straight line. In addition, it is preferable that the at least 3 feature points FP and the optical center OC of the optical element OE are on the same plane.

[0135] (Example 1 of the configuration of the eyepiece optical system)

[0136] Figure 4 A ~ Figure 4 C is a diagram for illustrating Example 1 of the configuration of the eyepiece optical system 20 of the display device 1. In Example 1 of the configuration of the eyepiece optical system 20, as Figure 4 A, 3 feature points FP having the same shape (for example, a circular shape) are provided in the contact lens CL so that the feature points FP are located at 3 vertices of a triangle, and are located at the pupil (iris) P, the cornea C, and the sclera S. In this case, the 3 feature points FP are arranged in a triangle shape, and the 3 feature points FP are arranged in a straight line. In addition, the 3 feature points FP are arranged in a straight line, and the 3 feature points FP are arranged in a triangle shape. Figure 4 A ~Figure 4 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right. Figure 4 B and Figure 4 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right.

[0137] (Second configuration example of eyepiece optical system)

[0138] Figure 5 A ~ Figure 5 C is a diagram for explaining a configuration example 2 of the eyepiece optical system 20 of the display device 1. In the configuration example 2 of the eyepiece optical system 20, as illustrated in Figure 5 A, three feature points FP having the same shape (for example, a circular shape) are provided in the contact lens CL so that the feature points FP are located at three vertices of a triangle and around the pupil (the darkest part in Figure 5 A ~ Figure 5 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right. Figure 4 A, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right. Figure 5 B and Figure 5 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right.

[0139] (Third configuration example of eyepiece optical system)

[0140] Figure 6 A ~ Figure 6 C is a diagram for explaining a configuration example 3 of the eyepiece optical system 20 of the display device 1. In the configuration example 3 of the eyepiece optical system 20, three relatively large feature points FP having different shapes (for example, a circular shape, a triangular shape, or a quadrangular shape) are provided in the contact lens CL so that the feature points FP are located at three vertices of a triangle and around the pupil (the darkest part in Figure 6 A ~ Figure 6 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right. Figure 6 B and Figure 6 C, the three feature points FP can be imaged by the camera 300a even if the eyeball EB moves relatively large left and right. Note that in the configuration example 3, colors can be different from each other instead of or in addition to the shapes of the at least two feature points FP.

[0141] (Fourth configuration example of eyepiece optical system)

[0142] Figure 7 A ~ Figure 7 E is a diagram for explaining a configuration example 4 of the eyepiece optical system 20 of the display device 1. In the configuration example 4 of the eyepiece optical system 20, as illustrated in Figure 7As shown in diagram A, six feature points FP with the same shape (e.g., circular) are set in the contact lens CL, such that the feature points FP are located at the six vertices of a hexagon and at the pupil ( Figure 7 A~ Figure 7 The area around the darkest part of E. The result is as follows: Figure 7 B~ Figure 7 As shown in the diagram in E, even if the eyeball EB moves relatively large up, down, left, and right, the camera 300a can still image at least 3 of the 6 feature points FP.

[0143] (Example 5 of the configuration of an eyepiece optical system)

[0144] Figure 8 A~ Figure 8 E is a diagram illustrating example 5 of the configuration of the eyepiece optical system 20 of the display device 1. In example 5 of the configuration of the eyepiece optical system 20, as... Figure 8 As shown in the diagram in A, three feature points FP with the same shape (e.g., circular) are set in the contact lens CL, which transmit visible light and reflect infrared light, such that the feature points FP are located at the three vertices of a triangle and are close to the pupil ( Figure 8 A~ Figure 8 The darkest part of E overlaps. The result is as follows: Figure 8 B~ Figure 8 As illustrated in Figure E, even if the eyeball EB moves greatly upward, downward, leftward, and rightward, the camera 300a (e.g., an infrared camera) can still image these three feature points FP.

[0145] (An example of the basic components of a display device's control system)

[0146] Figure 10 This is a block diagram illustrating an example of the basic configuration of the control system for display device 1. The acquisition unit 300 of display device 1 acquires two-dimensional position information (e.g., three-dimensional position information) of each of at least three or more feature points FP, whose relative positions with respect to the optical element OE are fixed. Figure 10 As shown in the diagram, the acquisition unit 300 includes, for example, a camera 300a and an xyz coordinate acquisition unit 300b. The control unit 400 includes a processing unit 400a and a display control unit 400b. The acquisition unit 300 and the control unit 400 are implemented, for example, by hardware such as a CPU and a chipset.

[0147] The xyz coordinate acquisition unit 300b acquires the xyz coordinates (3D position information) of each of at least three feature points FP from the captured image of the camera 300a.

[0148] The operation unit 400a calculates the angle of the movable mirror 200a, the position of the focusing lens 200c, and the brightness of the projection image based on the xyz coordinates of each of the at least three feature points FP.

[0149] The display control unit 400b controls the movable mirror 200a based on the calculation result of the angle of the movable mirror 200a, controls the focusing lens 200b based on the calculation result of the position of the focusing lens 200c, and controls the projection image based on the calculation result of the brightness of the projection image (specifically, outputs a brightness correction value based on the calculation result of the brightness of the projection image to the light source driving unit 100d).

[0150] (Flow of overall control in basic configuration example of control of display device)

[0151] Figure 11 is a flowchart illustrating Figure 10 the flow of overall control in the basic configuration example of control of the display device 1 illustrated in FIG. Figure 11 based on a processing algorithm executed by the CPU. When the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates), the series of processes in Figure 11 S1 are started.

[0152] In the first step S1, the acquisition unit 300 acquires the three-dimensional coordinates (x, y, z) of each of the at least three feature points FP.

[0153] In the next step S2, the operation unit 400a of the control unit 400 calculates the normal vector N based on the three-dimensional coordinates (x, y, z) of each of the at least three feature points FP.

[0154] In the next step S3, the operation unit 400a of the control unit 400 calculates θx, θy, and θz using the normal vector N.

[0155] In the next step S4, the operation unit 400a of the control unit 400 calculates the distance D from the camera 300a to the optical element OE (specifically, the plane specified by the three feature points FP) based on the three-dimensional coordinates (x, y, z) of each of the at least three feature points FP.

[0156] In the next step S5, the display control unit 400b of the control unit 400 controls the display of the image (specifically, controls the movable mirror 200a, the focusing lens 200c, and the brightness and shape correction of the projection image) based on x, y, θx, θy, θz, and D.

[0157] In the next step S6, the control unit 400 determines whether or not to end the processing. For example, when the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the control unit 400 negates the determination here (determination not to end the processing), and the series of processing of steps S1-S5 is performed again. For example, when the user issues an end instruction, or when the acquisition unit 300 cannot acquire the three-dimensional coordinates of each feature point FP (when each feature point FP deviates from the angle of view of the camera 300a), the control unit 400 affirms the determination here (determination to end the processing), and the flow is ended.

[0158] (stereo camera)

[0159] A stereo camera method can be introduced in which a plurality of cameras 300a are used to acquire the z coordinate (distance from the camera 300a to each feature point FP) of each feature point FP. In the stereo camera method, two cameras 300a are arranged so that their optical axes are parallel to each other, and each parameter is defined as follows (see Figure 24 ).

[0160] h: center-to-center distance between left and right cameras [mm]

[0161] f: camera lens focal length [mm]

[0162] size_x: sensor size lateral of camera [mm]

[0163] size_y: sensor size vertical of camera [mm]

[0164] width: horizontal number of pixels of sensor of camera

[0165] height: vertical number of pixels of sensor of camera

[0166] P: measured object coordinates (x, y, z) [mm]

[0167] pFP: imaging point on right camera sensor of (hr, vr) [pixels]

[0168] pl: imaging point on left camera sensor of P (hl, vl) [pixels]

[0169] In order to convert the coordinates of the imaging points to [mm], xr, yr, and xl are as follows.

[0170] xr = hr / (width x size_x)

[0171] yr = vr / (height x size_y)

[0172] xl = hl / (width x size_x)

[0173] xl = vl / (height * size_y)

[0174] According to the above, the coordinates of the measurement object P(x, y, z) can be calculated as follows.

[0175] x = (xl + xr) * h / 2 * (xl - xr)

[0176] y = yr * h / (xl - xr) = yl * h / (xl - xr)

[0177] z = f * h / (xl - xr)

[0178] In the above formulas and Figure 24 , the middle position between the left camera and the right camera is defined as the origin O. Note that the stereo camera method does not necessarily require the optical axes of the two cameras to be parallel, and the calculation formulas in the case of a non-parallel arrangement are also known, but are omitted here.

[0179] (Example 1 of the configuration of the control of the display device)

[0180] Figure 12 is a block diagram illustrating Example 1 of the configuration of the control of the display device 1.

[0181] Example 1 of the configuration of the control of the display device 1 is obtained by further embodying the configuration of the control unit 400, compared to the basic configuration example illustrated in Figure 10 . In this configuration example 1, the stereo camera SC is constituted by two cameras 300a (e.g., a visible light camera).

[0182] In configuration example 1, the arithmetic unit 400a of the control unit 400 includes an optical center xyz coordinate calculation unit 400a1 and an optical element tilt angle calculation unit 400a2, and the display control unit 400b of the control unit 400 includes a movable mirror control unit 400b1, a focus lens control unit 400b2, and a projection image control unit 400b3.

[0183] (Flow of the overall control in configuration example 1 of the control of the display device)

[0184] Figure 13 is a flowchart illustrating the flow of the overall control in configuration example 1 of Figure 12 . The flow of the overall control in configuration example 1 of the control of the display device 1 is as follows. Figure 13 The processing algorithm based on the CPU is executed. When the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates), a series of processes in Figure 13 is started.

[0185] First, in step S21, the acquisition unit 300 performs xyz coordinate acquisition processing. Specifically, the xyz coordinate acquisition unit 300b acquires the xy coordinate and the z coordinate of each of the at least three feature points FP from the captured images of the two cameras 300a and 300a that constitute the stereo camera SC.

[0186] Next, the control unit 400 performs, in parallel, movable mirror control processing in step S22-1, focus lens control processing in step S22-2, and projected image control processing in step S22-3.

[0187] Next, in step S23, the control unit 400 determines whether or not to end the processing. When the determination here is affirmative, the flow ends, whereas when the determination is negative, the processing returns to step S21, and the similar series of processing is performed again. When the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the control unit 400 negates the determination here (determination not to end the processing), and the processing of steps S21, S22-1, S22-2, and S22-3 is performed again. For example, when the user issues an end instruction, or when the acquisition unit 300 cannot acquire the three-dimensional coordinates of each feature point FP (when each feature point FP deviates from the angle of view of the camera 300a), the control unit 400 affirms the determination here (determination to end the processing), and the flow ends.

[0188] (Movable mirror control processing)

[0189] Figure 14 is a flowchart illustrating the movable mirror control processing (step S22-1). Figure 13

[0190] In a first step S22-1-1, the optical center xyz coordinate calculation unit 400al calculates the xyz coordinates of the optical center OC of the optical element OE.

[0191] In a next step S22-1-2, the movable mirror control unit 400bl determines the angle of the movable mirror 200a based on the xy coordinates of the optical center OC. Note that the z coordinate of the optical center OC can also be used for this determination.

[0192] In a final step S22-1-3, the movable mirror control unit 400bl controls the movable mirror 200a. Specifically, the movable mirror control unit 400bl sends a mirror drive signal to the movable mirror drive unit in accordance with the angle of the movable mirror 200a determined in step S22-1-2.

[0193] (Focus lens control processing)

[0194] Figure 15 is a flowchart illustrating the focus lens control processing (step S22-2). Figure 13 ​the flowchart of the focusing lens control process (step S22-2).

[0195] In a first step S22-2-1, the optical center xyz coordinate calculation unit 400al calculates the z coordinate of the optical center OC of the optical element OE.

[0196] In a next step S22-2-2, the focusing lens control unit 400b2 determines the position of the focusing lens 200c in the optical axis direction based on the z coordinate of the optical center OC.

[0197] In a final step S22-2-3, the focusing lens control unit 400b2 controls the focusing lens 200c. Specifically, the focusing lens control unit 400b2 sends a lens drive signal based on the position of the focusing lens 200c in the optical axis direction determined in step S22-2-2 to the focusing lens drive unit.

[0198] (Projected image control process)

[0199] Figure 16 is a flowchart illustrating the projected image control process (step S22-3) of the display device 1. Figure 13

[0200] In a first step S22-3-1, the optical element tilt angle calculation unit 400a2 calculates the tilt angle (θχ, θγ) of the optical element OE.

[0201] In a next step S22-3-2, the projected image control unit 400b3 determines the luminance of the projected image based on the tilt angle (θχ, θγ) of the optical element OE.

[0202] In a final step S22-3-3, the projected image control unit 400b3 controls the projected image. Specifically, the projected image control unit 400b3 sends a light source drive signal modulated in accordance with the luminance of the projected image determined in step S22-3-2 to the light source drive unit lOOd. Note that the light source drive signal can be a signal modulated in accordance with the luminance and shape correction of the projected image.

[0203] (Structure example 2 of control of display device)

[0204] Figure 17 is a block diagram illustrating structure example 2 of the control of the display device 1.

[0205] Structure example 2 of the control of the display device 1 is the same as structure example 1 of the control of the display device 1 except that the display device 1 further includes a light source drive unit lOOd. Figure 12 ​The configuration example 1 differs from the configuration example 2 shown in the diagram in that the acquisition unit 300 includes the xy coordinate acquisition unit 300c and the z coordinate acquisition unit 300d instead of the xyz coordinate acquisition unit 300b, and the operation unit 400a of the control unit 400 includes the optical center xy coordinate calculation unit 400a11 and the optical center z coordinate calculation unit 400a12 instead of the xyz coordinate calculation unit 400a1.

[0206] (Flow of overall control in configuration example 2 of control of display device)

[0207] Figure 18 is a flowchart illustrating the flow of overall control in the configuration example 2 of the control of display device. Figure 17 Figure 18 is based on a processing algorithm executed by the CPU. When the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates), the series of processes in Figure 18 are started.

[0208] First, in step S31-1, the xy coordinate acquisition process by the xy coordinate acquisition unit 300c and the z coordinate acquisition process by the z coordinate acquisition unit 300d are performed in parallel. Specifically, the xy coordinate acquisition unit 300c acquires the xy coordinate of each of the at least 3 feature points FP from the captured images of the two cameras 300a and 300a that constitute the stereo camera SC. The z coordinate acquisition unit 300d acquires the z coordinate of each of the at least 3 feature points FP from the captured images of the two cameras 300a and 300a that constitute the stereo camera SC.

[0209] Next, the control unit 400 performs the movable mirror control process in step S32-1, the focus lens control process in step S32-2, and the projection image control process in step S32-3 in parallel. Step S32-1 is similar to step S22-1 in Figure 13 . Step S32-2 is similar to step S22-2 in Figure 13 . Step S32-3 is similar to step S22-3 in Figure 13 .

[0210] ​Next, in step S33, the control unit 400 determines whether or not to end the processing each time the steps S32-1, S32-2, and S32-3 are executed. When the determination here is affirmative, the flow ends, and when the determination is negative, the processing returns to the steps S31-1 and S31-2, and similar processing is performed again. When the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the control unit 400 negates the determination (determination not to end the processing), and the processing of the steps S31-1, S31-2, S32-1, S32-2, and S32-3 is performed again. For example, when the user issues an end instruction, or when the acquisition unit 300 cannot acquire the three-dimensional coordinates of each feature point FP (when each feature point FP deviates from the angle of view of the camera 300a), the control unit 400 affirms the determination here (determination to end the processing), and the flow ends.

[0211] In the configuration example 2 of the control of the display device 1 described above, the control unit 400 controls the movable mirror 200a in parallel based on the two-dimensional position information (x, y) of each of the at least three feature points FP, controls the light source 100a based on the two-dimensional position information (x, y) and the distance information (distance in the z-axis direction) of the feature point FP, and controls the focusing lens 200c based on the distance information.

[0212] Here, generally, it is relatively easy to acquire the xy coordinates of each feature point FP, whereas the acquisition of the z coordinate requires restrictions on the equipment, arrangement, and computational load to be employed. Furthermore, the acquisition of the xy coordinates among the five axes of each feature point FP directly affects whether or not the observer can see the image, and has a higher priority than the remaining three parameters (z, θx, θy). On the other hand, the control of the projected image is limited by the frame rate of the image display, and there is a feature that high-speed control cannot be performed compared to the movable mirror 200a. Thus, in the configuration example 2, as described above, the information acquisition period of the xy coordinates of each feature point FP, the information acquisition period of the z coordinate, and the cycle period of the subsequent arithmetic operation are independent. In the processing related to the control of the movable mirror 200a (first control cycle), the xy coordinates are calculated at the fastest speed, and the projected optical axis is caused to follow the optical element OE (target). The processing that requires the z coordinate (second control cycle) updates the information at a slower period than the processing related to the control of the movable mirror 200a. The information of the xy coordinates required for the calculation of θx and θy only needs to be calculated using the latest information at the necessary timing. As described above, by dividing the control loop into two, the most important movable mirror can be controlled at high speed without being affected by the rate limiting factor of the control speed.

[0213] <2. Image display device according to a second embodiment of the present technology>

[0214] Figure 19 is a diagram illustrating a configuration example of a display device 2 according to a second embodiment of the present technology. As Figure 19 illustrated in FIG. 2, the display device 2 has a configuration generally similar to that of the display device 1 according to the first embodiment, except that the optical axis of the projection optical system 200 (projection optical axis) is coaxial with the optical axis of the camera 300a.

[0215] In the display device 2, a half mirror 200d is disposed on the optical path of the image light IL between the focusing lens 200c and the movable mirror 200a. The camera 300a is disposed facing the half mirror 200d at a position deviated from the optical path of the image light IL between the focusing lens 200c and the movable mirror 200a. In this case, the camera 300a continuously captures a position through which the projection optical axis passes within a predetermined angle of view (e.g., a central angle of view). Then, when the projection optical axis is deviated from the optical center OC of the optical element OE, the predetermined angle of view of the captured image of the camera 300a is also deviated from the optical center OC. Then, by moving the movable mirror 200a so as to eliminate the deviation, it is possible to make the projection image always continuously point to the optical center OC. Note that the "projection optical axis is coaxial with the optical axis of the camera 300a" is not limited to the case where the projection optical axis is strictly coaxial with the optical axis, but means that a deviation is allowed within a range where the same function and effect are obtained.

[0216] Figure 20 is a flowchart of a flow of the overall control of the configuration example of the display device 2 according to the second embodiment of the present technology. Figure 20 The flowchart of FIG. 4 is based on a processing algorithm executed by the CPU. When the camera 300a captures the coordinates (when the acquisition unit 300 acquires the coordinates), the series of processes in FIG. 4 is started. Figure 20

[0217] In the first step S11, the acquisition unit 300 acquires the two-dimensional coordinates (x, y) of each of the at least three feature points FP.

[0218] In the next step S12, the arithmetic unit 400a of the control unit 400 calculates the two-dimensional coordinates (x, y) of the optical center OC of the optical element OE based on the two-dimensional coordinates (x, y) of each of the at least three feature points FP.

[0219] In the next step S13, the control unit 400 determines whether the projection optical axis passes through the optical center OC of the optical element OE. Specifically, the control unit 400 determines whether there is a deviation between the projection optical axis and the optical center OC based on the position (two-dimensional coordinates) of the optical center OC in the captured image of the camera 300a. When the determination here is affirmative (when it is determined that there is no deviation), the processing returns to step S11, while when the determination is negative (when it is determined that there is a deviation), the processing proceeds to step S14. ​

[0220] In the next step S14, the control unit 400 controls the movable mirror 200a so that the projection optical axis passes through the optical center OC of the optical element OE.

[0221] In the last step S15, the control unit 400 determines whether or not to end the process. For example, when the acquisition unit 300 can acquire the three-dimensional coordinates of each feature point FP (when each feature point FP is within the angle of view of the camera 300a), the determination here is negated (the determination process is not ended), and the process returns to step Sll. On the other hand, for example, when the user issues an end instruction, or when the acquisition unit 300a cannot acquire the three-dimensional coordinates of each feature point FP (when each feature point FP deviates from the angle of view of the camera 300a), the determination here is affirmed (the determination ends the process), and the flow ends.

[0222] In the display device 2 described above, since the camera 300a is arranged to recognize the vicinity of the ocular optical system 20 via the movable mirror 200a, wide-range recognition and high-accuracy recognition can be achieved. The imaging range of the camera 300a can cover the entire range that the light projection system 10 can project. In addition, since the origin of the light projection coordinate system and the origin of the camera coordinate system coincide with each other, there is also an advantage that initial calibration can be simplified, and position deviation errors with respect to each disturbance can be suppressed. However, in this case, since recognition is performed by the movable mirror 200a, the detected coordinates are not absolute coordinates but relative coordinates. Then, when the observer is lost, it is difficult to know which direction the movable mirror 200a should be pointed. This can be solved by installing a camera for absolute coordinate recognition. As this camera, a relatively inexpensive camera can be prepared since it does not need high performance in terms of resolution and shooting speed, or a region that is not coaxial with the projection optical axis can be provided in a part of the imaging field of view of the camera 300a, and this region can be cropped out and used. Another solution is to prepare both a follow-up mode and a search mode in the method of controlling the movable mirror 200a. When the position of the observer is unknown, the search mode is set, the observer is searched by two-dimensional scanning within the movable range of the movable mirror 200a, and when the position of the observer is specified, the mode is switched to the follow-up mode, and is controlled by the above-described method. At least one camera 300a can be any type such as a visible light camera, an infrared camera, a stereo camera, or a TOF camera, and can be combined with the display device 1 according to the first embodiment.

[0223] <3. Display device according to a third embodiment of the present technology>

[0224] Figure 21is a diagram illustrating a configuration example of a display device 3 according to a third embodiment of the present technology. The display device 3 has a similar configuration to the display device 1 according to the first embodiment, except that the optical element OE and at least three feature points FP are provided in the eyeglass lens GL.

[0225] The eyeglass lens GL can or can not be a component of the ocular optical system 20. When the optical element OE is provided in the eyeglass lens GL, the pupil of the eyeball EB and the optical center OC of the optical element OE can not coincide with each other, so that there are disadvantages that the accuracy and the change over time need to be further reduced, or that the position of the pupil needs to be additionally detected and followed. On the other hand, there are advantages that the design freedom of the feature points FP is high, the detection accuracy of the feature points FP can be improved, the manufacturing is easy, the safety is high, and the wearing threshold is low. Here, for example, the HOE can be mainly used as the optical element OE, but a diffractive optical element (DOE) or the like can also be used.

[0226] In the display device 3, as an example, as illustrated in Figure 21 In the display device 3, as an example, as illustrated in

[0227] <4. Display device according to a fourth embodiment of the present technology>

[0228] Figure 22 is a diagram illustrating a configuration example of a display device 4 according to a fourth embodiment of the present technology. The display device 4 has a similar configuration to the display device 3 according to the third embodiment, except that the optical element OE is provided in the eyeglass lens GL, and at least three feature points FP are provided in the eyeglass frame GF.

[0229] In the display device 4, as an example, as illustrated in Figure 22 In the display device 4, as an example, as illustrated in

[0230] <5. Display device according to a fifth embodiment of the present technology>

[0231] Figure 23 is a diagram illustrating a configuration example of a display device 5 according to a fifth embodiment of the present technology. The display device 5 has a similar configuration to the display device 3 according to the third embodiment, except that the optical element OE and a part of at least three feature points FP are provided in the eyeglass lens GL, and the other part of the at least three feature points FP is provided in the eyeglass frame GF.

[0232] In the display device 5, as an example, as illustrated in Figure 23As illustrated in the diagram, the optical element OE and at least one feature point FP are provided in the spectacle lens GL, and at least two feature points FP are provided in the spectacle frame GF. Note that, in the display device 5, the optical element OE and at least two feature points FP can be provided in the spectacle lens GL, and at least one feature point FP can be provided in the spectacle frame GF.

[0233] <6. Display device according to a sixth embodiment of the present technology>

[0234] Figure 25 is a diagram illustrating a configuration example of a display device 6 according to a sixth embodiment of the present technology. As Figure 25 As illustrated in the diagram, the display device 6 has a configuration generally similar to that of the display device 1 according to the first embodiment, except that the acquisition unit 300 includes an infrared light source 300e (e.g., an LED, a laser, or the like) that irradiates at least three feature points FP with infrared light near the focusing lens 200c, and includes an infrared camera (a camera sensitive to infrared wavelengths) as the camera 300a. Note that the infrared light source 500 can be part of the light source 100a, or can be disposed on an optical path of light from the light source 100a.

[0235] In the display device 6, as a measure to reduce the processing amount, a visible light cut filter can be provided in the camera 300a. Here, a marker made of a retroreflective material is used as each feature point FP. As Figure 26 As illustrated in the diagram, unlike specular reflection (regular reflection) in a plane mirror, a marker made of a retroreflective material has a characteristic of having the same direction of incident light and reflected light. By utilizing this characteristic, in principle, even if the observer moves greatly, the strong light reflected by the marker always returns to the periphery of the projection optical system 200 (specifically, the infrared light source 300e), so that the position of the marker can be accurately detected. In addition, since the display device 6 has the infrared light source 300e as an illumination light source, even when the observer is in a dark environment, the position of the marker can be detected.

[0236] (TOF camera)

[0237] Figure 27 is a block diagram illustrating a configuration example of a time-of-flight (TOF) camera as the camera 300a.

[0238] As Figure 27 As illustrated in the diagram, the TOF camera as the camera 300a includes an emission unit 300a1, a light-receiving unit 300a2, and a distance calculation unit 300a3. As a measurement method of the TOF camera, a direct TOF method can be used, or an indirect TOF method can be used.

[0239] For example, a TOF camera can be used to acquire the z coordinate (distance) of each feature point FP, and can be used in combination with a visible light camera or an infrared camera for acquiring the xy coordinates of the feature point FP. Note that in the case of a TOF camera with high xy resolution, all xyz coordinates can be acquired by the TOF camera, and thus, due to the reduction in the number of components, a reduction in size, weight, and cost can be expected.

[0240] (Event camera)

[0241] Figure 28 is a block diagram illustrating a configuration example of a camera 300a (event camera) including an event sensor.

[0242] As Figure 28 As illustrated in

[0243] The event sensor is a sensor that asynchronously detects a change in luminance of each pixel and outputs only the changed data in combination with coordinate and time information to achieve high-speed and low-latency data output. Since the event camera including the event sensor can reduce the amount of information to be processed while capturing a change in a feature point at high speed compared to a normal camera, a tracking experience with less discomfort can be achieved, and at the same time, by reducing the processing load in the subsequent stage, it is helpful to reduce the size and weight and reduce power consumption.

[0244] <7. Effects of the display device according to the present technology>

[0245] The display device according to the above-described present technology (for example, any one of the display devices 1 to 6 according to the first to sixth embodiments) includes a light projection system 10 including a light source 100a and an eyepiece optical system 20 that guides light projected from the light projection system 10 to an eyeball EB. The eyepiece optical system 20 includes an optical element OE and three or more feature points FP whose relative positions with respect to the optical element OE are fixed. The light projection system 10 includes an acquisition unit 300 that acquires at least two-dimensional position information of each of at least three feature points FP among the three or more feature points.

[0246] The display device according to the present technology can accurately detect at least two-dimensional information of each of at least three feature points FP.

[0247] As a result, the display device according to the present technology can provide a display device capable of accurately detecting the relative position between the light projection system 10 and the eyepiece optical system 20. Since the display device according to the present technology can accurately detect the relative position, a display device with excellent image followability with respect to the eyepiece optical system 20 worn by the observer can be realized. The display device according to the present technology has excellent image followability even in a use mode in which the separate light projection system and the eyepiece optical system are arranged apart from each other in space (for example, separated from each other compared to a case in which both the light projection system and the eyepiece optical system are worn by the observer), and the positional relationship between the light projection system and the eyepiece optical system dynamically changes in real time.

[0248] The display method according to the present technology (for example, a display method performed using any one of the display devices 1 to 6 according to the first to sixth embodiments) includes a step of acquiring at least two-dimensional position information of each of at least three feature points FP out of the three or more feature points FP, the relative positions of which with respect to the optical element OE that guides the light projected from the light projection system 10 to the eyeball EB are fixed, and a step of controlling the light projection system 10 based on the result of the acquisition in the acquisition step.

[0249] In the display method according to the present technology, at least two-dimensional information of each of the at least three feature points FP can be accurately detected.

[0250] As a result, the display method according to the present technology can provide a display method capable of accurately detecting the relative position between the light projection system 10 and the eyepiece optical system 20. Since the display method according to the present technology can accurately detect the relative position, a display method with excellent image followability with respect to the eyepiece optical system 20 worn by the observer can be realized. The display method according to the present technology has excellent image followability even in a use mode in which the separate light projection system and the eyepiece optical system are arranged apart from each other in space (for example, separated from each other compared to a case in which both the light projection system and the eyepiece optical system are worn by the observer), and the positional relationship between the light projection system and the eyepiece optical system dynamically changes in real time.

[0251] <8. Variations of the Present Technology

[0252] The display device according to each of the embodiments of the present technology described above and the display method using the same can be appropriately changed.

[0253] For example, in the display device according to each of the above embodiments, the acquisition unit 300 can acquire only two-dimensional position information of each of the at least three feature points FP. Even in this case, by controlling the movable mirror 200a on the basis of the two-dimensional position information, it is possible to depict an image on the eyeball EB of the user who is the observer.

[0254] For example, in the display device according to each of the above embodiments, the acquisition unit 300 can acquire at least two-dimensional position information (for example, only two-dimensional position information, two-dimensional position information and distance information) of each of the at least four feature points FP. In this case, it is possible to obtain the relative position between the light projection system 10 and the eyepiece optical system 20 more accurately.

[0255] For example, in the display device according to each of the above embodiments, the movable deflection element (for example, the movable mirror 200a) is controlled on the basis of two-dimensional coordinates (x, y), but the rotatable deflection element can be controlled on the basis of three-dimensional coordinates (x, y, z).

[0256] For example, in the display device according to each of the above embodiments, the projection optical system 200 can not include the focusing lens 200c and the focusing lens driving unit.

[0257] For example, in the display device according to each of the above embodiments, the control unit 400 can not include the focusing lens control unit and / or the projection image control unit.

[0258] For example, in the display device according to each of the above embodiments, only one of the position control of the focusing lens and the brightness control of the projection image can be performed.

[0259] For example, in the display device according to each of the above embodiments, the control unit 400 can control only one of the light source 100a and the projection optical system 200.

[0260] For example, in the display device according to each of the above embodiments, the image light generating unit is scanning type, but can also be non-scanning type, for example, including a liquid crystal display or the like.

[0261] For example, in the display device according to each of the above embodiments, the light projection system 10 can also be used by being worn on the head of the user who is the observer.

[0262] At least a part of the configuration of the eyeball information detection device of the above embodiments can be combined within a range in which they do not contradict each other.

[0263] Further, the present technology can also adopt the following configuration.

[0264] (1) A display device including:

[0265] a light projection system including a light source; and

[0266] an eyepiece optical system that guides light projected from the light projection system to an eyeball,

[0267] wherein the eyepiece optical system includes:

[0268] an optical element; and

[0269] 3 or more feature points whose relative positions with respect to the optical element are fixed, and

[0270] provided with an acquisition unit that acquires at least two-dimensional position information of each of at least three feature points among the 3 or more feature points.

[0271] (2) The display device according to (1), wherein the at least three feature points are not on the same straight line.

[0272] (3) The display device according to (1) or (2), wherein the optical element is provided in a contact lens attached to the eyeball.

[0273] (4) The display device according to (3), wherein the at least three feature points are provided in the contact lens.

[0274] (5) The display device according to (4), wherein the acquisition unit is capable of acquiring at least two-dimensional position information of each of the at least three feature points even when an orientation of the eyeball is changed.

[0275] (6) The display device according to (1) or (2), wherein the optical element is provided in a lens mounted on a spectacle frame.

[0276] (7) The display device according to (6), wherein all of the at least three feature points are provided in the lens or the spectacle frame, or a part of the at least three feature points is provided in the lens and the other part is provided in the spectacle frame.

[0277] (8) The display device according to any one of (1) to (7), wherein the at least three feature points and an optical center of the optical element are located on the same plane.

[0278] (9) The display device according to any one of (1) to (8), wherein the acquisition unit includes at least one camera.

[0279] (10) The display device according to (9), wherein the at least one camera includes a visible light camera and / or a TOF camera.

[0280] (11) The display device according to (9) or (10), wherein the at least one camera includes a plurality of cameras configured to be a stereo camera.

[0281] (12) The display device according to any one of (1) to (11), wherein the light projection system includes the acquisition unit.

[0282] (13) The display device according to any one of (9) to (12), wherein the acquisition unit includes an infrared light source, the camera is sensitive to an infrared wavelength, and the at least three feature points are made of a retroreflective material.

[0283] (14) The display device according to any one of (9) to (13), wherein

[0284] the light projection system includes a projection optical system arranged on an optical path of light from the light source, and

[0285] an optical axis of the projection optical system and an optical axis of the camera are coaxial.

[0286] (15) The display device according to any one of claims (9) to (14), wherein

[0287] the light projection system includes:

[0288] a projection optical system arranged on an optical path of light from the light source; and

[0289] a control unit that controls the light source and / or the projection optical system based on an acquisition result of the acquisition unit.

[0290] (16) The display device according to (15), wherein

[0291] the projection optical system includes a movable deflection element and a focusing lens,

[0292] the acquisition unit acquires

[0293] two-dimensional position information of each of the at least three feature points in a plane orthogonal to an optical axis direction of the camera, and

[0294] distance information of the camera and each of the at least three feature points with respect to the optical axis direction of the camera, and

[0295] the control unit concurrently performs:

[0296] controlling the movable deflection element based on at least the two-dimensional position information; and

[0297] controlling the light source based on the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information.

[0298] (17) The display device according to any one of (1) to (16), wherein at least two of the at least three characteristic points are integrally provided.

[0299] (18) The display device according to any one of (1) to (17), wherein the light projection system and the eyepiece optical system are separated from each other.

[0300] (19) A display method comprising:

[0301] acquiring at least two-dimensional position information of each of at least three characteristic points among three or more characteristic points, the relative positions of which to an optical element that guides light projected from a light projection system to an eyeball are fixed; and

[0302] controlling a part of the light projection system based on the acquisition result in the acquiring step.

[0303] (20) The display method according to (19), wherein

[0304] the light projection system includes a light source, a movable deflecting element, and a focusing lens,

[0305] the acquiring at least two-dimensional position information includes:

[0306] acquiring two-dimensional position information of each of the at least three characteristic points in a plane orthogonal to the optical axis direction of the camera; and

[0307] acquiring distance information of the camera and each of the at least three characteristic points with respect to the optical axis direction of the camera, and

[0308] the controlling includes:

[0309] controlling the movable deflecting element based on at least the two-dimensional position information; and

[0310] controlling a light source based on the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information.

[0311] List of Reference Signs

[0312] 1, 2, 3, 4, 5, 6 display device

[0313] 10 light projection system

[0314] 20 eyepiece optical system

[0315] 100a light source

[0316] 200 projection optical system

[0317] 200a movable mirror (movable deflection element)

[0318] 200c focusing lens

[0319] 300 acquisition unit

[0320] 300a camera

[0321] 300e infrared light source

[0322] 400 control unit

[0323] EB eyeball

[0324] FP feature point

[0325] OE optical element

[0326] OC optical center

[0327] CL contact lens

[0328] GL glass lens (lens)

[0329] GF frame

[0330] IL image light (light)

Claims

1. A display device comprising: a light projection system including a light source; and an eyepiece optical system that guides light projected from the light projection system to an eyeball, wherein the eyepiece optical system includes: an optical element; and 3 or more feature points whose relative positions with respect to the optical element are fixed, and is provided with an acquisition unit that acquires at least two-dimensional position information of each of at least 3 feature points among the 3 or more feature points.

2. The display device according to claim 1, wherein the at least 3 feature points are not on the same straight line.

3. The display device according to claim 1, wherein the optical element is provided in a contact lens attached to the eyeball.

4. The display device according to claim 3, wherein the at least 3 feature points are provided in the contact lens.

5. The display device according to claim 4, wherein the acquisition unit is able to acquire at least two-dimensional position information of each of the at least 3 feature points even when an orientation of the eyeball changes.

6. The display device according to claim 1, wherein the optical element is provided in a lens mounted on a spectacle frame.

7. The display device according to claim 6, wherein all of the at least 3 feature points are provided in the lens or the spectacle frame, or a part of the at least 3 feature points is provided in the lens and the other part is provided in the spectacle frame.

8. The display device according to claim 1, wherein the at least 3 feature points and an optical center of the optical element are located on the same plane.

9. The display device according to claim 1, wherein the acquisition unit includes at least one camera.

10. The display device according to claim 9, wherein the at least one camera includes a visible light camera and / or a TOF camera.

11. The display device according to claim 9, wherein the at least one camera includes a plurality of cameras that constitute a stereo camera.

12. The display device according to claim 1, wherein the light projection system includes the acquisition unit.

13. The display device according to claim 9, wherein the acquisition unit includes an infrared light source, the camera is sensitive to an infrared wavelength, and the at least 3 feature points are made of a retroreflective material.

14. The display device according to claim 9, wherein the light projection system includes a projection optical system arranged on an optical path of light from the light source, and an optical axis of the projection optical system and an optical axis of the camera are coaxial.

15. The display device according to claim 9, wherein the light projection system includes: a projection optical system arranged on an optical path of light from the light source; and a control unit that controls the light source and / or the projection optical system based on an acquisition result of the acquisition unit.

16. The display device according to claim 15, wherein the projection optical system includes a movable deflection element and a focusing lens, the acquisition unit acquires ​ two-dimensional position information of each of the at least three feature points in a plane orthogonal to an optical axis direction of the camera, and distance information of each of the at least three feature points with respect to the optical axis direction of the camera, and the control unit performs the following operations in parallel: controlling the movable deflection element based on at least the two-dimensional position information; and controlling the light source based on the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information.

17. The display device according to claim 1, wherein at least two of the at least three feature points are provided integrally.

18. The display device according to claim 1, wherein the light projection system and the eyepiece optical system are separate from each other.

19. A display method comprising: a step of acquiring at least two-dimensional position information of each of at least three feature points among three or more feature points, the relative positions of the feature points with respect to an optical element that guides light projected from a light projection system to an eyeball being fixed; and a step of controlling a part of the light projection system based on a result of the acquisition in the acquiring step.

20. The display method according to claim 19, wherein the light projection system includes a light source, a movable deflection element, and a focusing lens, the step of acquiring at least two-dimensional position information includes: a step of acquiring two-dimensional position information of each of the at least three feature points in a plane orthogonal to an optical axis direction of the camera; and a step of acquiring distance information of each of the at least three feature points with respect to the optical axis direction of the camera, and the control step includes: a step of controlling the movable deflection element based on at least the two-dimensional position information; and a step of controlling the light source based on the two-dimensional position information and the distance information, and / or controlling the focusing lens based on the distance information. ​ ​

Images