Projection device and projection method

Abstract

To provide a visual augmentation technique that alleviates wearing discomfort and is not constrained by visual field restriction.SOLUTION: A projection system includes: a projector 30 which projects an image onto the eyelid; a sensor 20 which detects blinking; and a control device 10 which projects an image onto the eyelid at the timing at which the sensor 20 has detected blinking. The sensor 20 is a photoreflector that irradiates the eyelid with infrared light and detects the reflected infrared light and detects blinking on the basis of the changes in the measurement values of the sensor 20.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a projection device and a projection method.

Background Art

[0002] In recent years, research on augmented reality (AR) has been actively carried out. In order to utilize AR, it is common to wear a head-mounted display (HMD) or AR glasses. There is also a method of directly projecting an image onto the retina.

[0003] On the other hand, Non-Patent Document 1 discloses a projection device that uses its own eyelid as a screen.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Although HMDs and AR glasses are becoming smaller and lighter, there is a sense of discomfort in covering the front of the eyes, and it is difficult to completely remove the burden on the user. Since the method of projecting an image onto the retina projects the image through the pupil, there are restrictions on the direction of the eyeball.

[0006] The technology of Non-Patent Document 1 uses the eyelid as a screen by pressing an optical fiber against the eyelid with the eyelid closed, and cannot superimpose an image on the real world.

[0007] The present disclosure has been made in view of the above, and aims to provide a visual expansion technology that alleviates the sense of wearing discomfort and is not restricted by visual field limitations.

Means for Solving the Problem

[0008] A projection device according to an aspect of the present disclosure includes a projection unit that projects an image onto an eyelid, a sensor for detecting a blink, and a control unit that controls the projection unit at the timing when the sensor detects a blink to project an image onto the eyelid.

[0009] A projection method according to an aspect of the present disclosure detects a blink and projects an image onto an eyelid at the timing when the blink is detected.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to provide a visual augmentation technology that alleviates the feeling of discomfort when worn and is not restricted by visual field limitations.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Generally, a video refers to a moving image, and an image refers to a still image. However, in the present disclosure, a video and an image are not clearly distinguished and are treated as synonyms in some cases. That is, even if the word "image" is used, it can be read as a "video" without any problem, and vice versa. In the present disclosure, the word "video" is used to refer to all visual expressions including moving images and still images.

[0013] [System Configuration] Referring to FIG. 1, an example of the configuration of the projection system of the present embodiment will be described. This projection system is a system that realizes AR by projecting a video onto the eyelids during the blackout period when a person blinks and closes their eyelids, and superimposing the projected video on the normal visual information visible when the eyes are open. Blackout is a state where the eyelids cover the pupils and visual information is blocked.

[0014] The projection system shown in FIG. 1 includes a control device 10, a sensor 20, and a projector 30.

[0015] The control device 10 receives sensing data from the sensor 20, controls the projector 30 based on the sensing data, and projects a video onto the eyelids. The control device 10 includes a detection unit 11 and a control unit 12. Each part included in the control device 10 may be configured by at least one or more computers equipped with an arithmetic processing device, a storage device, etc., and the processing of each part may be executed by a program. This program is stored in the storage device included in the control device 10, and can also be recorded on a non-temporary recording medium readable by a computer such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network.

[0016] The detection unit 11 receives sensing data from the sensor 20, performs smoothing and threshold processing, and detects the blackout period when it becomes dark due to blinking. The detection unit 11 transmits a detection signal to the control unit 12 at the start and end timings of the blackout period. Generally, there is a blackout period of about several hundred milliseconds during a blink.

[0017] The sensor 20 may be of any type as long as it can detect blinking, i.e., the state where the eyelids are closed. For example, a photo-reflector may be used to detect blinking by measuring in real time the change in the distance to the eyelids and the change in the amount of light reflection. Alternatively, a sensor for measuring a biological signal may be used to detect blinking by measuring the change in the electromyogram of the muscles that move the eyelids or by measuring the electrooculogram. Alternatively, a Doppler sensor may be used to detect blinking, or a magnet may be attached to the eyelids to detect the change in magnetic force to detect blinking. Alternatively, if the speed of a camera or a computer is fast enough, blinking may be detected from an image taken of the vicinity of the eyes.

[0018] The control unit 12 controls the projector 30 in synchronization with the blackout period and switches the on / off of the projection of the video onto the eyelids. Specifically, the control unit 12 causes the projector 30 to start projecting the video at the timing when it receives the detection signal of the start of the blackout period, and stops the projection of the video onto the projector 30 at the timing when it receives the detection signal of the end of the blackout period.

[0019] The control unit 12 may supply the video to be projected to the projector 30. For example, when the projection system is used for route guidance, the control unit 12 supplies the projector 30 with a video corresponding to the position of the user.

[0020] The projector 30 projects the video onto the eyelids using the light from the built-in light source during the blackout period. Specifically, the projector 30 projects the video onto the eyelids during the blackout period in accordance with the control from the control unit 12, and does not output the video outside the blackout period.

[0021] Note that, instead of directly controlling the on / off of the output of the projector 30, a configuration in which an image is projected onto the eyelids only during the blackout period can also be used. For example, a shutter (e.g., a plate) that physically blocks the projection light is arranged near the lens of the projector 30, and the control unit 12 controls an actuator such as a motor to move the shutter so that the shutter blocks the projection light when it is not the blackout period. An active shutter type (e.g., DLP Link type) 3D glasses may be used, and the control unit 12 may control the on / off of the 3D glasses. When using a shutter, the projector 30 may continue to output an image.

[0022] The projector 30 can be attached to, for example, the user's head. When the projection system is used in a scene where it is necessary to wear a helmet such as in a cave or a construction site, the projector 30 may be attached to the visor of the helmet. The sensor 20 may also be attached to the helmet. By attaching the projection system to the helmet, in a scene where wearing a helmet is recommended, there is no need to separately wear an AR device such as an HMD.

[0023] If the user is indoors or sitting on a chair, the projector 30 may be installed at a position where an image can be projected onto the eyelids without the user wearing the projector 30.

[0024] [Superimposed Image] With reference to FIGS. 2 to 4, an example of the superimposed image obtained by the projection system will be described.

[0025] For example, the projection system is used for wayfinding inside a building. FIG. 2 shows an example of the scenery visible when the eyes are open. FIG. 3 shows an example of the wayfinding image projected when the eyes are closed.

[0026] When the user blinks about two or three times per second, the image of FIG. 3 is projected onto the eyelids during the blackout period, and the user can obtain a feeling that the wayfinding image is superimposed on the normal scenery as shown in FIG. 4.

[0027] [Operation of the Projection System] Referring to the flowchart of FIG. 5, an example of the processing flow of the projection system will be described.

[0028] In step S11, the control device 10 receives a measurement value from the sensor 20. Here, as the sensor 20, a photo-reflector composed of an infrared LED and a photo-transistor is used. The photo-reflector irradiates infrared rays onto the eyelid and detects the infrared rays reflected therefrom. Infrared rays are irradiated towards the user's eyes, and blinks are detected based on the difference in the amount of infrared ray reflection due to the movement of the eyelid accompanying blinks. When the user blinks, the infrared rays are reflected by the eyelid, so a difference appears in the amount of infrared ray reflection. It is considered that the measurement value obtained by the photo-reflector is affected not only by the change in distance but also by the difference in the infrared ray reflection characteristics of the pupil and the eyelid.

[0029] In step S12, the control device 10 determines whether the measurement value exceeds a first threshold value. FIG. 6 shows an example of the change in the measurement value of the sensor 20 when blinking. In FIG. 6, time is taken on the horizontal axis and the measurement value of the sensor 20 is taken on the vertical axis. The graph in FIG. 6 is the measurement value of the sensor 20 after smoothing processing. As shown in FIG. 6, the measurement value increases when the eyelid is closed and decreases when the eyelid is opened. The control device 10 compares the measurement value with the threshold value, and if the measurement value exceeds the threshold value, it determines that the eyelid has been closed.

[0030] If it is determined that the eyelid has been closed, in step S13, the control device 10 starts projecting an image onto the eyelid.

[0031] In step S14, the control device 10 receives a measurement value from the sensor 20.

[0032] In step S15, the control device 10 determines whether the measured value is less than the second threshold value. In this embodiment, different threshold values are used for detection when the eyelids are closed and when they are opened. Although the opening and closing state of the eyelids may be detected using a single threshold value, as a result of experiments, it was found that the blackout period can be detected with higher accuracy by using different threshold values for detection when the eyelids are closed and when they are opened. As a blinking operation, it is known that the time from when the eyes are open until they close is considerably faster than the time from when they are closed until they fully open. For this reason, in this embodiment, two different threshold values are set to detect the state of the eyelids. The control device 10 determines that the eyelids have been opened when the measured value is less than the second threshold value. That is, the control device 10 determines the period from when the measured value exceeds the first threshold value until it is less than the second threshold value as the blackout period.

[0033] When it is determined that the eyelids have been opened, in step S16, the control device 10 stops projecting the video onto the eyelids. The control device 10 returns the process to step S11 and continues detecting blinks.

[0034] Through the above processing, the video is projected onto the eyelids only during the blackout period when the visual information is blocked by a blink.

[0035] [Example] Next, an example in which the change in the degree of perception according to the blink frequency was investigated will be described.

[0036] One sensor 20 and one projector 30 were respectively attached to a frame equipped with a platform on which the chin is placed. When the subject places their chin on the platform, the sensor 20 is positioned at a location where it can detect blinks of the left eye, and the projector 30 is positioned at a location where it can project a video onto the right eyelid. A convex lens was placed in front of the lens of the projector 30 and adjusted so that the video of the projector 30 is in focus on the eyelid. The Arduino Due from Arduino was used for the control device 10, the RPR-220 (photo interrupter) from ROHM was used for the sensor 20, and the DLP-4500 from TEXAS INSTRUMENTS was used for the projector 30.

[0037] An LED was installed in front of the subject's eyes, and the LED was lit only during the period when blackout was detected, and the threshold was adjusted so that the subject could not perceive the lighting of the LED. For example, when the subject could perceive the lighting of the LED, the threshold was increased, and when the subject could not perceive the lighting of the LED, the threshold was decreased to obtain an appropriate threshold. Thresholds were set for the start and end of each blackout period.

[0038] The control device 10 executed processing every 1 ms. That is, the control device 10 received measurement values from the sensor 20 every 1 ms, compared the measurement values with the threshold, and detected the blackout period.

[0039] During the blackout period, the projector 30 projected an image 100 with the left and right sides painted in two colors as shown in FIG. 7 onto the eyelids. The image 100 is an image in which the left region 110 is painted green and the right region 120 is painted blue as seen from the subject.

[0040] The subject blinked at a frequency of 1 to 3 times per second once every 3 seconds, and subjectively evaluated the degree of perception at each blink frequency. Blinking once every 3 seconds is a natural blink frequency. Blinking 3 times per second is a frequency of blinking quickly consciously. FIG. 8 shows the experimental results. Color could be perceived at any blink frequency. At a rate of once every 3 seconds, the feeling of an image being superimposed on the real world was not obtained, but by increasing the blink frequency, the feeling of an image being superimposed on the real world could be obtained. That is, by blinking quickly consciously, an image can be superimposed on the real world.

[0041] In this embodiment, the sensor 20 was arranged in front of the subject to detect blinks. However, the sensor 20 may be arranged diagonally in front of or beside the subject to detect blinks from the diagonal front or side of the subject. A plurality of sensors 20 may be provided, or a combination of multiple types of sensors 20 may be used. Although the projector 30 projected the image only onto the right eyelid, the image may be projected onto the left eyelid, which is the same as the sensor 20, or a plurality of projectors 30 may be provided to project the image onto both eyes. Different images may be projected onto each eye separately. The image of the projector 30 may be projected onto the eyelid via an optical element such as a half mirror.

[0042] As described above, the projection system of this embodiment includes a projector 30 that projects an image onto the eyelid, a sensor 20 for detecting blinks, and a control device 10 that projects the image onto the eyelid at the timing when the sensor 20 detects a blink. Thereby, when the user blinks, the image is projected onto the eyelid, so that the image can be superimposed and displayed in the real world without wearing a device that covers in front of the eyes such as an HMD or AR glasses. Also, by consciously changing the blink frequency of the user, the degree of image superimposition can be adjusted.

Description of Reference Numerals

[0043] 10 Control device 11 Detection unit 12 Control unit 20 Sensor 30 Projector

Claims

1. A projection unit that projects an image onto the eyelid, A sensor for detecting blinks, A control unit that projects an image onto the eyelid at the timing when the sensor detects a blink, A projection device.

2. The projection device according to Claim 1, The sensor is a photo-reflector that irradiates infrared rays onto the eyelid and detects the reflected infrared rays, Detecting a blink based on a change in the measured value of the sensor A projection device.

3. The projection device according to Claim 2, When the measured value of the sensor exceeds a first threshold, it is determined as the start of the blackout period when the eyelid is closed, and when it falls below a second threshold, it is determined as the end of the blackout period, Projecting an image onto the eyelid during the blackout period A projection device.

4. Detecting a blink, Projecting an image onto the eyelid at the timing when a blink is detected A projection method.

5. The projection method according to Claim 4, Detecting a blink using a photo-reflector that irradiates infrared rays onto the eyelid and detects the reflected infrared rays A projection method.

6. The projection method according to Claim 5, When the measured value of the photo-reflector exceeds a first threshold, it is determined as the start of the blackout period when the eyelid is closed, and when it falls below a second threshold, it is determined as the end of the blackout period, Projecting an image onto the eyelid during the blackout period A projection method.

Images