Wearable device and display method

Abstract

To provide a wearable device and a display method for changing display contents according to the movement of a line of sight.SOLUTION: A wearable device according to one embodiment includes a display unit configured to display a real world image or an augmented reality image in which information is added to the real world image, a first detection unit configured to detect a movement of a line of sight of a user, a second detection unit configured to detect a motion of a head of the user, and a determination unit configured to determine whether or not a first detection result obtained from the first detection unit and a second detection result obtained from the second detection unit are in a synchronous relationship, in a case where the first detection result and the second detection result are in a synchronous relationship, the display unit displays a name related to a target in the real world image that is present in the line of sight of the user.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a wearable device and a display method. [Background technology]

[0002] Normally, when a person walks while looking straight ahead, their head sways up and down and side to side, but they can still see the scenery around them clearly. This is because the head sways when walking within a smaller range than the rest of the body (with an amplitude of about 2-3 cm regardless of the distance or proximity of the gaze), and the head and eyes move cooperatively within this small range to maintain a stable gaze (see Non-Patent Document 1). Non-Patent Document 1 teaches that "when walking, the head sways up and down and side to side, and compensatory rotation of the head and eyeballs occurs as an action to stabilize the gaze in response to this swaying."

[0003] Regarding electrooculography sensing technology, there is known example 1 (see Patent Document 1) that detects a person's gaze position based on electrooculography (EOG) signals generated by the movement of the person's eyeballs. In this known example 1, electro-oculography (EOG) electrodes are used for electro-oculography detection (electro-oculography sensing), and the user's gaze position is detected from electro-oculography data acquired from the EOG electrodes (paragraph 0025). In the embodiment of known example 1, the EOG electrodes are provided on goggles (paragraph 0061), but there is also known example 2 (see Patent Document 2) in which EOG electrodes are provided on eyeglasses. Goggles and eyeglasses are types of eyewear worn near a person's eyeballs. When a user wearing eyewear walks, the eyewear moves along with the head. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-288529 [Patent Document 2] Japanese Patent Application Laid-Open No. 2013-244370 [Non-patent literature]

[0005] [Non-Patent Document 1] Head and eye movements to maintain gaze stability while walking. 2000-03 Yazaki, Eiya. Osaka University Graduate School of Human Sciences Bulletin. 26 P.177-P.193 http: / / ir.library.osaka-u.ac.jp / dspace / bitstream / 11094 / 5672 / 1 / hs26-177.pdf Summary of the Invention [Problem to be solved by the invention]

[0006] When a user is sitting still (a state in which there are no disturbing movements of the user's head while wearing EOG electrode eyewear), electrooculography sensing can reliably detect gaze / eye rotation. However, when a user is walking (a state in which the user's head shakes up and down and side to side while wearing EOG electrode eyewear) and gazing at a fixed object in the real world, it is more difficult to detect the desired gaze / eye rotation than when the user is sitting. This is because the human body unconsciously rotates its eyes as a complementary action to stabilize the gaze, and the signal component of this unconscious eye rotation is superimposed (as noise or disturbance) on the electrooculography sensing results.

[0007] That is, when a user wearing eyewear with EOG electrodes walks, even if the user fixes their gaze on a specific object in front of them, the swaying of the head caused by walking causes the electro-oculogram data acquired from the EOG electrodes to fluctuate. This fluctuation in the electro-oculogram data becomes noise (disturbance) in the electro-oculogram data corresponding to the user's actual gaze.

[0008] One of the problems to be solved by the embodiments of the present invention is to provide a wearable device and a display method that change the display content according to the movement of the line of sight. [Means for solving the problem]

[0009] A wearable device according to an embodiment includes a display unit that displays a real-world image or an augmented reality image that adds information to the real-world image, a first detection unit that detects movement of a user's line of sight, a second detection unit that detects movement of the user's head, and a determination unit that determines whether a first detection result obtained from the first detection unit and a second detection result obtained from the second detection unit are synchronized. If the first detection result and the second detection result are synchronized, the display unit displays a name of an object in the real-world image that is in the user's line of sight. [Brief explanation of the drawings]

[0010] [Figure 1] A diagram explaining head sway (position change due to up / down / left / right movement) of a person walking while looking ahead. [Figure 2] A diagram illustrating head movement (angle change due to forward / backward bending) of a person gazing at a fixed object in front of them while walking. [Figure 3] This is a diagram explaining the relationship between the visual target distance (the distance from the eye to the gaze target) and eye movement when the head is bent forward and backward in the vertical direction. [Figure 4] FIG. 10 is a diagram illustrating the phase relationship between angle / velocity changes of head forward / backward bending and angle / velocity changes of eye movement. [Figure 5] 1A and 1B are diagrams illustrating eyeglass-type eyewear incorporating an eye movement detection device according to an embodiment (an example in which EOG electrodes are arranged on nose pads). [Figure 6] 1A and 1B are diagrams illustrating an example of implementation of EOG electrodes in eyeglass-type eyewear according to an embodiment. [Figure 7] 1 is a diagram for explaining the relationship between an information processing unit 11 that can be attached to various embodiments and its peripheral devices. [Figure 8] For example, this figure shows an example of an EOG waveform when you start walking with your right foot while gazing at a fixed object in the real world 25 m away and stop with your right foot on the 13th step. [Figure 9] This figure shows another example of an EOG waveform (confirming repeatability) when walking under the same conditions as in Figure 8, starting again with the right foot and stopping with the right foot on the 13th step. [Figure 10] Electrooculogram (EOG) illustrating the relationship between eye movement from front to upward and the detection signal levels (Ch0, Ch1, Ch2, and the average level Ch1+2 of Ch1 and Ch2) obtained from the three analog-to-digital converters (ADCs) shown in Figure 6 when noise caused by head movement is canceled out. [Figure 11] Electrooculogram illustrating the relationship between eye movement from front to downward and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in Figure 6 when noise caused by head movement is canceled out. [Figure 12] Electrooculogram illustrating the relationship between eye movement from left to right and the detected signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in Figure 6 when noise caused by head movement is canceled out. [Figure 13] An electrooculogram illustrating the relationship between eye movement, where blinking (both eyes) is repeated five times at 5-second intervals while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 14] An electrooculogram illustrating the relationship between eye movements, in which the eyes are closed for 1 second (both eyes) and then opened for 4 seconds (both eyes) five times while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 15] An electrooculogram illustrating the relationship between the eye movements of a person blinking both eyes five times, followed immediately by a wink of the left eye (blinking of the left eye) five times, when the line of sight is facing forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 16]An electrooculogram illustrating the relationship between the eye movements of a person blinking both eyes five times, followed immediately by a wink of the right eye (blinking one eye on the right side) five times, when the line of sight is facing forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 17] 10 is a flowchart illustrating an example of a process for minimizing noise caused by head movement (noise cancellation process). [Figure 18] 10A and 10B are diagrams illustrating an example of how EOG electrodes are mounted in eyeglass-type eyewear according to another embodiment (an example in which EOG electrodes are arranged around the eyeballs). [Figure 19] FIG. 10 is a diagram for explaining an example of implementation of EOG electrodes in goggle-type eyewear (a type in which the eye frames for both the left and right eyes are continuous) according to yet another embodiment (another example in which EOG electrodes are arranged around the eyeballs). [Figure 20] FIG. 10 is a diagram for explaining an example of implementation of EOG electrodes in goggle-type eyewear (a type in which the eyecups for the left and right eyes are separated) according to yet another embodiment (yet another example in which EOG electrodes are arranged around the eyeballs). DETAILED DESCRIPTION OF THE INVENTION

[0011] Below, basic information will be provided first, and then various embodiments will be described with reference to the drawings. <Basic information> The diameter of an adult's eyeball is about 25 mm. After birth, it is about 17 mm, and grows larger as the animal grows. The interpupillary distance of an adult male is approximately 65 mm. (Most commercially available stereo cameras are made with a 65 mm distance.) The interpupillary distance of adult women is several millimeters shorter than that of men. The electrooculography is several tens of mV. The eyeball has a positive potential on the cornea side and a negative potential on the retina side, which appears as a potential difference of several hundred microvolts when measured on the surface of the skin.

[0012] The types of eye movements and ranges of eye movement related to eye movement detection are, for example, as follows: <Types of eye movements> (01) Compensatory eye movements An involuntary eye movement that developed to stabilize the image of the external world on the retina regardless of head or body movement. (02) Voluntary eye movements This is an eye movement that has developed to center the visual image on the retina, and can be controlled voluntarily. (03) Impulsive eye movements (saccades) Eye movement (easy to detect) that occurs when changing the point of gaze to look at an object. (04) Sliding eye movements Smooth eye movements (hard to detect) that occur when tracking a slowly moving object. <Eye movement range (for a typical adult)> (11) Horizontal direction Left direction: 50° or less Right direction: 50° or less (12) Vertical direction Downward: 50° or less Upward: 30° or less (The vertical angle range that can be moved at will is narrow only in the upward direction. (Due to the "Bell phenomenon" in which the eyes roll upward when closed, the vertical range of eye movement shifts upward when the eyes are closed.) (13) Other Convergence angle: 20° or less. <Head movement during walking (Figures 1 and 2 are taken from Non-Patent Document 1)> Figure 1 is a diagram that explains head sway (position change due to up / down and left / right movement) of a person walking while looking ahead. The position of a person's head changes up and down due to the movement of the lower limbs when walking (Figure 1(a)). The walking period (one walking cycle) of a typical adult is, for example, about 0.6 seconds. At the same time, the position of the person's head also changes left and right while walking (Figure 1(b)). The period of this left and right position change is about 1.2 seconds, which is twice the walking period. The range of the up and down position change and the left and right position change is about 2 to 3 cm each. This up and down and left and right movement is approximately linear and regular, with a period that is closely related to the walking period.

[0013] Figure 2 is a diagram illustrating head sway (angle change due to forward / backward bending) of a person gazing at a fixed object in front of them while walking. When a person gazing at a visual target in front of them starts walking, the head position changes up and down as they walk (Figure 2(a)). While gazing at the target, if the head reaches an upper position, the head bends forward (pitches down), and if the head reaches a lower position, the head bends backward (pitches up) (Figure 2(b)). This forward and backward bending (vertical rotational movement of the head) occurs in response to (synchronized with) the up and down movement of the head during one walking cycle. Similarly, if the head position changes from side to side (Figure 1(b)), a left and right rotational movement of the head occurs (synchronized with the up and down movement of the head).

[0014] In other words, when walking, a person's head rotates to compensate for up / down / left / right movements, bending forward and backward at a high / low position to help the gaze capture the visual target (visual target) in front of them. The same is true for left and right; when the body sways from side to side as the person walks and the head moves left and right, the head rotates left and right (rotates back and forth). This type of head rotation is called "compensatory rotation" and is thought to help stabilize the gaze.

[0015] Furthermore, since the human trunk rotates back and forth and side to side while walking, the head must also compensate for these movements. By rotating the head in the opposite direction to these trunk movements, the angular position of the head is kept horizontal. <Eyeball movement during walking (Figures 3 and 4 are taken from Non-Patent Document 1)> Figure 3 illustrates the relationship between target distance (the distance from the eye to the gazed object) and eye movement when the head is bent forward or backward in the vertical direction. Here, the target position when the head is fixed is defined as HFP, and the head bend rotation angle (φh) and the eye compensatory rotation angles (φen, φef) are illustrated for the cases when the actual target is closer than the HFP (Near Target) and farther away (Far Target).

[0016] As mentioned above, the head performs compensatory rotation, but this alone is not enough to completely stabilize gaze. The head often rotates excessively or insufficiently to compensate for vertical movement. To compensate for the discrepancy caused by this excessive or insufficient compensatory rotation, periodic compensatory eye rotation is required during walking. For example, for a near target 30 cm away, the eyes perform a compensatory rotation (φen) in the same direction as the head rotation (φh). Even if the head rotation is insufficient, the eyes rotate in concert with the head in the same direction, compensating for the insufficient head rotation. On the other hand, for a far target 100 m away, the eyes perform a compensatory rotation (φef) in the opposite direction to the head rotation (φh). Even if the head rotation becomes excessive, the eyes rotate in the opposite direction, counteracting the excessive head rotation. (Note: There is no substantial difference in the amount of head movement during walking, whether the target is close or far.) Figure 4 is a diagram illustrating the phase relationship between angle / velocity changes of head forward / backward bending and angle / velocity changes of eye movement. The angle of head forward / backward bending (φh) shown in Figure 3 can be detected by a three-axis gyro (11e in Figure 7) attached to eyewear. The rotation angles of the eyeballs (φen, φef) shown in Figure 3 can be detected by an eye movement detection unit (15 in Figure 7) that uses EOG of the eyeballs.

[0017] The vertical head movement (vertical forward / backward bending) associated with walking changes in phase with the eye rotation when the visual target is near (Near Target) (Fig. 4(a)). On the other hand, when the visual target is far (Far Target), the vertical head movement (vertical forward / backward bending) changes in phase with the eye rotation (Fig. 4(b)). When the visual target is at the fixed position HFP in Fig. 3, eye rotation disappears, and the dashed waveform change in Fig. 4(a)(b) becomes minimal (almost horizontal in the figure). (At this minimal point, the phase between the vertical head bending and the eye rotation is reversed.) A similar phase reversal is also observed between the velocity changes of head vertical movement and eye rotation (Fig. 4(c)(d)).

[0018] Here, we will refer to the signal changes that occur in the EOG due to the above-mentioned "changes in head up and down movement associated with walking" as "body movement noise." Walking can be detected using an acceleration sensor or gyro sensor, and eye movement can be detected from the EOG. If body movement noise does not appear in the EOG when "walking" and "no eye movement" are detected, this indicates that "the eyewear user is gazing at the point (corresponding to position HFP in Figure 3) where the phase of the body movement noise is inverted depending on the distance of the visual target."

[0019] FIG. 5 is a diagram illustrating eyeglass-type eyewear 100 incorporating an eye movement detection device according to one embodiment (an example in which EOG electrodes are arranged on nose pads). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected by a bridge 103. The left and right eye frames 102, 101, and the bridge 103 can be made of, for example, aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to a left temple bar 106 via a left hinge 104, and a left end cap (left ear pad) 108 is attached to the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to a right temple bar 107 via a right hinge 105, and a right end cap (right ear pad) 109 is attached to the tip of the right temple bar 107.

[0020] An information processing unit 11 (an integrated circuit of a few millimeters square), which is the main part of the eye movement detection device, is attached to the bridge 103. This information processing unit 11 can be configured using an LSI that integrates a microcomputer, memory, a communication processing unit, etc. (Details of the information processing unit 11 will be described later with reference to FIG. 7).

[0021] A small battery (BAT) such as a lithium ion battery is embedded, for example, in the left temple bar 106 (or in the right temple bar 107, or in the endpieces 108 or 109), and provides the power source necessary for the operation of the eyeglass-type eyewear 100. A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be configured using ultra-small CCD image sensors.

[0022] These cameras (13L, 13R) may be configured as a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be placed in the positions of these cameras to configure a distance sensor using an infrared camera and laser. This distance sensor may also be configured with a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

[0023] Note that an embodiment is also possible in which a central camera (not shown) is provided in the bridge 103 instead of or in addition to the left and right cameras 13L / 13R. Conversely, an embodiment is also possible in which no camera is provided at all (these cameras are shown as cameras 13 in FIG. 7).

[0024] A left display 12L is fitted into the left eye frame 102, and a right display 12R is fitted into the right eye frame 101. This display is provided in at least one of the left and right eye frames and can be constructed using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L, 12R can be constructed using a film liquid crystal display device that employs polymer dispersed liquid crystal (PDLC) without using a polarizing plate (this display is shown as display 12 in Figure 7).

[0025] The transparent left and right displays 12L / 12R, which use film LCD, can be used as a means of providing augmented reality (AR), which adds image information such as numbers and letters to the real world seen through the glasses. A nose pad section is provided between the left and right eye frames 102, 101 and below the bridge 103. This nose pad section is composed of a pair of a left nose pad 150L and a right nose pad 150R. Right nose pad electrodes 151a and 151b are provided on the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided on the left nose pad 150L.

[0026] These electrodes 151a, 151b, 152a, and 152b are electrically isolated from one another and connected to three AD converters (ADCs 1510, 1520, and 1512) via insulated wiring (not shown). The outputs from these ADCs have signal waveforms that vary depending on the eye movement of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data corresponding to the user's eye movement. The electrodes 151a, 151b, 152a, and 152b are used as gaze detection sensors, and together with the three AD converters, are components of the eye movement detection unit 15 shown in FIG. 7.

[0027] The eyewear 100 is fixed to the user's head (not shown) by the left and right nose pads (150L, ​​150R), the left and right temple bars (106, 107), and the left and right end pieces (108, 109). In this embodiment, only the left and right nose pads (150L, ​​150R), the left and right temple bars (106, 107), and the left and right end pieces (108, 109) are in direct contact with the user's head (or face), but there may be embodiments in which parts other than these (nose pads, temple bars, end pieces) come into contact with the user, for example, to match the voltage between the ADCs (1510, 1520, 1512) and the user's body.

[0028] The film liquid crystal display of the right display 12R in FIG. 5 can display a right display image IM1 including, for example, a numeric keypad (numbers, operators, Enter, etc.), alphabets, marks of predetermined shapes, and other icons. The film liquid crystal display of the left display 12L can display a left display image IM2 including, for example, any character string, marks of any shape, various icons, etc. (The display content of the displays 12L and 12R can be anything). The numeric keypad and alphabets displayed on the right display 12R (or the left display 12L) can be used to input numbers and letters. The character strings, marks, icons, etc. displayed on the right display 12R (or the left display 12L) can be used to search for a specific information item, select / confirm a desired item, or alert the user.

[0029] Display images IM1 and IM2 can be used as an augmented reality (AR) display means that adds information such as numbers, letters, and marks to the real world seen through the glasses, and this AR display can be turned on and off as needed. The content of display image IM1 and display image IM2 may be the same (IM1 = IM2) or different (IM1 ≠ IM2) depending on the embodiment. Display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. If the content of the AR display is to be a 3D image that overlaps (has depth) with the real world seen through the glasses, IM1 and IM2 can be separate 3D images for the left and right.

[0030] Furthermore, if displays (12R, 12L) exist on the left and right, it is possible to adjust the convergence angle, for example, to shift the images displayed on the left and right (IM1, IM2) in opposite directions. This is thought to reduce the strain on the eyes when alternating between viewing objects visible in the real world and the AR display. However, normally, the left and right displays (12R, 12L) display the same image content.

[0031] Display control on displays 12L and 12R can be performed by an information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on a display is well known.) The power required for the operation of information control unit 11 and other components can be obtained from a battery BAT embedded in the left temple bar 106.

[0032] It should be noted that when a designer or engineer tries on a prototype of eyewear 100 corresponding to the embodiment, they may find that the weight balance is poor. If this is primarily due to the BAT in the left temple bar 106, then a "weight (or another battery of the same weight)" that matches the BAT in the left temple bar 106 can be placed in the right temple bar 107.

[0033] 6 is a diagram illustrating an example of how EOG electrodes are implemented in eyeglass-type eyewear 100 according to one embodiment. Right nose pad electrodes 151a and 151b are provided above and below right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below left nose pad 150L. The outputs of right nose pad electrodes 151a and 151b are provided to ADC 1510, the outputs of left nose pad electrodes 152a and 152b are provided to ADC 1520, and the outputs of lower electrodes 151b and 152b (or upper electrodes 151a and 152a) of the left and right nose pads are provided to ADC 1512.

[0034] ADC 1510 outputs a Ch1 signal that changes in response to the user's right vertical eye movement. ADC 1520 outputs a Ch2 signal that changes in response to the user's left vertical eye movement. ADC 1512 outputs a Ch0 signal that changes in response to the user's left and right eye movement. The up and down movement of both the left and right eyes can be evaluated by a Ch1+2 signal that corresponds to the average of the outputs of ADC 1510 and ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, and Ch1+2 and eye movement will be described later.) FIG. 7 is a diagram illustrating the relationship between an information processing unit 11 that can be attached to various embodiments and its peripheral devices. In the example of FIG. 7, the information processing unit 11 is composed of a processor 11a, a nonvolatile memory 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, and the like. The processor 11a can be composed of a microcomputer having processing capabilities according to the product specifications. Various programs executed by this microcomputer and various parameters used when executing the programs can be stored in the nonvolatile memory 11b. The main memory 11c provides a work area for executing the programs.

[0035] The sensor unit 11e includes a group of sensors for detecting the position and / or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensors include an acceleration sensor that detects movement in three axial directions (x, y, and z directions), a gyroscope that detects rotation in three axial directions, a geomagnetic sensor (compass function) that detects absolute orientation, and a beacon sensor that receives radio waves, infrared rays, and the like to obtain location information and other information. This location information and other information can be obtained using iBeacon (registered trademark) or Bluetooth (registered trademark) 4.0.

[0036] LSIs that can be used for the information processing unit 11 have been commercialized. One example is the "TZ1000 Series for Wearable Devices" from Toshiba Semiconductor & Storage Corporation. Of this series, the product named "TZ1011MBG" has a CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), a group of sensors (accelerometer, gyro, geomagnetic sensor) (11e), a 24-bit delta-sigma ADC, and I / O (USB, etc.).

[0037] The information processing unit 11 is attached to the bridge 103, which is an example of a location where the relative positional relationship between the sensor unit 11e, such as an acceleration sensor or a gyro sensor, and the EOG electrodes (151a, 151b, 152a, 152b) is not lost. What the processor 11a does can be instructed by an external server (or personal computer) (not shown) via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth, and Wi-Fi (registered trademark). The processing results of the processor 11a can be sent to a server (not shown) via the communication processing unit 11d.

[0038] The display 12 (12L and 12R), the camera 13 (13L and 13R), the eye movement detection unit 15, etc. are connected to the system bus of the information processing unit 11. Each device (11 to 15) in FIG. 7 is powered by a battery BAT. The communications processing unit 11d may incorporate a GPS (Global Positioning System) function, which is well known in mobile phones and smartphones (or a mobile phone function with GPS may be built into the eyewear 100 somewhere). By running an internet map service application using this GPS function on the processor 11a, it is possible to detect where the user wearing the eyewear 100 is currently walking and in what direction they are looking, while suppressing noise caused by compensatory eye rotation while walking. (Conventional mobile phone GPS can display the user's current location on a map screen, but cannot display where the user's eyes are directed.) 7 includes four eye movement detection electrodes (151a, 151b, 152a, 152b) that constitute a gaze detection sensor, three ADCs (1510, 1520, 1512) that extract digital signals corresponding to eye movements from these electrodes, and a circuit that outputs the output data from these ADCs to the processor 11a. The processor 11a can interpret commands corresponding to the type of eye movement from various types of eye movements of the user (up-down movement, left-right movement, blinking, closing the eyes, etc.) and execute those commands.

[0039] Specific examples of commands corresponding to the type of eye movement include a command to select an information item in the line of sight if the eye movement is, for example, eye closure (similar to one click of a computer mouse), and a command to start execution of processing for the selected information item if the eye movement is multiple consecutive blinks or winks (similar to double click of a computer mouse). This command is an example of information input B using the eye movement detection unit 15.

[0040] FIG. 8 shows an example of an EOG waveform when, for example, a person starts walking with the right foot and stops with the right foot on the 13th step while gazing at a fixed object in the real world 25 m away. The compensatory rotation of the eyes corresponding to the up and down movement of the head as they walk is reflected in the change in the ADC detection signal level (EOG change) of Ch1 and Ch2. The up and down movement change of the head synchronized with this EOG change can be detected by the triaxial acceleration sensor of the sensor unit 11e of the information processing unit 11. The compensatory rotation of the eyes corresponding to the left and right rotation of the head as they walk is reflected in the change in the ADC detection signal level (EOG change) of Ch0. The left and right rotation change of the head synchronized with this EOG change can be detected by the triaxial gyro sensor of the sensor unit 11e of the information processing unit 11.

[0041] In the example in Figure 8, the EOG fluctuation period of Ch1 and Ch2 (approximately 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. Because the EOG fluctuation periods corresponding to the up and down movement and left and right rotation of the head that accompanies walking are clear, EOG fluctuations caused by walking can be detected at a period synchronized with the walking pitch (approximately 0.6 seconds and twice that in this example).

[0042] Figure 9 shows another example of the EOG waveform (confirming repeatability) when walking again under the same conditions as Figure 8, starting with the right foot and stopping with the right foot on the 13th step. In the example of Figure 9, the EOG fluctuation period (approximately 0.6 seconds) of Ch1 and Ch2 is synchronized with the walking pitch, and the EOG fluctuation period (approximately 1.2 seconds) of Ch0 is synchronized with twice the walking pitch. The EOG fluctuation period corresponding to the vertical movement and left-right rotation of the head accompanying walking is also clearly observed, and it can be seen that the EOG fluctuation caused by walking can be detected using the period synchronized with the walking pitch (approximately 0.6 seconds and twice that in this example).

[0043] Figures 8 and 9 share the following characteristics, confirming the reproducibility of the event: (1) Vertical eye rotation can be observed in accordance with the walking cycle (approximately 0.6 seconds) (see EOG detection signal levels of Ch1 and Ch2). (2) Eyeball rotation in the left and right directions can be confirmed in accordance with a period twice the walking period (approximately 1.2 seconds) (see the EOG detection signal level of Ch0).

[0044] (3) The amplitude of the EOG detection signal caused by head movement during walking reaches 200 μV to 400 μV in some places. Since the EOG signal amplitude measured when the head is stationary is usually about 1 mV, when detecting arbitrary eye rotation, 200 μV to 400 μV becomes a significant amount of noise. This noise can be canceled (reduced or eliminated) as described below.

[0045] That is, the up and down movement of the head accompanying walking can be detected by the acceleration sensor in a period synchronized with the walking pitch (approximately 0.6 seconds in this example), and the left and right rotation of the head accompanying walking can be detected by the gyro sensor in another period synchronized with the walking pitch (approximately 1.2 seconds in this example). In this way, the EOG fluctuations (considered as noise) caused by the up and down movement (sway) of the head accompanying walking can be canceled out (reduced or eliminated) by the detection results of the acceleration sensor, and the EOG fluctuations (considered as noise) caused by the left and right rotation (sway) of the head accompanying walking can be canceled out (reduced or eliminated) by the detection results of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

[0046] FIG. 10 is an electrooculogram (EOG) illustrating the relationship between eye movement from front to upward and the detection signal levels (Ch0, Ch1, Ch2, and the average level Ch1+2 of Ch1 and Ch2) obtained from the three analog-to-digital converters (ADCs) shown in FIG. 6 when noise caused by head movement is canceled. Eye movement detection is performed based on the detection signal waveform within the dashed frame in the figure. The detection standard is that the user wearing the eyewear 100 looks straight ahead and there is no eye movement. (If noise caused by head movement is canceled, the output signal waveforms Ch0 to Ch2 from the three ADCs shown in FIG. 6 are approximately flat and show almost no change over time during the section where the user looks straight ahead without blinking.) With both the user's eyes facing forward, they instantly move their gaze upward, maintain the upward gaze for one second, and then instantly return their gaze to the front. FIG. 10 illustrates the change in the detection signal level when this is repeated five times.

[0047] Figure 11 illustrates an example of eye movement detection similar to that in Figure 10, where the gaze moves downward from the front. If noise caused by head movement is canceled out, even while walking, it is possible to detect whether the gaze is moving upward or downward from the waveform changes in Figures 10 and 11, with the gaze facing forward as the reference.

[0048] Figure 12 is an electrooculogram illustrating the relationship between eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in Figure 6 when noise caused by head movement is canceled out. When there is eye movement from left to right, the change over time in the detection signal waveform of Ch0 rises to the right (although not shown, when there is eye movement from right to left, the change over time in the detection signal waveform of Ch0 falls to the right). From this change in the waveform of Ch0, it can be detected whether the gaze is to the right or left, based on the case when the gaze is facing forward.

[0049] 10 to 12 together reveal the following: If noise caused by head movement is canceled out, it is possible to determine whether the gaze is directed up, down, left, or right, even while walking, based on the case where the gaze is directed straight ahead. Figure 13 is an electrooculogram illustrating the relationship between eye movement (blinking of both eyes) repeated five times at 5-second intervals while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. Blinking of both eyes can be detected by pulses appearing on Ch1 and Ch2. Unconscious blinking by the user is often not periodic. Therefore, by detecting multiple pulses at regular intervals as shown in Figure 13, intentional blinking by the user can be detected.

[0050] The duration of a person's blinking action, or the time their vision is blocked by a blink, is said to be about 300 msec (according to other sources, about 100 msec to 150 msec). In either case, the signal components due to blinking can be removed using a high-pass filter (or band-pass filter) that is shorter than the walking pitch period. The interval between blinks (e.g., about 300 msec) differs from the pitch of normal walking (about 600 msec), and unintentional blinks are not synchronized with normal walking pitch. Therefore, when canceling out EOG noise caused by head movement associated with normal walking, the EOG waveform changes due to blinking are not completely canceled out.

[0051] Figure 14 is an electrooculogram illustrating the relationship between eye movement, in which the eyes are closed for one second (both eyes) and opened for four seconds (both eyes) five times while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. Closing of both eyes can be detected by wide pulses appearing on Ch1 and Ch2 (the detected pulse width is wider because the time spent intentionally closing the eyes is longer than the time spent blinking / closing the eyes with a blink). By detecting wide pulses on Ch1 and Ch2 as shown in Figure 14, the user's intentional eye closure can be detected.

[0052] Although not shown, when the user closes only the right eye, a wide pulse with a large amplitude appears on Ch1, and when the user closes only the left eye, a wide pulse with a large amplitude appears on Ch2. This makes it possible to detect whether the left or right eye is closed separately. When you close only one eye, small waves with opposite phases appear on Ch0. When viewed through Ch0, a small uneven waveform like the one shown below appears.

[0053] When the right eye is closed, the positive potential of the - electrode decreases, resulting in a convex waveform; When the left eye is closed, the positive potential of the + electrode decreases, resulting in a concave waveform. Figure 15 is an electrooculogram illustrating the relationship between eye movement when the eyes are blinked five times with both eyes and then winked five times with the left eye (blinking of the left eye) while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out.

[0054] As illustrated in FIG. 6, the position of the ADC 1512 for Ch0 is offset downward from the center lines of the eyeballs of both the left and right eyes. Due to this offset, when both eyes blink simultaneously, a negative potential change appears at both the + input and the - input of the ADC 1512 in FIG. 6. In this case, if the potential changes (amount and direction) at both the + input and the - input are approximately the same, the changes are almost canceled out, and the value of the signal level output from the ADC 1512 for Ch0 becomes approximately constant (see the Ch0 level within the dashed line on the left side of FIG. 15). On the other hand, when one eye (the left eye) blinks, there is almost no potential change at the - input side of the ADC 1512, and a relatively large negative potential change appears at the + input side of the ADC 1512. This reduces the amount of cancellation of the potential change between the + and - inputs of ADC1512, and a small pulse (signal level ripple) appears in the signal level output from ADC1512 for Ch0 (see the Ch0 level within the dashed line on the right side of Figure 15). From the polarity of this signal level ripple (negative pulse), it is possible to detect a wink of the left eye (an example of left wink detection using Ch0 when EOG noise caused by head movement is canceled out).

[0055] If the potential changes of the + input and - input of ADC1512 are not equal due to distortion of the user's face or skin condition, calibration can be performed in advance so that the output of Ch0ADC is minimized (the amount of cancellation between the + input component and the - input component is maximized) when the user wears eyewear 100 and blinks both eyes simultaneously.

[0056] Furthermore, if the peak ratio SL1a / SL2a of the detection signals Ch1 / Ch2 when both eyes blink is used as a reference, the peak ratio SL1b / SL2b when a left wink occurs changes (SL1b / SL2b is not equal to SL1a / SL2a). This also makes it possible to detect a left wink.

[0057] Figure 16 is an electrooculogram illustrating the relationship between eye movement when the eyes are blinked five times with both eyes and then winked five times with the right eye (blinking of the right eye) while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out.

[0058] As mentioned above, because the position of the ADC 1512 in FIG. 6 is offset downward from the center line of the eyeballs of both the left and right eyes, when both eyes blink simultaneously, a negative potential change appears at both the + and - inputs of the ADC 1512. However, similar potential changes at the + and - inputs are almost canceled out, and the signal level value output from the ADC 1512 for Ch0 remains approximately constant (see the Ch0 level within the dashed line on the left side of FIG. 16). On the other hand, when one eye (the right eye) blinks, there is almost no potential change at the + input side of the ADC 1512, and a relatively large negative potential change appears at the - input side of the ADC 1512. As a result, the amount of cancellation of the potential change between the - and + inputs of the ADC 1512 becomes smaller, and a small pulse (small wave in the signal level) appears in the positive direction at the signal level output from the ADC 1512 for Ch0 (see the Ch0 level within the dashed line on the right side of FIG. 16). The polarity of this signal level wavelet (positive pulse) can be used to detect a wink of the right eye (an example of right wink detection using Ch0 when EOG noise caused by head movement is canceled out).

[0059] Furthermore, if the peak ratio SR1a / SR2a of the detection signals Ch1 / Ch2 when both eyes blink is used as a reference, the peak ratio SR1b / SR2b changes when the right eye winks (SR1b / SR2b is not equal to SR1a / SR2a). Furthermore, the peak ratio SL1b / SL2b when a left wink occurs has a different value from the peak ratio SR1b / SR2b when a right wink occurs (the extent of the difference can be confirmed experimentally). This makes it possible to detect left winks separately from right winks (an example of detecting left and right winks using Ch1 and Ch2).

[0060] Whether Ch0 or Ch1 / Ch2 is used for detecting left and right winks can be decided by the device designer as appropriate. The results of left and right wink detection using Ch0 to Ch2 can be used as operation commands. 17 is a flowchart illustrating an example of a process for minimizing noise caused by head movement (noise cancellation process). A computer program corresponding to this flowchart is stored in the nonvolatile memory 11b of FIG. 7, for example, and executed by the processor 11a.

[0061] First, a user wearing eyewear 100 as shown in Fig. 5 walks at a constant pace while gazing at a fixed object in front of them (ST10). As the user walks, their head sways (moves / rotates) in a predetermined pattern (see Figs. 1(a) and (b)). The movement / rotation of the user's head while walking is detected by an acceleration sensor / gyro sensor (11e in FIG. 7) provided inside the information processing unit 11 in FIG. 5 (ST12). The movement / rotation of the user's head while walking is mixed into the EOG detection signal waveform as noise (see FIG. 8 or 9). The detection signal of the acceleration sensor / gyro sensor is added to this EOG detection signal waveform at a predetermined phase (ST14).

[0062] If the signal amplitude increases as a result of the signal addition (NO in ST16), the phase of the detection signal to be added is inverted (ST18), and the detection signal of the acceleration sensor / gyro sensor is added to the EOG detection signal waveform again (ST14). If the signal amplitude decreases as a result of the signal addition (YES in ST16), the level of the signal to be added is adjusted so that the signal amplitude of the addition result becomes minimum (extremely small) (ST20). By minimizing the signal amplitude of the addition result, the noise mixed in with the EOG detection signal waveform is canceled (reduced or eliminated).

[0063] Based on the EOG detection result after noise caused by head movement (movement / rotation) has been cancelled, the user's eye movement (gaze movement, blinking, eye closing, wink, etc. in FIGS. 10 to 16) is determined (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, the name of the building in front of the user's line of sight can be displayed in AR on the display screen (IM1 / IM2 in FIG. 5) of the eyewear 100.

[0064] If the user's head movement pattern changes, the noise cancellation process also changes. Therefore, whether the user's head movement pattern has changed is detected from changes in the detection results of the acceleration sensor / gyro sensor (ST26). For example, suppose a user wearing eyewear 100 is walking at a constant speed while gazing at a fixed object ahead, but then stops in front of a staircase leading down to the subway and sweeps their gaze over the difference in level between adjacent steps by bending forward and backward and moving their eyes. Then, due to this gaze sweep before and after the difference in level, the head movement pattern will differ from that when walking, and the EOG detection signal waveform will also differ from that when walking.

[0065] Therefore, if the user's head movement pattern has changed (YES in ST26), the process returns to step ST12 and repeats the processes in ST14 to ST20. By repeating the processes in ST14 to ST20, the EOG noise caused by the new head movement pattern is canceled (reduced or eliminated), and the EOG corresponding to the pure gaze movement is detected. The EOG detection result corresponding to the pure gaze movement is compared with the detection result of the acceleration sensor / gyro sensor corresponding to the changed user's head movement pattern. If, as a result of this comparison, a phase inversion from FIG. 4(a) to FIG. 4(b) is detected at the position of a staircase step, for example, it is determined that a staircase step exists near the phase inversion point (ST22). Then, a mark indicating the presence of the step is displayed in AR on the eyewear 100 (ST24). This AR display can be automatically removed a certain time (e.g., 10 seconds) after the detection result of the acceleration sensor / gyro sensor corresponding to the user's head movement pattern disappears.

[0066] The AR display can also be turned on / off automatically depending on whether the EOG waveform (Ch1 / Ch2 waveform in Figure 8 or Figure 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyes are not rotating to compensate for head movement, so the above-mentioned synchronization relationship does not exist, but when the user is gazing at the real world rather than the AR display, the above-mentioned synchronization relationship occurs. Using a similar concept, the AR display can be turned on / off automatically depending on whether the EOG waveform (Ch0 waveform in Figure 8 or Figure 9) and the detection signal waveform of the gyro sensor are synchronized.

[0067] If the user continues to use the eyewear while the shaking pattern of the user's head does not change (NO in ST28), the processing in ST22 to ST24 continues. If the user closes their eyes for a few seconds, for example, to instruct the processor 11a in Fig. 7 to end use of the eyewear 100 (YES in ST28), the processing in Fig. 17 ends.

[0068] 18 is a diagram illustrating an example of how EOG electrodes are implemented in eyeglass-type eyewear 100 according to another embodiment (an example in which EOG electrodes are arranged around the eyeballs). The eyewear 100 in FIG. 18 differs from the eyewear 100 in FIG. 5 in the following ways. The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R are moved toward the right eye frame 101, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved toward the left eye frame 102. The EOG electrodes 151a and 151b are positioned approximately symmetrically with respect to the center position of the user's right eye (not shown), and the EOG electrodes 152a and 152b are positioned approximately symmetrically with respect to the center position of the user's left eye (not shown). In addition, the right-hand diagonal line connecting the EOG electrodes 151a and 151b and the left-hand diagonal line connecting the EOG electrodes 152a and 152b are approximately symmetrical with respect to a vertical line (not shown) along the bridge of the user's nose. The electrodes 151a, 151b, 152a, and 152b can be made of a conductive material (such as a metal or a conductive polymer) attached to the tip of an elastic body (such as a sponge or a silicone cushion). Each EOG electrode is lightly pressed against the skin surface of the face of a user wearing the eyewear 100 by the elastic repulsive force of an elastic body.

[0069] The EOG electrode arrangement structure shown in Fig. 18 makes it easier to detect the electric field generated around the charged eyeball compared to a structure in which the EOG electrodes are arranged on the nose pads. Therefore, the embodiment in Fig. 18 can detect an EOG signal with a larger amplitude than the embodiment in Fig. 5.

[0070] The second difference is that the sensor unit 11e, which includes an acceleration sensor and a gyro sensor, remains in the bridge 103, while the other functions of the information processing unit 11 are moved to the temple bar 107. For example, if the physical size of the information processing unit 11 increases due to enhanced information processing capabilities or the addition of a GPS function, attaching the enlarged information processing unit 11 to the bridge 103 could cause problems with the design and comfort of the eyewear 100. If only the sensor unit 11e is attached to the bridge 103, the structure on the bridge 103 can be made smaller and lighter than if the entire information processing unit 11 were installed, which could improve the design and comfort. On the other hand, if a relatively large structure is installed inside the temple bar 107 (and / or 106), there is little risk of problems with the design and comfort of the eyewear 100.

[0071] FIG. 19 illustrates an example of EOG electrode implementation in goggle-type eyewear (with continuous eye frames for both eyes) 100 according to yet another embodiment (another example in which EOG electrodes are arranged around the eyeballs). The structure shown in FIG. 19 provides higher skin contact stability for the EOG electrodes than the structure shown in FIG. 18 . This allows for more accurate detection of EOG signals while canceling out noise caused by head movement. Furthermore, the structure shown in FIG. 19 facilitates the incorporation of a larger information processing unit 11 and other devices. Therefore, the structure shown in FIG. 19 allows for the incorporation of GPS-equipped smartphone functions without significant design considerations. In this case, the screen display can be performed using AR display on the left and right displays 12L / 12R, which utilize film LCDs or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frames, and a small microphone can be attached near the nose cushions. Furthermore, commands can be input to the smartphone using eye movements such as gaze movement, blinking, closing the eyes, and winking.

[0072] FIG. 20 is a diagram illustrating an example of EOG electrode implementation in goggle-type eyewear (with separate eyecups for the left and right eyes) 100 according to yet another embodiment (another example in which EOG electrodes are arranged around the eyes). The embodiment in FIG. 20 employs the same EOG electrode arrangement and sensor unit 11e arrangement as those in FIG. 18 . However, the goggle structure in FIG. 20 is also suitable for underwater use (or spacewalk training). A diver's head experiences greater head movement when swimming underwater than when walking on land. However, this movement (up / down, left / right, and rotation) can be detected by the three-axis acceleration sensor and three-axis gyro sensor in the sensor unit 11e. The detected movement components can be used to cancel noise components mixed into the EOG signal. Using the EOG signal with reduced noise due to head movement, various commands can be input based on the eye movements (blinking, closing eyes, etc.) of a user (diver) with both hands occupied. <Summary of the embodiment> (a) Conventional gaze detection technology involves using an infrared camera to estimate gaze direction through image processing, but this requires a large amount of equipment (a high-intensity infrared LED, a camera capable of capturing the entire eyeball visible to the outside, a relatively high-performance computing device for image processing, and a power supply to operate these). For this reason, while conventional gaze detection technology is applied to stationary devices, its application to wearable devices has not progressed.

[0073] In contrast, gaze detection using electrooculography sensing requires a much smaller amount of equipment (electrodes, ADC, a relatively low-performance computing device, and a power supply to operate these) than infrared camera methods, making it possible to apply it to wearable devices such as AR glasses. EOG is highly compatible with AR glasses and consumes less power than infrared methods, making it a promising technology for application to wearable devices.

[0074] (b) It counteracts the head movement of the eyewear user when gazing at a fixed object in the real world, thereby improving the detection accuracy for arbitrary eye rotations. (c) Detect whether the fixed object in the real world that the user is gazing at is in the distance or near (detecting the phase inversion point in Figure 4). This allows the system to detect whether the user is gazing at a fixed object in the real world based on the shaking of the eyewear user's head.

[0075] (d) The eyewear user's head movements are used to automatically turn the AR display on and off. (e) Extract only arbitrary eye movements that cancel out the compensatory eye rotation. (f) Detect whether what you are looking at is the real world or not (if the waveform detected by the acceleration sensor and / or gyro sensor and the electrooculography EOG waveform are synchronized while walking, it is determined that you are looking at something in the real world, and if the waveform detected by the acceleration sensor and / or gyro sensor and the electrooculography EOG waveform are not synchronized, it is determined that you are looking at an AR display or the like while walking).

[0076] (g) Effective for noise cancellation when eye movement is used as a UI (user interface). (h) When used to collect work history information using eyewear, it can become an unprecedented indicator (for example, in a state where the effects of noise caused by head movement are eliminated, it can be used to make pass / fail judgments such as "if the degree to which the worker directs their gaze at the correct target is above a certain value, the worker is qualified" in picking work in a warehouse). <Examples of correspondence between the content of the claims originally filed and embodiments> [1] An electro-oculography detection device according to one embodiment (11 in Figure 7: including a processor 11a that executes the processing of Figure 17) includes an eye movement detection unit (15; 151a, 151b, 152a, 152b) that detects eye movements (gaze movement, blinking, etc.) including eye rotation of a user wearing eyewear (100) based on the electro-oculography (EOG) of the user, an acceleration sensor (part of 11e) that detects the movement of the eyewear, and a gyro sensor (part of 11e) that detects the rotation of the eyewear. This eye movement detection device includes a detection means (11a that executes ST12) that detects the movement and / or rotation of the eyewear using the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electrooculogram (EOG), and a noise reduction means (11a that executes ST14 to ST20) that reduces (minimizes or cancels) noise (electrooculogram components added by compensatory eye rotation that acts in a direction that cancels out the movement of the head) that is mixed into the electrooculogram due to the movement and / or rotation of the eyewear by combining signal components (components that are in opposite phase to the mixed noise) detected by the acceleration sensor and / or the gyro sensor.

[0077] [2] The device of [1] further comprises a phase selection means (11a) for executing ST16 to ST18) for selecting the phase of the signal component detected by the acceleration sensor and / or the gyro sensor so that the signal synthesis result by the reduction means tends to decrease.

[0078] [3] The device (11) of [1] is incorporated into eyewear (100 in Figs. 5, 18 to 20). [4] The eyewear (100) of [3] has a display unit (12L, 12R) and is equipped with a display means (11a that executes ST22 to ST24) that performs a display (AR display) on the display unit (12L, 12R) based on the electro-oculography (EOG) after noise has been reduced by the noise reduction means.

[0079] [5] The device of [1] detects the rotation corresponding to the forward and backward bending of the user's head using the gyro sensor (11e), detects the change in the user's gaze direction (Figures 10 and 11: up and down gaze direction changes can be detected from ± level changes in Ch1 / Ch2) using the eye movement detection unit (15), and is configured to identify the perspective of the place where the user is looking based on whether the signal phase of the forward and backward bending change and the signal phase of the gaze direction change are in phase (Figure 4(a)) or out of phase (Figure 4(b)) (it can detect whether a fixed object in the real world that the user is gazing at is in the distance or near).

[0080] [6] The device of [1] is configured to determine that the user is looking at the real world (it can detect whether what the user is gazing at is the real world) if the movement and / or rotation of the eyewear (Figure 1) detected by the acceleration sensor and / or the gyro sensor is synchronized with the change in the electrooculogram (EOG) (Figure 8 or Figure 9) (if the ``predetermined phase relationship'' can be maintained in ST14 of Figure 17) when the eye movement is detected based on the electrooculogram (EOG).

[0081] [7] An electro-oculography detection method according to one embodiment includes detecting eye movement (such as gaze movement or blinking) including eye rotation of a user wearing eyewear (100) based on the electro-oculography (EOG) of the user, detecting head movement of the user wearing the eyewear, and detecting head rotation of the user wearing the eyewear. This method includes the steps of detecting head movement and / or rotation of the user wearing the eyewear when the eye movement is detected based on the electro-oculography (EOG) (ST12), and reducing (minimizing or canceling) noise mixed into the electro-oculography due to head movement and / or rotation of the user wearing the eyewear (electro-oculography components added by compensatory eye rotation that act in a direction to cancel out the head movement) by combining signal components (components that are in phase opposite to the mixed noise) detected in response to the head movement and / or rotation of the user wearing the eyewear (ST14 to ST20).

[0082] [8] The method of [7] further comprises steps (ST16 to ST18) of selecting the phase of the signal components detected in response to the movement and / or rotation of the head of the user wearing the eyewear so that the combined result of the signal components tends to decrease. Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention.

[0083] For example, in the description of the embodiments, eyeglass-type and goggle-type devices were introduced as eyewear equipped with EOG electrodes, but it is also possible to use other items such as eye masks, eye patches, helmets, hats, and hoods as eyewear with EOG electrodes. These embodiments and modifications thereof are included within the scope and spirit of the inventions, and are included in the scope of the inventions and their equivalents as set forth in the claims. Note that a combination of part or all of one embodiment among the disclosed embodiments with part or all of another embodiment among the disclosed embodiments is also included within the scope and spirit of the inventions. [Explanation of symbols]

[0084] 100... Eyewear (goggle type or eyeglass type); 110... Eye frame; 11... Information processing unit (an integrated circuit including a processor 11a, a nonvolatile memory 11b, a main memory 11c, a communication processing unit (GPS processing unit) 11d, a sensor unit 11e, etc.) that is the main part of the eye movement detection device; 11e... Sensor unit including an acceleration sensor, a gyro sensor, etc.; BAT... Power supply (lithium ion battery, etc.); 12... Display unit (right display 12R and left display 12L: film LCD, etc.); IM1... Right display image ( Numeric keypad, alphabet, character string, marks, icons, etc.); IM2...left display image (numeric keypad, alphabet, character string, marks, icons, etc.); 13...camera (right camera 13R and left camera 13L, or a center camera not shown attached to the bridge 103); 15...eye movement detection unit (gaze detection sensor); 1510...right side (Ch1) AD converter; 1520...left side (Ch2) AD converter; 1512...left-right (Ch0) AD converter; 1514...left-right (Ch3) AD converter.

Claims

1. a display unit that displays a real world image or an augmented reality image that adds information to the real world image; a first detection unit that detects movement of the user's line of sight; a second detection unit that detects a movement of the user's head; a determination unit that determines whether a first detection result obtained from the first detection unit and a second detection result obtained from the second detection unit are in a synchronous relationship, A wearable device, wherein when the first detection result and the second detection result are synchronized, the display unit displays a name related to an object in the real world image that is in the user's line of sight.

2. a display unit that displays a real world image or an augmented reality image that adds information to the real world image; a first detection unit that detects movement of the user's line of sight; a second detection unit that detects a movement of the user's head; a determination unit that determines whether a first detection result obtained from the first detection unit and a second detection result obtained from the second detection unit are in a synchronous relationship, A display method in which, when the first detection result and the second detection result are in a synchronized relationship, the display unit displays a name related to an object in the real world image that is in the user's line of sight.

Images