Wearable device and display method

Abstract

To provide a wearable device and a display method that changes the content to be displayed as a line of sight moves.SOLUTION: A wearable device includes: a display unit which displays a real-world image and an augmented reality image; a first detection unit which detects a line of sight and eye movement; a second detection unit which detects motion of the head; and a determination unit which determines whether the eye movement is synchronized with the motion of the head. When the device determines that the eye movement is not synchronized with the motion of the head, the display unit displays the augmented reality image. The wearable device includes a processing unit configured to select, when the type of the eye movement indicates closed eyes, an information item gazed, out of a plurality of information items, and start, when the type of the eye movement indicates wink, execution of processing on the selected information item.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to a wearable device and a display method. [Background technology]

[0002] Usually, when a person walks while looking ahead, their head sways up and down and side to side, but they can see the surrounding scenery clearly. This is because the head's swaying during walking is limited to a smaller range than other parts of the body (with an amplitude of about 2 to 3 cm regardless of the distance of the line of sight), and the head and eyes move in a coordinated manner within that small range to keep the line of sight stable (see Non-Patent Document 1). Non-Patent Document 1 teaches that "when walking, the head sways up and down and side to side, and the head rotates and the eyes rotate compensatingly to stabilize the line of sight in response to this swaying."

[0003] Regarding electrooculography sensing technology, there is known example 1 (see Patent Document 1) which detects a person's gaze position based on the electrooculography caused by the movement of the person's eyeballs. In known example 1, EOG (Electro-Oculography) electrodes are used for electrooculography detection (electrooculography sensing), and the user's gaze position is detected from electrooculography data acquired from the EOG electrodes (paragraph 0025). In the embodiment of known example 1, the EOG electrodes are provided on goggles (paragraph 0061), but there is also known example 2 in which electrodes for EOG are provided on glasses (see Patent Document 2). Goggles and glasses are a type of eyewear worn near a person's eyeballs. When a user wearing eyewear walks, the eyewear also moves together with the head. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2009-288529 A [Patent Document 2] JP 2013-244370 A [Non-patent literature]

[0005] [Non-Patent Document 1] Head and eye movements to maintain gaze stability while walking Paper 2000-03 Yazaki, Eiya Osaka University Graduate School of Human Sciences Bulletin. 26 P.177-P.193 http: / / ir.library.osaka-u.ac.jp / dspace / bitstream / 11094 / 5672 / 1 / hs26-177.pdf Summary of the Invention [Problem to be solved by the invention]

[0006] In a stationary seated state (a state in which there is no disturbing movement of the user's head while wearing eyewear with EOG electrodes), the gaze / eye rotation can be stably detected by electrooculography sensing. However, when the user is walking (a state in which the head of the user wearing eyewear with EOG electrodes shakes up and down and left and right) and gazes at a fixed object in the real world, it is more difficult to detect the desired gaze / eye rotation than when the user is seated. This is because the human body unconsciously rotates the eyes as a complementary action to stabilize the gaze, and the signal component of this unconscious eye rotation is superimposed (as noise or disturbance) on the electrooculography sensing results.

[0007] In other words, when a user wearing eyewear with EOG electrodes walks, even if the user fixes their gaze on a specific object in front of them, the EOG data acquired from the EOG electrodes fluctuates due to the head shaking that accompanies walking. This fluctuation in the EOG data becomes noise (disturbance) in the EOG data that corresponds to the actual gaze.

[0008] One of the problems to be solved by the embodiments of the present invention is to provide a wearable device and a display method that change the display content according to the movement of the line of sight. [Means for solving the problem]

[0009] A wearable device according to an embodiment includes a display unit that displays a real world image or an augmented reality image that adds a plurality of information items to the real world image, a first detection unit that detects a user's gaze and eye movement and detects closing of eyes or winking as a type of the user's eye movement, a second detection unit that detects a head movement of the user, and a determination unit that determines whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized, and the display unit displays the augmented reality image when the determination unit determines that the eye movement detected by the first detection unit and the head movement detected by the second detection unit are not synchronized. The wearable device further includes a processing unit that selects an information item in the gaze of the user from among the plurality of information items when the determination unit determines that the type of eye movement is closing of eyes, and starts execution of a process for the selected information item when the determination unit determines that the type of eye movement is winking. [Brief description of the drawings]

[0010] [Figure 1] A diagram explaining head shaking (position change due to up / down / left / right movement) of a person walking while looking ahead. [Diagram 2] A diagram explaining head movement (angle change due to bending forward / backward) of a person gazing at a fixed object in front of them while walking. [Diagram 3] FIG. 1 is a diagram explaining the relationship between visual target distance (the distance from the eye to the object of gaze) and eye movement when the head is bent forward and backward vertically. [Figure 4] FIG. 13 is a diagram illustrating the phase relationship between angle change / velocity change of head forward / backward bending and angle change / velocity change of eye movement. [Diagram 5] FIG. 1 is a diagram for explaining glasses-type eyewear incorporating an eye movement detection device according to an embodiment (an example in which EOG electrodes are arranged on a nose pad). [Figure 6] 1A to 1C are diagrams illustrating an example of implementation of EOG electrodes in glasses-type eyewear according to one embodiment. [Figure 7] FIG. 2 is a diagram for explaining the relationship between an information processing unit 11 that can be attached to various embodiments and its peripheral devices. [Figure 8] For example, this figure shows an example of an EOG waveform when starting to walk with the right foot while gazing at a fixed object in the real world 25 m away and stopping with the right foot on the 13th step. [Figure 9] This figure shows another example of an EOG waveform (to confirm repeatability) when walking under the same conditions as in Figure 8, starting again with the right foot and stopping with the right foot on the 13th step. [Figure 10] Electrooculogram (EOG) illustrating the relationship between eye movement from front to upward and the detection signal levels (Ch0, Ch1, Ch2, and the average level Ch1+2 of Ch1 and Ch2) obtained from the three analog-to-digital converters (ADCs) shown in Figure 6 when noise caused by head movement is canceled. [Figure 11] Electrooculogram illustrating the relationship between eye movement from front to downward and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in Figure 6 when noise caused by head movement is canceled out. [Figure 12] Electrooculogram illustrating the relationship between eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in FIG. 6 when noise caused by head movement is cancelled out. [Figure 13] An electrooculogram illustrating the relationship between eye movement in which blinking (both eyes) is repeated five times at 5-second intervals while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 14] An electrooculogram illustrating the relationship between eye movements, which are repeated five times, with eyes closed for 1 second (both eyes) and eyes open for 4 seconds (both eyes), when the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 15]An electrooculogram illustrating the relationship between eye movement in which both eyes are blinked five times, followed immediately by a wink of the left eye (blink of the left eye) five times, while the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 16] An electrooculogram illustrating the relationship between eye movement in which both eyes are blinked five times, followed immediately by a wink of the right eye (blink of one eye on the right side) five times, when the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out. [Figure 17] 11 is a flowchart illustrating an example of a process for minimizing noise caused by head movement (noise cancellation process). [Figure 18] FIG. 11 is a diagram for explaining an example of implementation of EOG electrodes in glasses-type eyewear according to another embodiment (an example in which EOG electrodes are arranged around the eyeball). [Figure 19] FIG. 11 is a diagram for explaining an example of implementation of EOG electrodes in goggle-type eyewear (a type in which the eye frames for both the left and right eyes are continuous) according to yet another embodiment (another example in which EOG electrodes are arranged around the eyeballs). [Figure 20] FIG. 11 is a diagram for explaining an example of implementation of EOG electrodes in goggle-type eyewear (a type in which the eyecups for the left and right eyes are separate) according to yet another embodiment (yet another example in which EOG electrodes are arranged around the eyeballs). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In the following, basic information will first be provided, and then various embodiments will be described with reference to the drawings. <Basic information> The diameter of an adult's eyeball is about 25 mm. After birth, it is about 17 mm, and grows larger as the animal grows. The interpupillary distance of an adult male is about 65 mm. (Most commercially available stereo cameras are made with a 65 mm distance.) The interpupillary distance of adult women is several millimeters shorter than that of men. The electrooculography is several tens of mV. The eyeball has a positive electric potential on the cornea side and a negative electric potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

[0012] The types of eye movements and eye movement ranges related to eye movement detection are, for example, as follows: <Types of eye movements> (01) Compensatory eye movements An involuntary eye movement that developed to stabilize an image of the external world on the retina regardless of head or body movement. (02) Voluntary eye movements This is an eye movement that developed to center the visual image on the retina and can be controlled voluntarily. (03) Impulsive eye movement (saccade) Eye movement (easy to detect) that occurs when changing the point of gaze to look at an object. (04) Sliding eye movements Smooth eye movements (hard to detect) that occur when tracking a slowly moving object. <Eye movement range (for a typical adult)> (11) Horizontal direction Left direction: 50° or less Right direction: 50° or less (12) Vertical direction Downward: 50° or less Upward: 30° or less (The vertical angle range that can be moved at will is narrow only in the upward direction. (Due to the "Bell phenomenon" in which the eyes roll upward when the eyes are closed, the vertical range of eye movement shifts upward when the eyes are closed.) (13)Other Convergence angle: 20° or less. <Head movement during walking (Figs. 1 and 2 are taken from Non-Patent Document 1)> FIG. 1 is a diagram explaining head sway (position change due to up / down and left / right movement) of a person walking while looking ahead. The position of a person's head changes up and down due to the movement of the legs when walking (FIG. 1(a)). The walking period (one walking cycle) of a typical adult is, for example, about 0.6 seconds. At the same time, while walking, the position of the person's head also changes left and right (FIG. 1(b)). The period of this left and right position change is about 1.2 seconds, which is twice the walking period. The range of change in up and down position and left and right position change is about 2 to 3 cm each. This up / down and left and right movement is approximately linear and regular, with a period closely related to the walking period.

[0013] Figure 2 is a diagram explaining head sway (angle change due to forward / backward bending) of a person gazing at a fixed object in front while walking. When a person gazing at a visual target (target) in front starts walking, the head changes position up and down as they walk (Figure 2(a)). While gazing at the target, when the head reaches an upper position, the head bends forward (pitches down), and when the head reaches a lower position, the head bends backward (pitches up) (Figure 2(b)). Such forward / backward bending (vertical rotational movement of the head) occurs in response to (synchronized with) the up and down movement of the head during one walking cycle. Similarly, when the head changes position left and right (Figure 1(b)), a left and right rotational movement of the head occurs (synchronized with the up and down movement of the head).

[0014] In other words, while walking, a person's head rotates to compensate for up-down and left-right movements, bending forward and backward at a high or low position to help the line of sight capture the visual target (target) in front of the person. The same is true for left and right: when the body sways from side to side as the person walks and the head moves from side to side, the head rotates left and right (rotates back and forth). This type of head rotation is called "compensatory rotation" and is thought to help stabilize the line of sight.

[0015] In addition, the human torso rotates back and forth and side to side while walking, so the head must also compensate for this. By rotating the head in the opposite direction to these changes in the torso, the angular position of the head is kept horizontal. <Eye movement during walking (Fig. 3 and Fig. 4 are taken from Non-Patent Document 1)> Figure 3 is a diagram explaining the relationship between target distance (distance from the eye to the gaze target) and eye movement when the head is bent forward and backward in the vertical direction. Here, the target position when the head is fixed is set to HFP, and the head forward and backward rotation angle (φh) and the eyeball compensatory rotation angle (φen, φef) are illustrated for the cases when the actual target is closer than the HFP (Near Target) and farther away (Far Target).

[0016] As mentioned above, the head performs compensatory rotation, but this alone is not enough to completely stabilize the gaze. The head often rotates excessively or insufficiently to compensate for vertical movement. To compensate for the misalignment caused by the excessive or insufficient compensatory rotation, periodic compensatory rotation of the eyes is required while walking. For example, for a near target 30 cm away, the eyes perform compensatory rotation (φen) in the same direction as the head rotation / rotation (φh). As a result, even if the head rotation / rotation is insufficient, the eyes rotate in coordination with the head in the direction of movement to compensate for the insufficient head rotation. On the other hand, for a far target, for example 100 m away, the eyes perform compensatory rotation (φef) in the opposite direction to the head rotation / rotation (φh). As a result, even if the head rotation / rotation becomes excessive, the eyes rotate in the opposite direction to cancel out the excessive head rotation. (Note that there is no substantial difference in the amount of head movement while walking, whether the target is close or far away.) Fig. 4 is a diagram illustrating the phase relationship between the angle change / speed change of head bending and the angle change / speed change of eye movement. The head bending angle (φh) shown in Fig. 3 can be detected by a three-axis gyro (11e in Fig. 7) attached to eyewear. The eyeball rotation angle (φen, φef) shown in Fig. 3 can be detected by an eyeball movement detection unit (15 in Fig. 7) that uses the EOG of the eyeball.

[0017] The vertical head movement (vertical forward and backward bending) that accompanies walking changes in phase with the change in eye rotation when the visual target is near (Fig. 4(a)). On the other hand, when the visual target is far (Far Target), the vertical head movement (vertical forward and backward bending) changes in phase with the change in eye rotation (Fig. 4(b)). When the visual target is at the fixed position HFP in Fig. 3, eye rotation disappears, and the change in the dashed waveform shown in Fig. 4(a)(b) becomes minimal (almost horizontal in the figure) (the phase between the vertical head bending and eye rotation is inverted at this minimal point). A similar phase inversion is also seen between the change in the velocity of head vertical movement and the change in the velocity of eye rotation (Fig. 4(c)(d)).

[0018] Here, we call the signal change that occurs in the EOG due to the above-mentioned "changes in the vertical movement of the head accompanying walking" "body movement noise." Walking behavior can be detected by an acceleration sensor or gyro sensor, and eye movement behavior can be detected from the EOG. If body movement noise does not appear in the EOG when "walking" and "no eye movement" are detected, it indicates that "the eyewear user is gazing at the point (corresponding to position HFP in Figure 3) where the phase of the body movement noise is inverted depending on the distance of the visual target."

[0019] FIG. 5 is a diagram for explaining eyeglass-type eyewear 100 incorporating an eye movement detection device according to an embodiment (an example in which EOG electrodes are arranged on nose pads). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected by a bridge 103. The left and right eye frames 102, 101 and the bridge 103 can be made of, for example, an aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to a left temple bar 106 via a left hinge 104, and a left end piece (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to a right temple bar 107 via a right hinge 105, and a right end piece (right ear pad) 109 is provided at the tip of the right temple bar 107.

[0020] An information processing unit 11 (an integrated circuit of a few millimeters square), which is the main part of the eye movement detection device, is attached to the bridge 103. This information processing unit 11 can be configured with an LSI that integrates a microcomputer, memory, a communication processing unit, etc. (Details of the information processing unit 11 will be described later with reference to FIG. 7).

[0021] A small battery (BAT) such as a lithium ion battery is embedded, for example, in the left temple bar 106 (or in the right temple bar 107, or in the end piece 108 or 109) and serves as a power source necessary for the operation of the eyeglass-type eyewear 100. A left camera 13L is attached to the end of the left eye frame 102 closer to the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 closer to the right hinge 105. These cameras can be configured using ultra-small CCD image sensors.

[0022] These cameras (13L, 13R) may be configured as a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be placed in the position of these cameras to configure a distance sensor using an infrared camera and a laser. This distance sensor may also be configured with a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

[0023] Note that an embodiment may be considered in which a central camera (not shown) is provided in the bridge 103 instead of or in addition to the left and right cameras 13L / 13R. Conversely, an embodiment may also be considered in which no camera is provided at all (these cameras are shown as cameras 13 in FIG. 7).

[0024] A left display 12L is fitted into the left eye frame 102, and a right display 12R is fitted into the right eye frame 101. This display is provided in at least one of the left and right eye frames, and can be configured using film liquid crystal or the like. Specifically, one or both of the left and right displays 12L, 12R can be configured using a film liquid crystal display device that employs polymer dispersed liquid crystal (PDLC) without using a polarizing plate (this display is shown as display 12 in FIG. 7).

[0025] The transparent left and right displays 12L / 12R, which use film LCD, can be used as a means of providing Augmented Reality (AR), which adds image information such as numbers and letters to the real world seen through the glasses. A nose pad section is provided between the left and right eye frames 102, 101 and below the bridge 103. This nose pad section is composed of a pair of a left nose pad 150L and a right nose pad 150R. Right nose pad electrodes 151a and 151b are provided on the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided on the left nose pad 150L.

[0026] These electrodes 151a, 151b, 152a, and 152b are electrically isolated from one another and connected to three AD converters (ADCs 1510, 1520, and 1512) via insulated wiring materials (not shown). Outputs from these ADCs have different signal waveforms according to the eye movement of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the user's eye movement. The electrodes 151a, 151b, 152a, and 152b are used as gaze detection sensors, and together with the three AD converters, are components of the eye movement detection unit 15 in FIG. 7.

[0027] The eyewear 100 is fixed to the user's head (not shown) by the left and right nose pads (150L, ​​150R), the left and right temple bars (106, 107), and the left and right end pieces (108, 109). In this embodiment, only the left and right nose pads (150L, ​​150R), the left and right temple bars (106, 107), and the left and right end pieces (108, 109) directly touch the user's head (or face), but there may be an embodiment in which parts other than these (nose pads, temple bars, end pieces) come into contact with the user, for example, to match the voltage between the ADCs (1510, 1520, 1512) and the user's body.

[0028] The film liquid crystal of the right display 12R in FIG. 5 can display a right display image IM1 including, for example, a numeric keypad (numbers, operators, Enter, etc.), alphabets, marks of a predetermined shape, and other icons. The film liquid crystal of the left display 12L can display a left display image IM2 including, for example, an arbitrary character string, an arbitrary shaped mark, various icons, etc. (The display contents of the displays 12L and 12R can be anything). The numeric keypad and alphabets displayed on the right display 12R (or the left display 12L) can be used to input numbers and letters. The character strings, marks, icons, etc. displayed on the right display 12R (or the left display 12L) can be used to search for a specific information item, select / confirm a target item, or alert the user.

[0029] The display images IM1 and IM2 can be used as an augmented reality (AR) display means that adds information such as numbers, letters, and marks to the real world seen through the glasses, and this AR display can be turned on and off as appropriate. The contents of the display images IM1 and IM2 may be the same (IM1=IM2) or different (IM1≠IM2) depending on the embodiment. In addition, the display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. When it is desired to display the contents of the AR display as a 3D image (with depth) that overlaps with the real world seen through the glasses, IM1 and IM2 can be separate 3D images for the left and right.

[0030] In addition, when displays (12R, 12L) exist on the left and right, for example, the convergence angle can be adjusted to shift the images of the left and right display images (IM1, IM2) in opposite directions. This is thought to reduce the strain on the eyes when alternating between looking at objects visible in the real world and the AR display. However, normally, the left and right displays (12R, 12L) display images with the same content.

[0031] Display control of displays 12L, 12R can be performed by an information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on a display is well known.) The power required for the operation of the information control unit 11 and other components can be obtained from a battery BAT embedded in the left temple bar 106.

[0032] It should be noted that when a designer or engineer tries on a prototype of eyewear 100 corresponding to the embodiment, he or she may find that the weight balance is poor. If the main cause of this is the BAT in the left temple bar 106, then a "weight (or another battery of the same weight)" that matches the BAT in the left temple bar 106 can be placed in the right temple bar 107.

[0033] 6 is a diagram illustrating an example of implementation of EOG electrodes in glasses-type eyewear 100 according to an embodiment. Right nose pad electrodes 151a and 151b are provided above and below right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below left nose pad 150L. The outputs of the right nose pad electrodes 151a and 151b are provided to ADC 1510, the outputs of the left nose pad electrodes 152a and 152b are provided to ADC 1520, and the outputs of the lower electrodes 151b and 152b (or upper electrodes 151a and 152a) of the left and right nose pads are provided to ADC 1512.

[0034] ADC1510 provides a Ch1 signal that changes in response to the user's right up and down eye movement. ADC1520 provides a Ch2 signal that changes in response to the user's left up and down eye movement. ADC1512 provides a Ch0 signal that changes in response to the user's left and right eye movement. The up and down movement of both the left and right eyes can be evaluated by the Ch1+2 signal that corresponds to the average of the outputs of ADC1510 and ADC1520. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, Ch1+2 and eye movement will be described later.) Fig. 7 is a diagram for explaining the relationship between the information processing unit 11 that can be attached to various embodiments and its peripheral devices. In the example of Fig. 7, the information processing unit 11 is composed of a processor 11a, a non-volatile memory 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, and the like. The processor 11a can be composed of a microcomputer having a processing capacity according to the product specifications. Various programs executed by this microcomputer and various parameters used when executing the programs can be stored in the non-volatile memory 11b. The main memory 11c provides a work area when executing the programs.

[0035] The sensor unit 11e includes a group of sensors for detecting the position and / or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensors include an acceleration sensor that detects movement in three axial directions (x, y, and z directions), a gyro that detects rotation in three axial directions, a geomagnetic sensor (compass function) that detects absolute orientation, and a beacon sensor that receives radio waves, infrared rays, and the like to obtain location information and other information. To obtain this location information and other information, iBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used.

[0036] LSIs that can be used for the information processing unit 11 have been commercialized. One example is the "TZ1000 series for wearable devices" from Toshiba Semiconductor & Storage Corporation. Of this series, the product with the name "TZ1011MBG" has a CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), a group of sensors (accelerometer, gyro, geomagnetic sensor) (11e), a 24-bit delta-sigma ADC, and I / O (USB, etc.).

[0037] The information processing unit 11 is attached to the bridge 103 as an example of a location where the relative positional relationship between the sensor unit 11e, such as an acceleration sensor or a gyro sensor, and the EOG electrodes (151a, 151b, 152a, 152b) is not lost. What the processor 11a does can be instructed from an external server (or personal computer) (not shown) via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth, and Wi-Fi (registered trademark). The processing results of the processor 11a can be sent to a server (not shown) via the communication processing unit 11d.

[0038] The display 12 (12L and 12R), the camera 13 (13L and 13R), the eye movement detection unit 15, etc. are connected to the system bus of the information processing unit 11. Each device (11 to 15) in FIG. 7 is powered by a battery BAT. The communication processing unit 11d may incorporate a GPS (Global Positioning System) function well known in mobile phones and smartphones (or a mobile phone function with GPS may be installed somewhere in the eyewear 100). By executing an Internet map service application using this GPS function on the processor 11a, it is possible to detect where the user wearing the eyewear 100 is currently walking and in what direction he or she is looking while suppressing noise caused by compensatory eye rotation while walking. (Although conventional mobile phone GPS can display the user's current location on a map screen, it cannot display where the user's gaze is directed.) 7 includes four eye movement detection electrodes (151a, 151b, 152a, 152b) constituting a gaze detection sensor, three ADCs (1510, 1520, 1512) that extract digital signals corresponding to eye movements from these electrodes, and a circuit that outputs the output data from these ADCs to the processor 11a. The processor 11a can interpret commands corresponding to the types of eye movements from various types of eye movements of the user (up-down movements, left-right movements, blinking, closing the eyes, etc.) and execute the commands.

[0039] A specific example of a command corresponding to the type of eye movement is a command to select an information item in front of the line of sight if the eye movement is, for example, eye closing (similar to one click of a computer mouse), and to start execution of processing for the selected information item if the eye movement is, for example, multiple consecutive blinks or winks (similar to double click of a computer mouse). This command is an example of information input B using the eye movement detection unit 15.

[0040] FIG. 8 is a diagram showing an example of an EOG waveform when, for example, a person starts walking with the right foot while gazing at a fixed object in the real world located 25 m ahead and stops with the right foot on the 13th step. Compensatory rotation of the eyeballs corresponding to the vertical movement of the head accompanying walking is expressed in the change in the ADC detection signal level (EOG change) of Ch1 and Ch2. The vertical movement change of the head synchronized with this EOG change can be detected by the three-axis acceleration sensor of the sensor unit 11e of the information processing unit 11. Compensatory rotation of the eyeballs corresponding to the left-right rotation of the head accompanying walking is expressed in the change in the ADC detection signal level (EOG change) of Ch0. The left-right rotation change of the head synchronized with this EOG change can be detected by the three-axis gyro sensor of the sensor unit 11e of the information processing unit 11.

[0041] In the example of Figure 8, the EOG fluctuation period of Ch1 and Ch2 (approximately 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. Since the EOG fluctuation period corresponding to the up and down movement and left and right rotation of the head accompanying walking is clear, the EOG fluctuation caused by walking can be detected with a period synchronized with the walking pitch (approximately 0.6 seconds and twice that in this example).

[0042] Figure 9 shows another example of the EOG waveform (confirmation of repeatability) when walking again with the right foot under the same conditions as in Figure 8 and stopping with the right foot on the 13th step. In the example of Figure 9, the EOG fluctuation period of Ch1 and Ch2 (about 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (about 1.2 seconds) is synchronized with twice the walking pitch. The EOG fluctuation period corresponding to the vertical movement and left and right rotation of the head accompanying walking is also clearly recognized, and it can be seen that the EOG fluctuation caused by walking can be detected by the period synchronized with the walking pitch (about 0.6 seconds and twice that in this example).

[0043] Figures 8 and 9 share the following characteristics, confirming the reproducibility of the event: (1) Up and down eye rotation can be observed in time with the walking cycle (approximately 0.6 seconds) (see EOG detection signal levels of Ch1 and Ch2). (2) Eyeball rotation in the left and right direction can be confirmed in accordance with a period (approximately 1.2 seconds) twice the walking period (see the EOG detection signal level of Ch0).

[0044] (3) The amplitude of the EOG detection signal caused by head swaying accompanying walking reaches 200μV to 400μV in some places. Since the EOG signal amplitude measured with the head stationary is usually about 1mV, when detecting any eye rotation, 200μV to 400μV becomes a noise of a magnitude that cannot be ignored. This noise can be canceled (reduced or eliminated) as described below.

[0045] That is, the vertical movement of the head accompanying walking can be detected by the acceleration sensor in a period (about 0.6 seconds in this example) synchronized with the walking pitch, and the left-right rotation of the head accompanying walking can be detected by the gyro sensor in another period (about 1.2 seconds in this example) synchronized with the walking pitch. In this way, the EOG fluctuation (considered as noise) caused by the vertical movement (sway) of the head accompanying walking can be canceled (reduced or eliminated) by the detection result of the acceleration sensor, and the EOG fluctuation (considered as noise) caused by the left-right rotation (sway) of the head accompanying walking can be canceled (reduced or eliminated) by the detection result of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

[0046] FIG. 10 is an electrooculogram (EOG) illustrating the relationship between eye movement from the front to the upward direction and the detection signal level (Ch0, Ch1, Ch2, and the average level Ch1+2 of Ch1 and Ch2) obtained from the three analog-to-digital converters (ADCs) shown in FIG. 6 when noise caused by head movement is canceled. Eye movement detection is performed based on the detection signal waveform within the dashed frame in the figure. The detection standard is that the user wearing the eyewear 100 looks straight ahead and has no eye movement (if noise caused by head movement is canceled, the output signal waveforms Ch0 to Ch2 from the three ADCs shown in FIG. 6 are approximately flat and have almost no change over time in the section where the user looks straight ahead without blinking). With the gaze of both the left and right eyes of the user facing forward, the gaze of both eyes is instantly moved upward, the gaze is maintained upward for one second, and then the gaze is instantly returned to the front. The change in the detection signal level when this is repeated five times is illustrated in FIG. 10.

[0047] Figure 11 illustrates an example of eye movement detection similar to that in Figure 10, where the gaze moves downward from the front. If noise caused by head movement is cancelled out, it is possible to detect whether the gaze is moving up or down, based on the case where the gaze is facing forward, from the waveform changes in Figures 10 and 11, even while walking.

[0048] Fig. 12 is an electrooculogram illustrating the relationship between eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in Fig. 6 when noise caused by head movement is cancelled. When there is eye movement from left to right, the change over time in the detection signal waveform of Ch0 rises to the right (although not shown, when there is eye movement from right to left, the change over time in the detection signal waveform of Ch0 falls to the right). From such a change in the waveform of Ch0, it can be detected whether the gaze is to the right or left, based on the case where the gaze is facing forward.

[0049] 10 to 12 together reveal the following: If noise caused by head movement is cancelled out, it is possible to determine whether the gaze is directed up, down, left, or right, even while the person is walking, based on the case where the gaze is directed straight ahead. FIG. 13 is an electrooculogram illustrating the relationship between eye movement in which blinking (both eyes) is repeated five times at five-second intervals when the gaze is directed forward and noise caused by head movement is cancelled, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. Blinking of both eyes can be detected by pulses appearing on Ch1 and Ch2. Blinking that is performed unconsciously by the user is often not periodic. Therefore, intentional blinking of the user can be detected by detecting multiple pulses at regular intervals as shown in FIG. 13.

[0050] According to one theory, the duration of a person's "blinking" action or the time that vision is blocked by "blinking" is about 300 msec (according to another theory, it is about 100 msec to 150 msec). In either case, the signal components due to "blinking" can be removed by using a high-pass filter (or band-pass filter) that is lower than the walking pitch period. The interval between blinks (for example, about 300 msec) is different from the pitch of normal walking (about 600 msec), and the interval between unintentional blinks is not synchronized with the normal walking pitch. Therefore, when canceling out EOG noise caused by head movement associated with normal walking, the EOG waveform changes due to blinking are not all canceled out.

[0051] Fig. 14 is an electrooculogram illustrating the relationship between eye movement in which one second of eye closing (both eyes) and four seconds of eye opening (both eyes) are repeated five times when the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Fig. 6, in the case where noise caused by head shaking is cancelled. Closing of both eyes can be detected by wide pulses appearing on Ch1 and Ch2 (the time when the eyes are intentionally closed is longer than the time when the eyes are closed by blinking / blinking, so the detected pulse width is wider). By detecting the wide pulses on Ch1 and Ch2 as exemplified in Fig. 14, the user's intentional closing of the eyes can be detected.

[0052] Although not shown, when the user closes only the right eye, a wide pulse with a large amplitude appears on Ch1, and when the user closes only the left eye, a wide pulse with a large amplitude appears on Ch2. This makes it possible to detect eye closure separately for the left and right eyes. When one eye is closed, small waves with opposite phases appear on Channel 0. When viewed through Channel 0, a small uneven waveform like the one shown below appears.

[0053] When the right eye is closed, the positive potential of the - electrode decreases, resulting in a convex waveform; When the left eye is closed, the positive potential of the + electrode decreases, resulting in a concave waveform. Figure 15 is an electrooculogram illustrating the relationship between eye movement in which both eyes are blinked five times, followed immediately by a wink of the left eye (blink of the left eye) five times, when the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in Figure 6, when noise caused by head movement is canceled out.

[0054] As illustrated in Fig. 6, the position of the ADC1512 of Ch0 is offset downward from the eyeball center lines of both the left and right eyes. Due to this offset, when both eyes blink simultaneously, a negative potential change appears on both the + input and the - input of the ADC1512 of Fig. 6. In this case, if the potential changes (amount and direction) of both the + input and the - input are approximately the same, the changes are almost cancelled out, and the value of the signal level output from the ADC1512 of Ch0 becomes approximately constant (see the Ch0 level within the dashed line on the left side of Fig. 15). On the other hand, when one eye (left eye) blinks, there is almost no potential change on the - input side of the ADC1512, and a relatively large negative potential change appears on the + input side of the ADC1512. Then, the amount of cancellation of the potential change between the + and - inputs of the ADC1512 becomes smaller, and a small pulse (signal level wavelet) appears in the negative direction in the signal level output from the ADC1512 for Ch0 (see the Ch0 level within the dashed line on the right side of Figure 15). From the polarity of this signal level wavelet (negative pulse), it is possible to detect that a wink of the left eye has occurred (an example of left wink detection using Ch0 when EOG noise caused by head movement is cancelled).

[0055] If the potential changes in the + and - inputs of ADC1512 are not equal due to distortion of the user's face or skin condition, calibration can be performed in advance so that the output of Ch0ADC is minimized (the amount of cancellation between the + and - input components is maximized) when the user wears eyewear 100 and blinks both eyes simultaneously.

[0056] In addition, if the peak ratio SL1a / SL2a of the detection signals Ch1 / Ch2 when both eyes blink is used as a reference, the peak ratio SL1b / SL2b when a left wink is performed changes (SL1b / SL2b is not equal to SL1a / SL2a). This also makes it possible to detect a left wink.

[0057] FIG. 16 is an electrooculogram illustrating the relationship between eye movement in which both eyes are blinked five times, followed immediately by a wink of the right eye (blinking of one eye on the right side) five times, when the gaze is directed forward, and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6, when noise caused by head movement is cancelled out.

[0058] As described above, since the position of the ADC1512 in FIG. 6 is offset downward from the eyeball center line of both the left and right eyes, a negative potential change appears on both the + input and the - input of the ADC1512 when both eyes blink simultaneously. However, similar potential changes at the + input and the - input are almost cancelled out, and the value of the signal level output from the ADC1512 of Ch0 is approximately constant (see the Ch0 level within the dashed line on the left side of FIG. 16). On the other hand, when one eye (right eye) blinks, there is almost no potential change on the + input side of the ADC1512, and a relatively large negative potential change appears on the - input side of the ADC1512. Then, the amount of cancellation of the potential change between the - input and the + input of the ADC1512 becomes small, and a small pulse (small wave of the signal level) appears in the positive direction on the signal level output from the ADC1512 of Ch0 (see the Ch0 level within the dashed line on the right side of FIG. 16). From the polarity of this signal level wavelet (positive-going pulse), it is possible to detect a wink of the right eye (an example of right wink detection using Ch0 when EOG noise caused by head movement is cancelled out).

[0059] In addition, if the peak ratio SR1a / SR2a of the detection signal Ch1 / Ch2 when both eyes blink is used as a reference, the peak ratio SR1b / SR2b changes when a right wink is performed (SR1b / SR2b is not equal to SR1a / SR2a). In addition, the peak ratio SL1b / SL2b when a left wink occurs has a different value from the peak ratio SR1b / SR2b when a right wink occurs (the extent to which they differ can be confirmed through experiments). This makes it possible to detect a left wink separately from a right wink (an example of left / right wink detection using Ch1 and Ch2).

[0060] The device designer can decide whether to use Ch0 or Ch1 / Ch2 for left and right wink detection. The left and right wink detection results using Ch0 to Ch2 can be used as operation commands. Fig. 17 is a flowchart for explaining an example of a process for minimizing noise caused by head movement (noise cancellation process). A computer program corresponding to this flowchart is stored in the non-volatile memory 11b in Fig. 7, for example, and executed by the processor 11a.

[0061] First, a user wearing eyewear 100 as shown in Fig. 5 walks at a constant pace while gazing at a fixed object in front of him (ST10). As he walks, the user's head sways (moves / rotates) in a predetermined pattern (see Figs. 1(a) and (b)). The movement / rotation of the user's head accompanying walking is detected by an acceleration sensor / gyro sensor (11e in FIG. 7) provided inside the information processing unit 11 in FIG. 5 (ST12). The movement / rotation of the user's head accompanying walking is mixed into the EOG detection signal waveform as noise (see FIG. 8 or FIG. 9). The detection signal of the acceleration sensor / gyro sensor is added to this EOG detection signal waveform at a predetermined phase (ST14).

[0062] If the signal amplitude increases as a result of the above signal addition (NO in ST16), the phase of the detection signal to be added is inverted (ST18), and the detection signal of the acceleration sensor / gyro sensor is added to the EOG detection signal waveform again (ST14). If the signal amplitude is reduced as a result of the signal addition (YES in ST16), the level of the signal to be added is adjusted so that the signal amplitude of the addition result becomes minimum (extremely small) (ST20). By minimizing the signal amplitude of the addition result, the noise mixed in with the EOG detection signal waveform is cancelled (reduced or eliminated).

[0063] Based on the EOG detection result after the noise caused by head shaking (movement / rotation) is cancelled, the user's eye movement (gaze movement, blinking, closing eyes, winking, etc. in Figs. 10 to 16) is determined (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, the name of the building in front of the line of sight can be displayed in AR on the display screen (IM1 / IM2 in Fig. 5) of the eyewear 100.

[0064] When the user's head vibration pattern changes, the noise cancellation content also changes. Therefore, whether the user's head vibration pattern has changed is detected from the change in the detection result of the acceleration sensor / gyro sensor (ST26). For example, suppose that a user of the eyewear 100 who is walking at a constant speed while gazing at a fixed object in front of him / her stops in front of a staircase leading down to the subway and sweeps his / her eyes over the step between adjacent steps of the staircase by bending forward and backward and moving his / her eyes. Then, the head vibration pattern becomes different from that when walking due to the gaze sweep before and after this step, and the EOG detection signal waveform also becomes different from that when walking.

[0065] Therefore, if the shaking pattern of the user's head has changed (YES in ST26), the process returns to step ST12 and repeats the processes of ST14 to ST20. By repeating the processes of ST14 to ST20, the EOG noise caused by the new shaking pattern of the head is cancelled (reduced or eliminated), and the EOG corresponding to the pure movement of the line of sight is detected. The EOG detection result corresponding to the pure movement of the line of sight is compared with the detection result of the acceleration sensor / gyro sensor corresponding to the changed shaking pattern of the user's head. If, as a result of this comparison, a phase inversion from FIG. 4(a) to FIG. 4(b) is detected at the step position of the stairs, for example, it is determined that there is a step of the stairs near the phase inversion point (ST22). Then, a mark indicating the presence of the step is displayed in AR on the eyewear 100 (ST24). This AR display can be automatically erased a certain time (for example, 10 seconds) after the detection result of the acceleration sensor / gyro sensor corresponding to the shaking pattern of the user's head has disappeared.

[0066] The AR display can also be turned on / off automatically depending on whether the EOG waveform (Ch1 / Ch2 waveform in Fig. 8 or Fig. 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeballs are not rotating to compensate for head movement, so the above-mentioned synchronization relationship does not exist, but when the user is gazing at the real world instead of the AR display, the above-mentioned synchronization relationship occurs. Using a similar concept, the AR display can be turned on / off automatically depending on whether the EOG waveform (Ch0 waveform in Fig. 8 or Fig. 9) and the detection signal waveform of the gyro sensor are synchronized.

[0067] If the user continues to use the eyewear while the shaking pattern of the user's head does not change (NO in ST28), the processes in ST22 to ST24 continue. If the user closes their eyes for a few seconds, for example, to instruct the processor 11a in Fig. 7 to end use of the eyewear 100 (YES in ST28), the process in Fig. 17 ends.

[0068] Fig. 18 is a diagram for explaining an example of implementation of EOG electrodes in glasses-type eyewear 100 according to another embodiment (an example in which EOG electrodes are arranged around the eyeball). The eyewear 100 in Fig. 18 differs from the eyewear 100 in Fig. 5 in the following points. The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R are moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved to the left eye frame 102 side. The EOG electrodes 151a and 151b are arranged at positions that are approximately point-symmetrical with respect to the center position of the user's right eye (not shown), and the EOG electrodes 152a and 152b are arranged at positions that are approximately point-symmetrical with respect to the center position of the user's left eye (not shown). In addition, the right oblique line connecting the EOG electrodes 151a and 151b and the left oblique line connecting the EOG electrodes 152a and 152b are approximately line-symmetrical with respect to a vertical line (not shown) along the bridge of the user's nose. The electrodes 151a, 151b, 152a, and 152b can be made of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, silicone cushion, etc.). Each EOG electrode is lightly pressed against the facial skin of a user wearing eyewear 100 by the elastic repulsive force of an elastic body.

[0069] The EOG electrode arrangement structure shown in Fig. 18 makes it easier to detect the electric field generated around the charged eyeball compared to a structure in which the EOG electrodes are arranged on the nose pads. Therefore, the embodiment in Fig. 18 can detect an EOG signal with a larger amplitude than the embodiment in Fig. 5.

[0070] The second difference is that the sensor unit 11e including the acceleration sensor and gyro sensor is left in the bridge 103, and the other functions of the information processing unit 11 are moved to the temple bar 107 side. For example, if the physical size of the information processing unit 11 increases due to an improvement in information processing capability or the addition of a GPS function, attaching the enlarged information processing unit 11 to the bridge 103 may cause problems in the design and fit of the eyewear 100. If the structure attached to the bridge 103 is only the sensor unit 11e, the structure on the bridge 103 is smaller and lighter than if the entire information processing unit 11 is provided, so the design and fit may be improved. On the other hand, if a relatively large structure is provided inside the temple bar 107 (and / or 106), there is little risk of problems in the design and fit of the eyewear 100.

[0071] FIG. 19 is a diagram for explaining an example of mounting an EOG electrode in goggle-type eyewear (a type in which eye frames for both the left and right eyes are continuous) 100 according to yet another embodiment (another example in which an EOG electrode is disposed around the eyeball). When the structure shown in FIG. 19 is adopted, the skin contact stability of the EOG electrode is higher than that of the structure shown in FIG. 18. Therefore, the EOG signal can be detected with higher accuracy while canceling out noise caused by head shaking. In addition, the structure shown in FIG. 19 makes it easy to incorporate a large information processing unit 11 and other devices. Therefore, the structure shown in FIG. 19 can incorporate the functions of a smartphone with GPS without worrying too much about design. In that case, the screen display can be performed by AR display on the left and right displays 12L / 12R using film liquid crystal or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frame, and a small microphone can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movement such as gaze movement, blinking, closing the eyes, and winking.

[0072] FIG. 20 is a diagram for explaining an example of the implementation of EOG electrodes in goggle-type eyewear (a type in which eyecups for the left and right eyes are separated) 100 according to yet another embodiment (another example in which EOG electrodes are arranged around the eyeballs). In the embodiment of FIG. 20, the same EOG electrode arrangement structure and arrangement structure of the sensor unit 11e as in FIG. 18 are adopted. However, the goggle structure of FIG. 20 can withstand underwater use (or spacewalk training use). The head of a diver swimming underwater vibrates more than when walking on land, but the large vibration (positional movement and rotation in the up and down and left and right directions) can be detected by a three-axis acceleration sensor and a three-axis gyro sensor in the sensor unit 11e. The detected vibration component can be used to cancel noise components mixed into the EOG signal. Using the EOG signal in which noise caused by head vibration is suppressed, various commands can be input based on the eye movement (blinking, closing the eyes, etc.) of a user (diver) with both hands full. <Summary of the embodiment> (a) Conventional gaze detection technology involves estimating gaze direction through image processing using an infrared camera, but the amount of equipment required (high-brightness infrared LEDs, a camera capable of capturing an image of the entire eyeball as seen from the outside, a relatively high-performance computing device for image processing, and a power supply device to operate these) is large. For this reason, while conventional gaze detection technology is applied to stationary devices, its application to wearable devices has not progressed.

[0073] In contrast, gaze detection using electrooculography requires a much smaller amount of equipment (electrodes, ADC, a relatively low-performance computing device, and a power supply to operate these) than the infrared camera method, making it possible to apply it to wearable devices such as AR glasses. EOG is highly compatible with AR glasses and consumes less power than the infrared method, so it is a technology that is expected to be applied to wearable devices.

[0074] (b) Cancelling out the head movement of the eyewear user when gazing at a fixed object in the real world, thereby improving the detection accuracy for arbitrary eye rotations. (c) Detect whether the fixed object in the real world that the user is gazing at is a distant or close object (detection of the phase inversion point in Figure 4). This allows the system to detect whether the user is gazing at a fixed object in the real world from the shaking of the eyewear user's head.

[0075] (d) The eyewear user's head shaking is used to automatically turn the AR display on and off. (e) Extract only arbitrary eye movements that cancel out compensatory eye rotations. (f) Detect whether what you are looking at is the real world or not (if the detection waveform of the acceleration sensor and / or gyro sensor and the electro-oculography EOG waveform are synchronized while walking, it is determined that you are looking at something in the real world, and if the detection waveform of the acceleration sensor and / or gyro sensor and the electro-oculography EOG waveform are not synchronized, it is determined that you are looking at an AR display or the like while walking).

[0076] (g) Effective for canceling noise when using eye movements as a UI (user interface). (h) When used to collect work history information using eyewear, it can become an unprecedented indicator (for example, in a state where the effects of noise caused by head movement are eliminated, it can be used to determine whether a worker passes or fails a job, such as when picking a warehouse worker, if the degree to which the worker directs their gaze at the correct target is above a certain value). <Examples of correspondence between claims and embodiments at the time of filing> [1] An electro-oculography detection device according to one embodiment (including processor 11a for executing the process of FIG. 17: 11 in FIG. 7) includes an eye movement detection unit (15; 151a, 151b, 152a, 152b) that detects eye movement (gaze movement, blinking, etc.) including eye rotation of a user wearing eyewear (100) based on the electro-oculography (EOG) of the user, an acceleration sensor (part of 11e) that detects the movement of the eyewear, and a gyro sensor (part of 11e) that detects the rotation of the eyewear. This eye movement detection device includes a detection means (11a executing ST12) that detects movement and / or rotation of the eyewear using the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electrooculogram (EOG), and a noise reduction means (11a executing ST14 to ST20) that reduces (minimizes or cancels) noise (electrooculogram components added by compensatory eye rotation acting in a direction that cancels out the movement of the head) that is mixed into the electrooculogram due to the movement and / or rotation of the eyewear by combining signal components (components that are in phase opposite to the mixed noise) detected by the acceleration sensor and / or the gyro sensor.

[0077] [2] The device of [1] further includes a phase selection means (11a which executes ST16 to ST18) for selecting the phase of the signal components detected by the acceleration sensor and / or the gyro sensor so that the signal synthesis result by the reduction means tends to decrease.

[0078] [3] The device (11) of [1] is incorporated into eyewear (100 in Figs. 5, 18 to 20). [4] The eyewear (100) of [3] has display units (12L, 12R) and is equipped with a display means (11a that executes ST22 to ST24) that performs display (AR display) on the display units (12L, 12R) based on the electro-oculography (EOG) after noise has been reduced by the noise reduction means.

[0079] [5] The device of [1] detects the rotation corresponding to the forward and backward bending of the user's head using the gyro sensor (11e), detects the change in the user's gaze direction (Figures 10 and 11: up and down gaze direction changes can be detected from the ± level changes of Ch1 / Ch2) using the eye movement detection unit (15), and is configured to identify the perspective of the place where the user is looking depending on whether the signal phase of the forward and backward bending change and the signal phase of the gaze direction change are in phase (Figure 4(a)) or out of phase (Figure 4(b)) (it can detect whether a fixed object in the real world that is being gazed at is a distant or close view).

[0080] [6] The device of [1] is configured to determine that the user is looking at the real world (it can detect whether what is being gazed at is the real world) when the eye movement is detected based on the electro-oculography (EOG) and the movement and / or rotation of the eyewear detected by the acceleration sensor and / or the gyro sensor (Figure 1) is synchronized with the change in the electro-oculography (EOG) (Figure 8 or Figure 9) (when a "predetermined phase relationship" can be maintained in ST14 of Figure 17).

[0081] [7] An electro-oculography detection method according to one embodiment includes a process for detecting eye movement (gaze movement, blinking, etc.) including eye rotation of a user wearing eyewear (100) based on the electro-oculography (EOG) of the user, detecting head movement of the user wearing the eyewear, and detecting head rotation of the user wearing the eyewear. This method includes a step of detecting head movement and / or rotation of the user wearing the eyewear when the eye movement is detected based on the electro-oculography (EOG) (ST12), and a step of reducing (minimizing or canceling) noise (electro-oculography component added by compensatory eye rotation acting in a direction to cancel the head movement) mixed into the electro-oculography due to the head movement and / or rotation of the user wearing the eyewear by combining signal components (components in phase opposite to the mixed noise) detected in response to the head movement and / or rotation of the user wearing the eyewear (ST14 to ST20).

[0082] [8] The method of [7] further includes steps (ST16 to ST18) of selecting the phase of the signal components detected in response to the movement and / or rotation of the head of a user wearing the eyewear so that the combined result of the signal components tends to decrease. Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention.

[0083] For example, in the description of the embodiments, glasses-type and goggle-type devices were introduced as eyewear equipped with EOG electrodes, but it is also possible to use other items such as eye masks, eye patches, helmets, hats, and hoods as eyewear with EOG electrodes. These embodiments and modifications thereof are included within the scope and spirit of the invention, and are included within the scope of the invention and its equivalents as set forth in the claims. Note that a combination of a part or all of one embodiment among the disclosed embodiments and a part or all of another embodiment among the disclosed embodiments is also included within the scope and spirit of the invention. [Explanation of symbols]

[0084] 100... Eyewear (goggle type or glasses type); 110... Eye frame; 11... Information processing unit which is the main part of the eye movement detection device (integrated circuit including processor 11a, non-volatile memory 11b, main memory 11c, communication processing unit (GPS processing unit) 11d, sensor unit 11e, etc.); 11e... Sensor unit including acceleration sensor, gyro sensor, etc.; BAT... Power source (lithium ion battery, etc.); 12... Display unit (right display 12R and left display 12L: film liquid crystal, etc.); IM1... Right display image ( Numeric keypad, alphabet, character string, marks, icons, etc.); IM2...left display image (numeric keypad, alphabet, character string, marks, icons, etc.); 13...camera (right camera 13R and left camera 13L, or a center camera not shown attached to the bridge 103 portion); 15...eye movement detection unit (gaze detection sensor); 1510...right side (Ch1) AD converter; 1520...left side (Ch2) AD converter; 1512...left-right (Ch0) AD converter; 1514...left-right (Ch3) AD converter.

Claims

1. a display unit for displaying a real world image or an augmented reality image that adds a number of information items to the real world image; A first detection unit that detects a line of sight and eye movement of a user and detects closing of the eyes or winking as a type of the eye movement of the user; A second detection unit that detects a movement of the user's head; a determination unit that determines whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized, a wearable device in which, when the determination unit determines that the eye movement detected by the first detection unit and the head movement detected by the second detection unit are not synchronized, the display unit displays the augmented reality image, The wearable device further includes a processing unit that, when the determination unit determines that the type of eye movement is the eye closing, selects an information item in front of the line of sight from among the multiple information items, and, when the determination unit determines that the type of eye movement is the wink, starts executing processing on the selected information item.

2. a display unit for displaying a real world image or an augmented reality image that adds a number of information items to the real world image; A first detection unit that detects a line of sight and eye movement of a user and detects closing of the eyes or winking as a type of the eye movement of the user; A second detection unit that detects a movement of the user's head; a determination unit that determines whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized, When the determination unit determines that the eye movement detected by the first detection unit and the head movement detected by the second detection unit are not synchronized, the display unit displays the augmented reality image; and When the determination unit determines that the type of eye movement is the eye closing, an information item in front of the line of sight is selected from the plurality of information items, A display method, comprising: starting execution of a process for the selected information item when the determination unit determines that the type of eye movement is the wink.

Images