Wearable device, system, and display method

Abstract

To provide a wearable device, a system, and a display method that control the display on the basis of the results of detecting the rotation of the right eye and the rotation of the left eye.SOLUTION: The wearable device is configured to determine whether a user is looking at a first object at a first distance in the real world or looking at a second object at a second distance which is different from the first distance on the basis of the angle of convergence detected by a detector, display a first augmented reality image on a display when the user is determined to be looking at the first object, transmit a request signal to an electronic device when the user is determined to be looking at the second object, and display a second virtual reality image transmitted from the electronic device in response to the request signal on the display.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present invention relates to a wearable device, a system, and a display method. [Background technology]

[0002] One method for detecting eye rotation is the electrooculography (EOG) method. By attaching electrodes to the skin near the left and right eyeballs, the electrooculography of the left and right eyeballs can be detected. Eye rotation can be detected based on the change pattern of the electrooculography of the left and right eyeballs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-125693 A [Patent Document 2] U.S. Pat. No. 8,449,116 [Patent Document 3] JP 2011-125692 A [Patent Document 4] U.S. Pat. No. 8,434,868 [Patent Document 5] JP 2013-240469 A [Patent Document 6] JP 2000-259336 A [Patent Document 7] US Patent Application Publication No. 2011 / 0,178,784 [Patent Document 8] JP 2013-244370 A [Patent Document 9] JP 2013-215356 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional eye rotation detection devices cannot detect right eye rotation and left eye rotation separately.

[0005] An object of the present invention is to provide a wearable device, system, and display method that controls display based on the detection results of right eyeball rotation and left eyeball rotation. [Means for solving the problem]

[0006] According to an embodiment, a wearable device connected to an electronic device includes: a frame that is worn by a user and includes a transparent portion through which the user can see the real world and a display that displays a virtual reality image; A detector for detecting a convergence angle between a gaze direction of a right eyeball of a user and a gaze direction of a left eyeball of the user; A processor.

[0007] The processor determining whether the user is viewing a first object at a first distance in the real world or whether the user is viewing a second object at a second distance different from the first distance in the real world based on the convergence angle detected by the detector; displaying a first augmented reality image on the display when determining that the user is looking at the first object; If it is determined that the user is looking at a second object, a request signal is sent to the electronic device, and a second virtual reality image transmitted from the electronic device in response to the request signal is displayed on the display. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a glasses-type eye rotation detection device according to an embodiment as viewed from the front. [Diagram 2] FIG. 2 is a diagram showing an example of a glasses-type eye rotation detection device as viewed from above and behind. [Diagram 3] 1 is a diagram showing a user wearing an example of a glasses-type eye rotation detection device, viewed from the right front. [Figure 4]2 is a block diagram showing an example of an electrical configuration of an eyeball rotation detection device. FIG. [Diagram 5] FIG. 13 is a diagram showing a first modified example of the arrangement of the neutral electrode 46. [Figure 6] FIG. 13 is a diagram showing a second modified example of the arrangement of the neutral electrode 46. [Figure 7] FIG. 13 is a diagram showing a third modified example of the arrangement of the neutral electrode 46. [Figure 8] FIG. 13 shows an EOG signal when the gaze is directed forward. [Figure 9] FIG. 13 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the left from a state in which the line of sight is directed forward. [Figure 10] FIG. 13 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the right from a state in which the gaze is directed forward. [Figure 11] 13 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction that increases the convergence angle from a state in which the gaze is directed forward. FIG. [Figure 12] 13 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction that reduces the convergence angle from a state in which the gaze is directed forward. FIG. [Figure 13] 1A to 1C are diagrams showing examples of waveforms of EOG signals for various eye movements. [Figure 14] 11A and 11B are diagrams illustrating an example of an experimental result for detecting a change in a convergence angle. [Figure 15] FIG. 11 is a front view of an example of a glasses-type eye rotation detection device according to a second embodiment. [Figure 16] 1 is a block diagram showing an example of the electrical configuration of a surgery support system including a glasses-type eye rotation detection device. [Figure 17] 11A and 11B are diagrams illustrating an example of the operation of the glasses-type eyeball rotation detection device. [Figure 18] 13A to 13C are diagrams illustrating another example of the operation of the glasses-type eye rotation detection device. [Figure 19] 13A to 13C are diagrams illustrating an example of the operation of a glasses-type eye rotation detection device according to a modified example of the second embodiment. [Figure 20]FIG. 11 is a diagram showing an example of the electrical configuration of a system including a glasses-type eye rotation detection device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described with reference to the drawings. Note that the disclosure is merely an example, and the invention is not limited to the contents described in the following embodiments. Modifications that a person skilled in the art can easily conceive are naturally included in the scope of the disclosure. In order to make the explanation clearer, the size, shape, etc. of each part may be changed from the actual embodiment and shown diagrammatically. In multiple drawings, corresponding elements may be given the same reference numerals, and detailed explanations may be omitted.

[0010] Here are some basic details about the eyeball. The diameter of an adult eyeball is about 25mm. After birth, it is around 17mm and grows larger as the child grows. The interpupillary distance of an adult male is about 65mm. For this reason, many commercially available stereo cameras are made with a 65mm gap between the pupils. The interpupillary distance of an adult female is several mm shorter than that of a male. The electrooculogram is several tens of mV. The eyeball has a positive electric potential on the cornea side and a negative electric potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV (called the electrooculogram).

[0011] The range of rotation of the eyeballs (for a typical adult) is 50° or less to the left and 50° or less to the right in the left-right direction (also called the horizontal direction), and 50° or less to the down direction and 30° or less to the up direction (also called the vertical direction). The range of angles that can be moved voluntarily in the up and down direction is narrow in the upward direction. This is because the "Bell phenomenon" occurs when the eyes are closed, causing the eyeballs to roll upward, so when the eyes are closed, the range of eye movement in the up and down direction shifts upward. The convergence angle (the angle at which the gaze directions of the left and right eyes intersect) is 20° or less.

[0012] [First embodiment] An example of the configuration of an eye rotation detection device according to an embodiment will be described with reference to Figs. 1, 2, and 3. There are various forms of eye rotation detection devices, but here, an example of an eye rotation detection device in the form of eyewear will be shown. Eyewear includes goggles and glasses (sunglasses are equivalent to glasses), but here, a glasses-type eye rotation detection device will be described. Fig. 1 is a front view of an example of a glasses-type eye rotation detection device, and Fig. 2 is a rear and upper view of an example of a glasses-type eye rotation detection device. Fig. 3 is a view of a user wearing an example of a glasses-type eye rotation detection device, viewed from the front right.

[0013] There are two types of eye rotation: up-down and left-right. Up-down rotation includes blinking, closing the eyes, winking, etc. Left-right rotation is a slow movement in which the left and right eyeballs unconsciously rotate in the same direction. Eye Movement is broadly divided into gaze movement in which the left and right eyeballs rotate in the same direction consciously, and convergence and divergence in which the left and right eyeballs rotate in opposite directions. Convergence is when the gaze directions of the left and right eyeballs cross, and divergence is when the gaze directions of the left and right eyeballs diverge. Eye rotation is detected based on changes in the electrooculogram. The electrooculogram can be detected by the difference in voltage from a pair of electrodes that sandwich the eyeball. The direction in which the eyeball is sandwiched can be left and right, up and down, front and back, or diagonally. Blinking, closing the eyes, winking, etc. can be detected from the electrooculogram detected by the electrode pairs that are arranged to sandwich the eyeball from above and below. Blinking, closing the eyes, winking, slow movement, and gaze movement can be detected from the electrooculogram detected by the electrode pairs that are arranged to sandwich the eyeball from above and below and from left and right. Slow movement, gaze movement, and convergence and divergence can be detected from the electrooculogram detected by the electrode pairs that are arranged to sandwich the eyeball from front and back and from left and right.

[0014] [Electrode arrangement] The glasses include a right frame 12, a left frame 14, and a bridge 26 connecting the frames 12 and 14. In this specification, right and left refer to the right and left as seen by a user wearing the glasses. In FIG. 1, which is viewed from the front, the frames are reversed, and the frame on the right side of FIG. 1 is the left frame 14. If the device is only for detecting electrooculography, lenses or glass need not be fitted into the right frame 12 and the left frame 14. However, if the user regularly wears glasses, lenses with a degree suited to the user may be fitted into the right frame 12 and the left frame 14 instead of the regularly used glasses. If the user does not regularly wear glasses, simple glass may be fitted into the right frame 12 and the left frame 14. If the device is not only for detecting electrooculography but is a product that detects line of sight movement or change in convergence angle from electrooculography and applies the detection results, for example, a glasses-type wearable device capable of AR display, a liquid crystal panel or an organic EL panel for AR display may be fitted into at least a part of the right frame 12 and the left frame 14.

[0015] In an embodiment, in order to detect convergence and divergence, electrodes are arranged on the same plane on either side of the eyeballs at anterior-posterior positions that are in phase (same vector) with respect to each of the left and right eyeballs, and at lateral positions that are in opposite phase (opposite vector) with respect to each of the left and right eyeballs.

[0016] 2, in order to detect the electrooculography of the right eyeball ER, a right temple electrode 32 is provided on the right side of the right eyeball ER, for example, on the part of the right temple 18 that hangs over the ear, and a right nose pad electrode 42 is provided on the left side of the right eyeball ER, for example, on the surface that contacts the nose of the right nose pad 22 attached near the connection point between the right frame 12 and the bridge 16. In a plan view (FIG. 2 is considered to be a plan view), the right temple electrode 32 and the right nose pad electrode 42 are arranged so that a line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER.

[0017] In a front view (FIG. 1 is considered as a front view), the right temple electrode 32 is provided on the left side of the right eyeball ER, and the right nose pad electrode 42 is provided on the right side of the right eyeball ER. The right temple electrode 32 and the right nose pad electrode 42 are arranged so that a line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER. Also, in the front view, the right nose pad electrode 42 is provided slightly above the right temple electrode 32.

[0018] In the side view, the right temple electrode 32 is provided behind the right eyeball ER, i.e., on the left side of the right eyeball ER in the right side view and on the right side of the right eyeball ER in the left side view, and the right nose pad electrode 42 is provided in front of the right eyeball ER, i.e., on the right side of the right eyeball ER in the right side view and on the left side of the right eyeball ER in the left side view. The right temple electrode 32 and the right nose pad electrode 42 are positioned so that a line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER.

[0019] 3 shows how the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in the front view, plan view, and side view of the head. Note that the line connecting the two electrodes does not have to pass through the center of the right eyeball ER, but may pass through any part of the eyeball. The same is true for the left eyeball, which is hidden by the face in FIG. 3.

[0020] The right temple electrode 32 and right nose pad electrode 42, which detect the electrooculogram of the right eye ER, are positioned so that the line connecting them passes through the right eye ER in any of the plan view, front view, and side view; however, it is sufficient that they are positioned so that the line connecting them passes through the right eye ER in at least one of the plan view, front view, and side view.

[0021] Similarly, to detect the electro-oculography of the left eyeball EL, a left nose pad electrode 44 is provided on the right side of the left eyeball EL in a plan view, for example, on the surface of the left nose pad 24 that contacts the nose, which is attached near the connection between the left frame 14 and the bridge 16, and a left temple electrode 36 is provided on the left side of the left eyeball EL, for example, on the part of the left temple 20 that hangs over the ear. The left nose pad electrode 44 and the left temple electrode 36 are positioned so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0022] The right temple electrode 32 and the left temple electrode 36 are symmetrical with respect to a line (for example, a line extending from the center of the nose to the back of the head) perpendicular to the midpoint of the line connecting the right frame 12 and the left frame 14.

[0023] In the front view, the left nose pad electrode 44 is provided to the left of the left eyeball EL, and the left temple electrode 36 is provided to the right of the left eyeball EL. The left nose pad electrode 44 and the left temple electrode 36 are arranged so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL. In addition, in the front view, the left nose pad electrode 44 is provided slightly above the second left electrode 42.

[0024] In the side view, the left nose pad electrode 44 is provided in front of the left eyeball EL, i.e., to the right of the left eyeball EL in the right side view and to the left of the left eyeball EL in the left side view, and the left temple electrode 36 is provided behind the left eyeball EL, i.e., to the left of the left eyeball EL in the right side view and to the right of the left eyeball EL in the left side view. The left nose pad electrode 44 and the left temple electrode 36 are arranged so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0025] The right temple electrode 32 is provided over the side (contacting the temporal region) and underside (contacting the base of the ear) of the right temple 18, and when the glasses are worn on the face, the weight of the temple 18 causes the right temple electrode 32 to come into contact with the area at the base of the ear where there is no coarse hair. The left temple electrode 36 is provided over the side (contacting the temporal region) and underside (contacting the base of the ear) of the left temple 20, and when the glasses are worn on the face, the weight of the temple 20 causes the left temple electrode 36 to come into contact with the area at the base of the ear where there is no coarse hair. This allows the right temple electrode 32 and the left temple electrode 36 to be in close contact with the user's skin, allowing the eye potential to be accurately sensed.

[0026] In addition, the left nose pad electrode 44 and the left temple electrode 36, which detect the electrooculography of the left eyeball EL, may also be positioned so that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL in at least one of the plan view, front view, and side view.

[0027] A forehead pad 26 is provided on the inside of the bridge 16 so as to contact the forehead, and a neutral electrode 46 is provided on the surface of the forehead pad 26 that contacts the forehead. The neutral electrode 46 is an electrode for ensuring a neutral potential for electrooculography detection, and is in contact with the skin, for example, the forehead. The neutral electrode 46 is disposed so that the distance between the neutral electrode 46 and the right temple electrode 32 is equal to the distance between the neutral electrode 46 and the left temple electrode 36, and the distance between the neutral electrode 46 and the right nose pad electrode 42 is equal to the distance between the neutral electrode 46 and the left nose pad electrode 44. The reason why the neutral electrode 46 is disposed in such a position is for the purpose of detecting the convergence angle, which will be described later. The convergence angle is detected based on the result of detecting symmetrical eyeball rotation when viewed from the front for each of the left and right eyeballs. For example, in an electrocardiograph, a neutral potential is taken at a body part where the effect of eyeball rotation can be ignored, for example, at the extremity of the right foot. Although it is somewhat affected by eye rotation, by taking a neutral potential at the center of the forehead, a location that is equally affected by both eyes, it is possible to equalize the influence of the eye potential that the neutral electrode receives from each of the left and right eyes.

[0028] The right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, left temple electrode 36, and neutral electrode 46 are made of metal foil such as copper, metal pieces, metal balls such as stainless steel, conductive silicone rubber sheets, etc. These electrodes 32, 42, 44, and 36 are electrodes for detecting the electrooculogram, as will be described later, and are therefore also referred to as EOG electrodes.

[0029] [EOG signal] As shown in FIG. 2, a processing unit 30 for detecting the electrooculogram is built in or externally attached to one of the temples, for example, the right temple 18, near the frame 12. A battery 34 for the processing unit 30 is built in or externally attached to the other temple, for example, the left temple 20, near the frame 12. The processing unit 30 may not only detect the electrooculogram, but also control the display in a glasses-type wearable device capable of AR display. The processing unit 30 may not be built in the glasses, but may be provided outside the glasses, and the glasses and the processing unit 30 may be connected wirelessly or by wire. In that case, the battery 34 may be built in the processing unit 30 and provided outside the glasses. Also, the function of the processing unit 30 may be divided into two, and only the first processing unit that senses the signal from the electrode may be provided in the glasses, and the second processing unit that detects the electrooculogram from the sensed signal and controls according to the detection result may be provided outside the glasses. A mobile terminal such as a smartphone may be used as the second processing unit. The second processing unit is not limited to a mobile terminal directly connected to the glasses, but may also include a server connected via a network.

[0030] The signal from the right temple electrode 32 is input to the negative terminal of a first analog / digital (A / D) converter 62, and the signal from the second left electrode 36 is input to the positive terminal of the first A / D converter 62, which outputs a first EOG signal ADC Ch0, which is a difference signal. Since the right temple electrode 32 and the left temple electrode 36 sandwich the eyeball from the left and right, the first EOG signal ADC Ch0 indicates the left and right rotation of the left and right eyeballs.

[0031] The signal from the right temple electrode 32 is input to the - terminal of the second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, and a second EOG signal ADC Ch1, which is a difference signal, is output. Since the right temple electrode 32 and the right nose pad electrode 42 sandwich the right eyeball from above and below and from left and right, the second EOG signal ADC Ch1 indicates the left and right rotation and the up and down rotation of the right eyeball.

[0032] The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, and the signal from the left temple electrode 36 is input to the - terminal of the second A / D converter 66, and a third EOG signal ADC Ch2, which is a difference signal, is output. Since the left nose pad electrode 44 and the left temple electrode 36 sandwich the left eyeball from above and below and from left and right, the third EOG signal ADC Ch2 indicates the left and right rotation and the up and down rotation of the left eyeball.

[0033] Since the left and right positions of the two electrodes related to the second EOG signal ADC Ch1 and the left and right positions of the two electrodes related to the third EOG signal ADC Ch2 are opposite (the + / - of the A / D converter input is inverted), it is possible to detect from the waveforms of the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 whether the left and right eyeballs are rotating in the same direction or in opposite directions left and right.

[0034] The voltage signals from the right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, and left temple electrode 36 are weak, and are therefore greatly affected by noise. To cancel this noise, a series circuit of resistors R1 and R2 is connected between the reference analog voltage Vcc (=3.3V or 5.5V) of the A / D converters 62, 64, and 66 and ground (GND), and the neutral electrode 46 is connected to the connection point of the resistors R1 and R2. The resistors R1 and R2 have the same value, for example, 1MΩ. The A / D converters 62, 64, and 66 can detect analog voltages from 0V (ground) to the reference analog voltage Vcc, and convert the input analog voltage into a digital value in the range from 0V to 3.3V, centered on the midpoint of the detectable range, for example, a voltage that is 1 / 2 of 3.3V (referred to as the midpoint voltage). Since the connection point of resistors R1 and R2 is connected to the midpoint voltage terminal and the neutral electrode 46 is connected to the connection point of resistors R1 and R2, the midpoint voltage of the A / D converters 62, 64, 66 becomes the same as the voltage of the human body. As a result, the midpoint voltage of the A / D converters 62, 64, 66 fluctuates in conjunction with the voltage of the human body, and noise mixed in the voltage signals from the EOG electrodes 32, 42, 44, 36 does not mix with the digital values ​​output from the A / D converters 62, 64, 66. This improves the S / N ratio of the electrooculography detection.

[0035] 4 is a block diagram showing an example of the electrical configuration of the eye rotation detection device. The processing unit 30 may include A / D converters 62, 64, and 66, or the A / D converters 62, 64, and 66 may be external to the processing unit 30.

[0036] The signal from the right temple electrode 32 is input to the negative terminal of the first A / D converter 62, and the signal from the left temple electrode 36 is input to the positive terminal of the first A / D converter 62, to obtain a first channel EOG signal ADC Ch0. The signal from the right temple electrode 32 is input to the negative terminal of the second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the positive terminal of the second A / D converter 64, to obtain a second channel EOG signal ADC Ch1. The signal from the left nose pad electrode 44 is input to the positive terminal of the third A / D converter 66, and the signal from the left temple electrode 36 is input to the negative terminal of the second A / D converter 66, to obtain a third channel EOG signal ADC Ch2.

[0037] The signal from the neutral electrode 46 is supplied to the midpoint voltage terminal of the A / D converters 62 , 64 , 66 , and the midpoint voltage of the A / D converters 62 , 64 , 66 is taken as the voltage of the human body detected by the neutral electrode 46 .

[0038] The EOG signals output from the A / D converters 62, 64, and 66 are input to an eye movement detection unit 75 that detects eye rotation (hereinafter, also referred to as eye movement). The eye movement detection unit 75 may be configured from hardware or software. In the latter case, the CPU 74, the ROM 76, and the RAM 78 are connected to a bus line, and the eye movement detection unit 75 is also connected to the bus line. The eye movement detection unit 75 is realized by the CPU 74 executing a program stored in the ROM 76. A wireless LAN device 80 is also connected to the bus line, and the processing unit 30 is connected to a mobile terminal 84 such as a smartphone via the wireless LAN device 80. The mobile terminal 84 may be connected to a server 88 via a network 86 such as the Internet. The eye movement detection unit 75 detects the electrooculogram based on the EOG signals output from the A / D converters 62 and 64, and can detect the left and right rotation (convergence and divergence) of each of the left and right eyeballs, the left and right rotation (gaze movement), and the up and down rotation (blinking and closing of the eyes) from the detected electrooculogram. Furthermore, the eye movement detection unit 75 can estimate various states of the user from the detected eye movement (for example, a state of lack of concentration and restlessness, a state of concentration, a state of tension and mental stress, a state of fatigue and difficulty concentrating on work or tasks). The type of eye rotation to be detected and the type of state to be estimated can be changed by changing the program executed by the CPU 74. This change instruction may be given from the mobile terminal 84.

[0039] Instead of the wireless LAN device 80, ZigBee (registered trademark), Bluetooth A communication device using a communication method such as Low Energy (registered trademark) or Wi-Fi (registered trademark) may be used. The detection result of the eye movement detection unit 75 (eye movement detection result, state estimation result) may be temporarily stored in the RAM 78 and then sent to the mobile terminal 84 via a communication device such as a wireless LAN device 80. Alternatively, the detection result of the eye movement detection unit 75 may be sent to the mobile terminal 84 in real time. The mobile terminal 84 may store the detection result of the eye movement detection unit 75 in a built-in memory (not shown) or may transfer the detection result to the server 88 via the network 86. The mobile terminal 84 may start some processing depending on the detection result of the eye movement detection unit 75, may store the processing result in the built-in memory, or may transfer the processing result to the server 88 via the network 86. The server 88 may aggregate the detection results from many eye movement detection units 75 and the processing results of many mobile terminals 84 to perform so-called big data analysis.

[0040] [Variations of electrode arrangement] 5, 6, and 7 show modified arrangements of the neutral electrode 46. In the above description, separate left and right nose pads 22, 24 are provided, but in the modified example in Fig. 5, an integrated left and right inverted V- or U-shaped nose pad 52 is provided. A right nose pad electrode 42 is provided on the right inside of both open sides of the nose pad 52, a left nose pad electrode 44 is provided on the left inside, and a neutral electrode 46 is provided on the inside of the apex of the V or U shape. This makes it possible to provide the neutral electrode 46 in contact with the forehead without providing a forehead pad 26.

[0041] 6 also has an integrated V- or U-shaped nose pad 54. The nose pad 52 spreads downward, but the nose pad 54 differs in that it spreads forward. The right nose pad electrode 42 is provided on the right side of the nose pad 54, the left nose pad electrode 44 is provided on the left side, and the neutral electrode 46 is provided in the center.

[0042] In the modified examples of Figures 5 and 6, nose pads with a larger area than normal nose pads are used, so even if the glasses-type electro-oculography detection device or glasses-type wearable terminal glasses capable of AR display, which are heavier than normal glasses, are used for a long period of time, the weight is less likely to cause pain in the nose.

[0043] 7, separate left and right nose pads 22, 24 are used, but forehead pad 26 is not necessary. Here, right nose pad electrode 42 and right neutral electrode 46a are provided on the surface of right nose pad 22 that contacts the nose, and left nose pad electrode 44 and left neutral electrode 46b are provided on the surface of left nose pad 24 that contacts the nose. Right neutral electrode 46a and left neutral electrode 46b are electrically shorted and are equivalent to one neutral electrode 46.

[0044] [Relationship between eye movement and EOG signals] Referring to Figures 8 to 12, an example of changes in the waveforms of the EOG signal ADC Ch0 output from A / D converter 62, the EOG signal ADC Ch1 output from A / D converter 64, and the EOG signal ADC Ch2 output from A / D converter 66 are shown when the right and left eyeballs are rotated left and right from a state in which the line of sight is facing forward.

[0045] FIG. 8 shows a state in which the user's line of sight is in the forward direction. When looking at infinity, the line of sight of the right eyeball and the line of sight of the left eyeball are parallel, but when looking at a finite distant point, the line of sight of the right eyeball and the line of sight of the left eyeball intersect at the distant point. When both the right eyeball ER and the left eyeball EL rotate leftward from this state as shown in FIG. 9 (the line of sight of the right eyeball and the line of sight of the left eyeball move leftward), the positively charged cornea of ​​the right eyeball ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When both the left and right eyeballs rotate rightward in this state, the state of FIG. 8 is restored. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32, 36 are connected is a signal with a convex waveform (a waveform that is convex upwards). The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and right temple electrode 32 are connected is a signal with a convex waveform (a waveform that is convex upwards). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and left temple electrode 36 are connected is a signal with a concave waveform (a waveform that is convex downwards).

[0046] In this way, the left and right eyeballs rotate in the same direction (left rotation), so the second EOG signal ADC An EOG signal of opposite phase appears in Ch1 and the third EOG signal ADC Ch2. An EOG signal of the same phase as the second EOG signal ADC Ch1, which has the same + / - relationship, appears in the first EOG signal ADC Ch0.

[0047] When the right eyeball ER and the left eyeball EL both rotate to the right as shown in FIG. 10 (the gaze direction of the right eyeball and the gaze direction of the left eyeball move to the right) from the state in which the gaze direction is forward as shown in FIG. 8, the positively charged cornea of ​​the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When both the left and right eyes rotate to the left in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected becomes a concave waveform (a downward convex waveform). The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected is a concave waveform (a waveform that is convex downwards). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected is a convex waveform (a waveform that is convex upwards).

[0048] In this way, the left and right eyeballs rotate in the same direction (right rotation), so the second EOG signal ADC An EOG signal of opposite phase appears in Ch1 and the third EOG signal ADC Ch2. However, they are opposite phases when the right eyeball ER and the left eyeball EL both rotate to the left. An EOG signal of the same phase as the second EOG signal ADC Ch1, which has the same + / - relationship, appears in the first EOG signal ADC Ch0. However, the first EOG signal ADC Ch0 when the right eyeball ER and the left eyeball EL both rotate to the right is opposite phase to the first EOG signal ADC Ch0 when the right eyeball ER and the left eyeball EL both rotate to the left.

[0049] When the right eyeball ER rotates left (the gaze direction of the right eyeball moves left) and the left eyeball EL rotates right (the gaze direction of the left eyeball moves right) as shown in FIG. 11 from the state in which the gaze direction is forward as shown in FIG. 8, and convergence occurs in which the gaze directions of the left and right eyeballs cross, so-called cross-eyed, the positively charged cornea of ​​the right eyeball ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When the right eyeball rotates right and the left eyeball rotates right in this state, the state returns to that of FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected does not change, and neither a convex waveform nor a concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected is a signal with a convex waveform (a waveform that is convex upward). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected is a signal with a convex waveform (a waveform that is convex upward).

[0050] Since the left and right eyeballs rotate in opposite directions in this manner, waveforms of the same phase appear in the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2.

[0051] When the electrooculograms of the left and right eyeballs are the same and the absolute values ​​of the rotation angles are also the same, both the positive and negative terminals of the A / D converter 62 change by the same amount in the same direction (negative direction), so there is no change in the relative value between the two and no change in the electrooculogram appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrodes and temple electrodes is slightly shifted from the center of the left and right eyeballs, so a slight change appears according to the amount of this shift.

[0052] When the right eyeball ER rotates right (the gaze direction of the right eyeball moves to the right) and the left eyeball EL rotates left (the gaze direction of the left eyeball moves to the left) as shown in FIG. 12 from the state in which the gaze direction is forward as shown in FIG. 8, i.e., the gaze directions of the left and right eyeballs diverge, i.e., the so-called wide-set eyes occur, the positively charged cornea of ​​the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When the right eyeball rotates left in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected does not change, and neither a convex waveform nor a concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected is a concave waveform (a waveform that is convex downwards). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected is a concave waveform (a waveform that is convex downwards). As the left and right eyeballs rotate in opposite directions in this way, the second EOG signal ADC In-phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. However, the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 in the case of wide-eyed eyes are out of phase with respect to the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 in the case of cross-eyed eyes.

[0053] When the electrooculograms of the left and right eyeballs are the same and the absolute values ​​of the rotation angles are also the same, both the positive and negative terminals of the A / D converter 62 change by the same amount in the same direction (positive direction), so there is no change in the relative value between the two and no change in the electrooculogram appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrodes and temple electrodes is slightly shifted from the center of the left and right eyeballs, so a slight change appears according to the amount of this shift.

[0054] 13 is an electrooculogram (EOG) for explaining an example of the relationship between various eye movements of a user and the EOG signals ADC Ch0, ADC Ch1, and ADC Ch2 obtained from the A / D converters 62, 64, and 66. The vertical axis indicates the sample values ​​of the A / D converters 62, 64, and 66 (e.g., 3.3V, 24-bit A / D converters), and the horizontal axis indicates time.

[0055] As shown in Fig. 11, neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and convex waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, so that the eye movement detection unit 75 detects a "cross-eyed" state in which the left and right eyeballs converge in the direction of their gaze. Although not shown in Fig. 13, as shown in Fig. 12, neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and concave waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, so that the eye movement detection unit 75 detects a "wide-eyed" state in which the left and right eyeballs diverge in the direction of their gaze. As described above, the convergence and divergence are different in the concave and convex waveforms of the EOG signals ADC Ch1 and ADC Ch2, but the rest is the same, so in the following description, the convergence and divergence may be collectively referred to as "convergence". The amplitude of the waveforms of the EOG signals ADC Ch1 and ADC Ch2 corresponds to the degree of convergence (convergence angle) and the degree of divergence. 14, the degree of change in amplitude is greater at closer distances and becomes smaller at longer distances, so the sensitivity of detecting amplitude change is higher at closer distances and becomes lower at longer distances.

[0056] As shown in FIG. 9, a convex waveform appears in the EOG signal ADC Ch0, a convex waveform appears in the EOG signal ADC Ch1, and a concave waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 detects a leftward movement in the gaze direction.

[0057] As shown in FIG. 10, a concave waveform appears in the EOG signal ADC Ch0, a concave waveform appears in the EOG signal ADC Ch1, and a convex waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 detects a rightward movement in the gaze direction.

[0058] In the EOG signal ADC Ch1 and the EOG signal ADC Ch2, one blink 1, two blinks 2, and three blinks 3 are detected by the convex pulse waveforms of 1 to 3 waves in phase, whose level rises momentarily and returns to the original level. In the EOG signal ADC Ch1 and the EOG signal ADC Ch2, the eye movement detection unit 75 detects the vertical eyeball rotation, i.e., closing of the eyes, by a combined waveform of a convex waveform (upward convex waveform) when the gaze is directed upward and a concave waveform (downward convex waveform) when the gaze is directed downward. In this way, the eyeball rotation caused by the vertical blink and closing of the eyes can be detected based on either the EOG signal ADC Ch1 or ADC Ch2, and in the application of detecting only blinks and closing of the eyes, it is not necessary to provide an electrode pair for each of the left and right eyes, and an electrode pair may be provided for only one of the eyes.

[0059] An example of an experiment result for detecting a change in a convergence state according to an embodiment is shown. FIG. 14 shows an example of a change in the second EOG signal ADC Ch1 when a subject changes the intersection position of the gaze direction of the left and right eyeballs from a state in which the subject looks at a fingertip 10 cm in front of the nose to a marker further away, using the prototype of the electrooculography detection device shown in FIG. 1 to FIG. 4. In this case, the intersection point of the gaze direction changes from near to far, so the convergence angle decreases. The horizontal axis is the moving distance of the intersection point of the gaze direction from 10 cm away to the depth direction, and the first plot shows the EOG amplitude when the intersection point of the gaze direction is moved from 10 cm away to 20 cm away. Note that 10 cm is the shortest distance at which stable gaze can be maintained. The minimum detection voltage of the EOG signal is set to 50 μV. That is, when the EOG signal changes by 50 μV or more, the eye movement detection unit 75 can detect the change in the EOG signal, and based on that, can detect the change in the intersection position of the gaze direction, that is, the change in the convergence angle, and cannot detect the change in the EOG signal unless the EOG signal changes by 50 μV or more. If the average value obtained by integrating the measured values ​​is used, the minimum detection voltage of the EOG signal will be smaller, but here it is set to 50 μV. Note that the amplitude of the EOG signal depends on the contact resistance of the electrodes, and if an electrode material with low contact resistance is used, the amplitude of the EOG signal will be larger, and the minimum detection voltage of the EOG signal will be larger.

[0060] For example, when the convergence angle is reduced from a state where the subject is looking at a marker 30 cm ahead of the nose to a state where the subject is looking at a marker further away, the eye movement detection unit 75 detects a change in the EOG amplitude when the EOG amplitude increases by 50 μV. The EOG amplitude obtained by adding 50 μV to the EOG amplitude when the subject is looking at a marker 30 cm ahead corresponds to 40 cm ahead. In other words, when the convergence angle is reduced from a state where the subject is looking at a marker 30 cm ahead to a state where the subject is looking at a marker 40 cm ahead, the eye movement detection unit 75 can detect a change in the EOG amplitude. The fact that a change in the EOG amplitude corresponding to a decrease in the convergence angle corresponding to a change of 10 cm can be detected means that the detection resolution is quite good. From FIG. 14, it can be seen that a change in the EOG amplitude can be detected when the convergence angle is reduced from a state where the subject is looking at a marker 50 cm ahead to a state where the subject is looking at a marker 65 cm ahead or further away, and a change in the EOG amplitude can be detected when the convergence angle is reduced from a state where the subject is looking at a marker 70 cm ahead to a state where the subject is looking at a marker 1.4 m ahead or further away.

[0061] As described above, according to the first embodiment, a glasses-type eye rotation position detection device is provided that includes right and left temple electrodes, right and left nose pad electrodes, and a neutral electrode arranged so as to be equally affected by the rotation of the right and left eyeballs, and detects the left and right rotation of each of the right and left eyeballs independently, thereby detecting convergence. Since the neutral potential from the neutral electrode is set as the midpoint potential of the A / D converter that samples the EOG signal from the electrode, the EOG signal from the electrode is not affected by noise, and the eye potential is accurately detected, so that accurate eye rotation is detected. According to the first embodiment, the left and right rotation of both eyes (left and right movement in the gaze direction) and the up and down rotation of the eyeballs can also be detected.

[0062] According to the electro-oculography detection device of the first embodiment, it is possible to detect convergence, which is a conscious eye rotation of a subject, and thus to realize an application example in which control according to the subject's intention is performed by performing control according to the detection result. For example, in eyewear capable of displaying Augmented Reality (hereinafter referred to as AR), it is possible to control the AR display on / off and the adjustment of the display position of an AR image in a hands-free manner according to the detection of convergence.

[0063] In addition, certain tasks may require task-specific convergence changes, and by comparing the user's pattern of convergence changes with a reference pattern, it is possible to determine the user's level of proficiency with the task and whether or not the task is being performed correctly.

[0064] Hereinafter, application examples of the convergence detection result will be described. One of the application examples is to incorporate an electrooculography detection device into a glasses-type wearable device capable of AR display, and to control the AR display based on the detection of the convergence. For example, the detection of the convergence can be applied as a function changeover switch. For example, as shown in FIG. 8, when the user is looking at a distant object in front of the user, the AR display is turned off, and when the line of sight of both eyes is changed to look at a close distance (convergence state) as shown in FIG. 11, the display can be controlled to be turned on and off so that the AR display is performed. Since the display position of the AR image is set to a close distance, the AR display is continued because the user is looking at the AR image in a convergence state, and when the user changes the line of sight of both eyes to look at a distant object, the AR display is turned off. This makes it possible to control the AR display as intended by the user. Hereinafter, as a second embodiment, a glasses-type wearable device for surgery support, which is an application example of this, will be described.

[0065] [Second embodiment] FIG. 15 is a front view of an example of the glasses-type electro-oculography detecting device 100 according to the second embodiment. The difference from the detecting device of the first embodiment is that displays (e.g., organic EL panels or liquid crystal panels) 102, 104 for AR display are embedded in at least a part of the right frame 12 and the left frame 14. When the AR image is not displayed in 3D, the display 102 or 104 does not have to be embedded in one of the right frame 12 and the left frame 14. Here, the AR image is displayed in 3D. The arrangement of the EOG electrodes is the same as in the first embodiment. There are various application examples of the glasses-type electro-oculography detecting device 100 capable of AR display, but the second embodiment will be described taking a surgery support system as an example. The doctor performing the surgery and staff involved in the surgery wear the glasses-type electro-oculography detecting device 100.

[0066] Fig. 16 is a block diagram showing an example of the electrical configuration of a surgery support system including the glasses-type electro-oculography detecting device 100. The processing unit 30 further includes a display controller 112 in addition to the configuration of the first embodiment shown in Fig. 4. The display controller 112 controls the AR display on the displays 102 and 104. The control of the AR display includes on / off control of the AR display, control of the display position of the AR image (convergence angle control of the AR image), etc.

[0067] In a surgery support system, multiple doctors and staff members are involved in the same surgery and multiple pairs of glasses are used, so it is preferable that the control device 120, such as a personal computer with higher performance than the mobile terminal 82 such as a smartphone, be connected to the glasses processing device 30. Although not shown, multiple glasses processing devices 30 are connected to the control device 120. The control device 120 includes a vital data memory 124, an support image memory 122, and a support information memory 126. The eye movement detection unit 75 may also be provided in the control device 120.

[0068] A vital data measuring unit 127 is connected to the control device 120, and measures the patient's electrocardiogram, blood pressure, pulse rate, cumulative amount of blood transfusion, etc., and stores the data for each patient in a vital data memory 124 in the control device 120. A support image memory 122 stores support images for supporting surgery. If necessary, support images in a surgery support database 130 in the server 88 are downloaded to the control device 120 and stored in the support image memory 122. A support information memory 126 stores support text for supporting surgery. If necessary, support text in a surgery support database 130 in the server 88 is downloaded to the control device 120 and stored in the support information memory 126.

[0069] An example of the operation of the glasses-type electro-oculography detection device 100 will be described with reference to Figs. 17 and 18. Here, it is assumed that the distance from the eyeball to the affected part (real world) during surgery is about 40 cm. When the doctor is looking at a point in front of the affected part (real world) during surgery, for example, a point about 30 cm away from the eyeball, the eye movement detection unit 75 detects convergence. As shown in Fig. 11, the eye movement detection unit 75 detects convergence based on the fact that the waveform of the EOG signal ADC Ch0 does not change and the waveforms of the EOG signal ADC Ch1 and the EOG signal ADC Ch2 become upwardly convex waveforms. When the eye movement detection unit 75 detects convergence, it requests an AR image related to surgical support from the control device 120 and causes the display controller 112 to perform AR display. An example of AR display is to superimpose a vital data window (AR image) semi-transparently on the affected part (real world) during surgery, as shown in Fig. 17(a). The window includes multiple pages, and each page displays an electrocardiogram, blood pressure, pulse, total amount of transfusion, etc. The window is restricted to a partial area of ​​the screen so that the affected area is not hidden. In response to a request from the processing unit 30, the control device 120 reads out the vital data of the patient from the vital data memory 124 and transfers it to the processing unit 30.

[0070] When an AR image is displayed in 3D, the display position can be set arbitrarily by adjusting the convergence angle of the left and right images. Here, the display position of the vital data window is set to a position about 30 cm forward, which is equal to the distance at which the eye movement detection unit 75 detects convergence. Therefore, when the vital data window is displayed, the doctor is looking at a point at the same distance as the display position, so that the doctor can instantly check the contents of the window, and there is no need to rotate the eyes to adjust the convergence angle in order to gaze at the window, and eye strain does not occur.

[0071] The page switching of the vital data window may be performed automatically at a fixed time interval, for example, every second, or may be performed at the user's will based on the rotation of the eyeball in another direction detected by the eye movement detection unit 75. For example, when closing of the eyes for 0.5 seconds or more is detected, the page of the window may be switched. Furthermore, the page of the window may be switched based on the direction of movement of the gaze. For example, when the gaze moves to the right, the page may be switched to the next page, and when the gaze moves to the left, the page may be switched to the previous page. FIG. 17(b) shows an example of switching of the page of the window.

[0072] 17(a) or 17(b), when the doctor or the like changes his / her line of sight to view the affected area (real world) (more than about 40 cm away) during surgery, the eye movement detection unit 75 detects that the convergence angle becomes smaller (the distance to the intersection of the line of sight of both eyes becomes longer) and stops the AR display in the display controller 112. Therefore, the doctor or the like observes only the affected area during surgery as shown in FIG. 17(c) through the right frame 12 and left frame 14 of the glasses.

[0073] When a doctor or the like wishes to refer to vital data in the state shown in Fig. 17(c), the doctor or the like changes the direction of the line of sight to look at something close to them (approximately 30 cm). The eye movement detection unit 75 detects that the convergence angle is increasing (the distance to the intersection of the line of sight of both eyes is getting closer) and causes the display controller 112 to perform AR display, and a vital data window such as that shown in Fig. 17(a) or (b) is displayed.

[0074] The increase / decrease in the convergence angle can be determined based on the relationship between the EOG amplitude and the movement distance of the intersection of the lines of sight as shown in Figure 14. If the EOG voltage decreases / increases by a certain voltage or more, it can be determined that the convergence angle has increased / decreased.

[0075] In this way, doctors can turn on and off the AR images that support surgery hands-free. Since the hands are occupied during surgery, being able to turn the AR display on and off hands-free is very effective.

[0076] Another example of the AR display may be a support image as shown in FIG. 18(a). The support image may be an image of a previous surgery performed on another patient with similar symptoms, or an image from another surgery performed on the patient. The image may be a still image or a video. For this reason, images during the surgery are taken by a camera (not shown), and the taken images are uploaded to the server 88 and stored in the surgery support database 130. Another example of the AR display may be support information as shown in FIG. 18(b). The support information is various text data related to the surgery being performed.

[0077] The display position of the support information is set close to the patient, like the vital data window. However, the display position of the support image may be set at a distance of about 40 cm, the same as the affected area during surgery (real world). The vital data window and support information are not viewed at the same time as the affected area during surgery, but it is assumed that the support image is compared side by side with the affected area. When comparing the affected area and the support image, if the display positions are different, the patient will need to rotate the eyes left and right when comparing them, which may cause eye strain. Therefore, unlike the above description, the support image is displayed when the eye is in a state as shown in FIG. 8, and is turned off when convergence as shown in FIG. 11 is detected. Note that if the position of the support image and the affected area during surgery (real world) are slightly misaligned, the convergence angle detected by the eye movement detection unit 75 will change slightly when comparing them. If the display controller 112 adjusts the display position of the support image (convergence angle of the left and right images) so that this change is reduced, the possibility of eye strain can be further reduced.

[0078] Switching from the vital data window to the support image and support information may be performed at the user's discretion based on the rotation of the eyeball in another direction detected by the eye movement detection unit 75, in the same manner as page switching of the vital data window.

[0079] In the above example, the change in the convergence angle is used to switch the AR display on / off, but the detected convergence angle may be used for another control, and the AR display may be switched on / off based on another eye movement. For example, when multiple eye closures are detected, the AR display may be switched on / off. The detected convergence angle may be used to control the display position of the AR image. That is, the distance between the left and right images of the AR image (sometimes referred to as the convergence angle of the images) is adjusted based on the distance to the intersection of the line of sight of the left and right eyeballs at the start of displaying the AR image. The convergence angle of the images is adjusted by the display controller 112. As a result, there is no need to rotate the eyes left and right to gaze at the AR display, so that eye strain does not occur when viewing the AR display.

[0080] Another application example of the change in the convergence angle and the control of the AR display will be described with reference to Fig. 19. This example is an example of picking in a warehouse, and a worker wears the glasses-type electro-oculography detecting device 100 during the picking work.

[0081] The configuration of the processing unit 30 may be the same as that shown in FIG. 16, and is therefore omitted from the illustration. The configuration of the control device 120 is the same except that the vital data, support image, and support information are changed to a pick-up list, and is therefore omitted from the illustration. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used as in the first embodiment shown in FIG. 4. The mobile terminal 84 may acquire location information. The configuration of the server 88 is also the same except that the surgical support data is changed to map information on the positions of shelves in the warehouse, a list of products on each shelf in the warehouse, and a list of products to be picked up, and is therefore omitted from the illustration. The pick-up list for each worker is downloaded from the server 88 to the control device 120.

[0082] In this application example as well, initially, the AR display is off and the displays of the right frame 12 and left frame 14 of the glasses are transparent and show nothing. For this reason, the worker gazes at the inside of the warehouse as shown in FIG. 19(a) through the right frame 12 and left frame 14 of the glasses. If the mobile terminal 84 serving as the control device 120 is capable of acquiring location information, air tags may be displayed in AR on products on each shelf based on the map information in the server 88 and the acquired location information. At this time, the worker is looking several meters or even tens of meters or more ahead.

[0083] When the worker changes from looking at a distant target to looking at a closer target as shown in FIG. 11, the left and right eyeballs rotate in opposite directions to the left and right, causing convergence, i.e., the eyes become cross-eyed, and the eye movement detection unit 75 detects the convergence.

[0084] In response to the detection of the congestion, the processing unit 30 requests the control device 120 for a pickup list showing the products to be picked up by the worker, and causes the display controller 112 to start AR display. An example of the pickup list is shown in FIG. 19(b). The display position of the pickup list is also set to nearby. When the left and right eyeballs are rotated from a state in which the worker is looking at the pickup list to a state in which the worker is looking at the distant warehouse as shown in FIG. 8, the eye movement detection unit 75 no longer detects the congestion, and causes the display controller 112 to stop displaying the pickup list.

[0085] Pickup work, like surgery, requires constant use of your hands, so being able to turn the AR display on and off hands-free is extremely effective.

[0086] In FIG. 19, the air tag is always displayed, but it may be displayed only when the user is looking at a distant target, and turned off when the user is looking at a nearby target. For example, when a worker goes out to perform maintenance work on an elevator in a building and is looking for the target building during the visit, the worker is looking at a distant target, so an air tag including the building's name, distance, work purpose, etc. is displayed superimposed on the target building. Here, when the worker looks at his / her hand to check a map, etc., the air tag display may be turned off. When the worker arrives at the destination and comes in front of the elevator, an air tag is displayed at the inspection point when the worker is looking at the area to be inspected. When the worker looks at his / her hand, a work manual, etc. is displayed in AR instead of the air tag. This example can also be applied to tourist guides. For example, when a tourist is looking at a landscape, an air tag related to that spot may be displayed, and when the worker looks at his / her hand, the air tag may be turned off and a detailed tourist guide may be displayed instead.

[0087] Another example of AR display based on convergence detection is to incorporate an electrooculography detection device into AR glasses used for watching sports. The game is filmed from various angles and stored as replay footage on a server. The AR glasses are connected to the server via a network. Spectators are always watching the game and looking at distant targets. At this time, the AR display is turned off. When a spectator in the outfield seats at a baseball stadium wants to see an enlarged image of a cross play on home base, he or she can rotate their eyes to cross their eyes and start the AR display. In the AR display, various replay images are downloaded from the server and displayed. This allows spectators to instantly enjoy the replay images by rotating their eyes.

[0088] In the above description, the AR display is started when congestion is detected, and stopped when congestion is no longer detected, but the reverse is also possible, that is, the AR display may be performed when congestion is not detected, and stopped when congestion is detected.

[0089] [Third embodiment] Another application example of convergence detection is confirmation of work procedures. FIG. 20 is a block diagram showing an example of the electrical configuration of a work confirmation system including a glasses-type electrooculography detection device. Although AR display is not essential for work confirmation, AR display may be used to immediately inform the worker of the confirmation result. The processing unit 30 is the same as that of the second embodiment shown in FIG. 16. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used as in the first embodiment shown in FIG. 4. The control device 120 includes a gaze movement discrimination unit 148 and a gaze movement pattern memory 142. The gaze movement discrimination unit 148 discriminates gaze movement (right gaze movement, left gaze movement, gaze crossing (convergence)) from the change pattern of eye rotation detected by the eye movement detection unit 75, and stores the gaze movement pattern in the gaze movement pattern memory 142. The data in the gaze movement pattern memory 142 is uploaded to a server 88 via a network 86.

[0090] The server 88 includes a standard line-of-sight movement pattern memory 144 and a work procedure discrimination unit 146. Some work requires a line-of-sight movement specific to the work. For example, in the railway industry, when pointing and checking after departure, it is required to check distant and nearby targets in a specified order. Also, in visual inspection of structures such as bridges and tunnels, the places to be visually checked are fixed, so it is required to move the line of sight in a specified pattern. The standard pattern memory 144 thus stores a standard pattern of line-of-sight movement specific to the work. The work procedure discrimination unit 146 can compare the line-of-sight movement pattern of the worker uploaded from the control device 120 with the standard line-of-sight movement pattern in the standard pattern memory 144 and judge whether the worker is performing the work correctly. It may be stored for each worker in a database not shown. The proficiency of the worker can be estimated based on the change over time in the judgment result. Alternatively, if the work procedure discrimination unit 146 determines that the worker is not performing the work correctly, it may notify the display controller 112 of the processing unit 30 via the control device 120, and cause a warning message to be displayed on the displays 102, 104.

[0091] Another example of standard eye movement patterns is a pattern related to driving a car. In driving lessons for license renewal, a video of a good driver performing safety checks may be introduced. In this case, rather than simply watching, the eye movement patterns when actually performing the same safety checks may be compared with the eye movement patterns of good drivers to determine the proficiency of safety checks. In the transportation industry, accidents can also be reduced by new drivers learning the same eye movement patterns as veteran drivers.

[0092] The above description is not limited to glasses, but can also be applied to goggles. For example, electrodes may be provided on a part of the foam on the front of the goggles instead of the nose pads, and electrodes may be provided on the belt instead of the temples.

[0093] [calibration] In the electrooculogram, the specific angle of the convergence angle is not obtained, but only the rotation direction of the eyeballs in the convergence state is obtained. Therefore, in the above embodiment, the convergence is detected, and in the application example, control is performed based on the binary state of whether or not the convergence is detected, regardless of the convergence angle itself. However, when there are multiple targets with known distances, the multiple targets are viewed and the electrooculogram is periodically measured for multiple distances, thereby obtaining the value of the electrooculogram for the multiple known distances, and three or more convergence states can be identified. For example, when a distant target is viewed, the AR display may be turned off, and when a medium-distance target is viewed, a first AR image may be displayed, and when a close-distance target is viewed, a second AR image may be displayed.

[0094] In addition, since the contact resistance of the electrodes changes (decreases over time), the absolute value of the electrooculogram with respect to the distance changes over time, and the absolute value of the distance cannot be guaranteed in the long term. For example, immediately after starting to use the electrodes, the contact resistance is large, so the amplitude of the electrooculogram is small, and as time passes, the resistance decreases and the amplitude of the electrooculogram increases. However, if the electrooculogram is calibrated periodically with respect to the distance, the change in the electrooculogram with time can be estimated and the absolute value of the electrooculogram with respect to the distance can be guaranteed. For example, in the case of work with a constant focal length such as surgery or desk work, or in the case of work in which the work process guarantees that a target at a known distance (level) is being viewed at a certain point in time, it is possible to perform calibration periodically with the distance known.

[0095] Furthermore, the body potential itself may fluctuate, but this can also be compensated for by periodic calibration.

[0096] [Summary of Effects of the Embodiments] According to the above-mentioned embodiment, the left and right rotation of the right eyeball and the left and right rotation of the left eyeball can be detected independently, so that convergence, which is the intersection of the gaze directions of the left and right eyeballs, can be detected. Convergence does not occur unconsciously, but occurs due to the subject's conscious eyeball rotation. Therefore, by performing control according to the detection of convergence, hands-free control according to the subject's intention can be performed. For example, in a glasses-type wearable terminal that displays AR, the display of an AR image can be started by crossing the eyes, and the display of the AR image can be ended by looking at a distant target.

[0097] There are monocular display and binocular display in AR display. This embodiment can be applied to monocular display, but in the case of binocular display, detection of the convergence of the left and right eyes is effective. In 3D display of AR images by both eyes, the interval between the left and right images (also called the convergence angle) is set arbitrarily, but if the convergence angle of the left and right eyes when viewing the real world is different from the convergence angle of the AR image, the convergence angle of the left and right eyes must be adjusted each time when comparing the real world and the AR image, which causes eye strain. However, if the convergence angle of the left and right images of the AR image is set to match the convergence angle of the left and right eyes, the rotation of the left and right eyes for such convergence angle adjustment is unnecessary, and eye strain does not occur.

[0098] By detecting changes in eye rotation during work, determining the pattern over time, and comparing this with a standard pattern, the work content can be objectively confirmed. For example, in maintenance and inspection work, when confirmation of distant and nearby targets is required, it is possible to confirm whether the work is being performed according to procedure by detecting convergence during actual work. The confirmation result may be notified to the worker as an alarm. Although not a maintenance inspection, the embodiment can also be applied to pointing confirmation after a train departs.

[0099] Comparison with the standard pattern can be used not only to confirm the work content, but also to master the work procedure involving convergence. For example, in the driving lesson for license renewal, a video of a good driver's safety check may be introduced. In this case, rather than simply watching, it is possible to understand the ideal convergence change in more detail by comparing the eye movement pattern when actually performing the same safety check with the eye movement pattern of a good driver.

[0100] The left and right rotation of the eyeballs is not limited to the right and left eyeballs rotating in opposite directions (convergence), but also includes gaze rightward and gaze leftward movements in which the right and left eyeballs rotate in the same direction, and these may be combined with convergence detection to execute hands-free control. In addition, in a state of increased drowsiness, it is also possible to detect nystagmus based on the left and right gaze rightward and leftward movements that occur with the occurrence of a convergence state. This makes it possible to issue a warning to a drowsy subject. Furthermore, not only the horizontal rotation of the eyeballs, but also the vertical rotation may be detected, and hands-free control may be executed by combining the horizontal and vertical rotation of the eyeballs.

[0101] The present invention is not limited to the above-described embodiment as it is, and the components can be modified and embodied in the implementation stage without departing from the gist of the invention. In addition, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0102] 32...right temple electrode, 36...left temple electrode, 42...right nose pad electrode, 44...left nose pad electrode, 46...neutral electrode, 62, 64, 66...A / D converter, 75...eye movement detection unit.

Claims

1. A wearable device connected to an electronic device, a frame that is worn by a user and includes a transparent portion through which the user can see the real world, and a display that displays a virtual reality image; a detector for detecting a convergence angle between a gaze direction of a right eyeball of the user and a gaze direction of a left eyeball of the user; A processor, The processor, determining whether the user is viewing a first object at a first distance in the real world or whether the user is viewing a second object at a second distance in the real world different from the first distance based on the convergence angle detected by the detector; displaying a first augmented reality image on the display when it is determined that the user is looking at the first object; A wearable device that, when it is determined that the user is looking at the second object, sends a request signal to the electronic device and displays on the display a second virtual reality image sent from the electronic device in response to the request signal.

2. Electronic devices, A system comprising: a wearable device connected to the electronic device, The wearable device includes: a frame that is worn by a user and includes a transparent portion through which the user can see the real world, and a display that displays a virtual reality image; a detector for detecting a convergence angle between a gaze direction of a right eyeball of the user and a gaze direction of a left eyeball of the user; A processor, The processor, determining whether the user is viewing a first object at a first distance in the real world or whether the user is viewing a second object at a second distance in the real world different from the first distance based on the convergence angle detected by the detector; displaying a first augmented reality image on the display when it is determined that the user is looking at the first object; If it is determined that the user is looking at the second object, the system sends a request signal to the electronic device and displays on the display a second virtual reality image sent from the electronic device in response to the request signal.

3. An electronic device and a wearable device connected to the electronic device, The wearable device includes: a frame that is worn by a user and includes a transparent portion through which the user can see the real world, and a display that displays a virtual reality image; A display method for a system including a detector for detecting a convergence angle between a gaze direction of the user's right eyeball and a gaze direction of the user's left eyeball, comprising: determining whether the user is viewing a first object at a first distance in the real world or whether the user is viewing a second object at a second distance in the real world different from the first distance based on the convergence angle detected by the detector; displaying a first augmented reality image on the display when it is determined that the user is looking at the first object; A display method comprising: when it is determined that the user is looking at the second object, sending a request signal to the electronic device; and displaying on the display a second virtual reality image sent from the electronic device in response to the request signal.

Images