Wearable device and display method

Abstract

To provide a wearable device which controls display according to the gaze direction, and to provide a display method.SOLUTION: A wearable device according to an embodiment comprises a frame with a display for displaying the real world and an augmented reality image, a detector for detecting the gaze direction of a user, and a processor. The processor uses the gaze direction to display a first augmented reality image including a name of an object seen by the user in the real world on the display.SELECTED DRAWING: Figure 16

Description

[Technical Field]

[0001] FIELD An embodiment of the present invention relates to a wearable device and a display method. [Background technology]

[0002] One method for detecting eye rotation is the electro-oculography (EOG) method. Attaching electrodes to the skin near the left and right eyeballs allows the electro-oculography of both eyes to be detected. Eye rotation can be detected based on the pattern of change in the electro-oculography of the left and right eyeballs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-125693 [Patent Document 2] U.S. Patent No. 8,449,116 [Patent Document 3] Japanese Patent Application Laid-Open No. 2011-125692 [Patent Document 4] U.S. Patent No. 8,434,868 [Patent Document 5] Japanese Patent Application Laid-Open No. 2013-240469 [Patent Document 6] Japanese Patent Application Laid-Open No. 2000-259336 [Patent Document 7] U.S. Patent Application Publication No. 2011 / 0,178,784 [Patent Document 8] Japanese Patent Application Laid-Open No. 2013-244370 [Patent Document 9] Japanese Patent Application Laid-Open No. 2013-215356 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional wearable devices do not control the display based on the gaze direction detected by an eye rotation detection device.

[0005] An object of the present invention is to provide a wearable device and a display method that control display based on gaze direction. [Means for solving the problem]

[0006] According to an embodiment, the wearable device comprises: a frame worn by a user, the frame having a display for displaying real world and augmented reality images; a detector for detecting a user's gaze direction; and a processor.

[0007] The processor causes the display to display a first augmented reality image including the name of the object the user is viewing in the real world based on the gaze direction detected by the detector. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a glasses-type eyeball rotation detection device according to an embodiment, as seen from the front. [Figure 2] FIG. 1 is a diagram showing an example of a glasses-type eyeball rotation detection device as seen from above and behind. [Figure 3] 1 is a diagram showing a user wearing an example of a glasses-type eyeball rotation detection device, viewed from the front right. [Figure 4] 2 is a block diagram showing an example of the electrical configuration of an eyeball rotation detection device. FIG. [Figure 5] FIG. 10 is a diagram showing a first modified example of the arrangement of the neutral electrode 46. [Figure 6] FIG. 10 is a diagram showing a second modified example of the arrangement of the neutral electrode 46. [Figure 7] FIG. 10 is a diagram showing a third modified example of the arrangement of the neutral electrode 46. [Figure 8] FIG. 10 is a diagram showing an EOG signal when the gaze is directed forward. [Figure 9]FIG. 10 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the left from a state in which the gaze is directed forward. [Figure 10] FIG. 10 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the right from a state in which the line of sight is directed forward. [Figure 11] FIG. 10 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction that increases the convergence angle from a state in which the line of sight is directed forward. [Figure 12] FIG. 10 is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction that reduces the convergence angle from a state in which the line of sight is directed forward. [Figure 13] 1A to 1C are diagrams showing examples of waveforms of EOG signals for various eye movements. [Figure 14] 10A and 10B are diagrams illustrating an example of experimental results for detecting a change in convergence angle. [Figure 15] FIG. 10 is a front view of an example of a glasses-type eyeball rotation detection device according to a second embodiment. [Figure 16] 1 is a block diagram showing an example of the electrical configuration of a surgery assistance system including a glasses-type eyeball rotation detection device. [Figure 17] 10A and 10B are diagrams illustrating an example of the operation of the eyeglass-type eyeball rotation detection device. [Figure 18] 10A and 10B are diagrams illustrating another example of the operation of the eyeglass-type eyeball rotation detection device. [Figure 19] 10A and 10B are diagrams illustrating an example of the operation of the eyeglass-type eyeball rotation detecting device according to a modified example of the second embodiment. [Figure 20] FIG. 10 is a diagram showing an example of the electrical configuration of a system including a glasses-type eyeball rotation detection device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described with reference to the drawings. Note that the disclosure is merely an example, and the invention is not limited to the contents described in the following embodiments. Modifications that can be easily conceived by a person skilled in the art are naturally included within the scope of the disclosure. For clearer explanation, the size, shape, etc. of each part may be changed from the actual embodiment and shown schematically in the drawings. In multiple drawings, corresponding elements may be given the same reference numerals, and detailed description may be omitted.

[0010] Here is some basic information about the eyeball. The diameter of an adult eyeball is approximately 25mm. At birth, it is around 17mm and grows larger as the child grows. The interpupillary distance of an adult male is approximately 65mm. For this reason, many commercially available stereo cameras are made with a 65mm gap. The interpupillary distance of an adult female is several mm shorter than that of a male. The electrooculography is several tens of millivolts. The eyeball has a positive potential on the corneal side and a negative potential on the retinal side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred microvolts (called the electrooculography).

[0011] The range of eye rotation (for a typical adult) is 50° or less to the left and 50° or less to the right in the left-right direction (also called the horizontal direction), and 50° or less to the down direction and 30° or less to the up direction (also called the vertical direction). The range of vertical angle that can be moved at will is narrow in the upward direction. This is because there is a "Bell phenomenon" in which the eyeballs roll upward when the eyes are closed, so when the eyes are closed, the range of vertical eye movement shifts upward. The convergence angle (the angle at which the gaze directions of the left and right eyes intersect) is 20° or less.

[0012] [First embodiment] An example of the configuration of an eye rotation detection device according to an embodiment will be described with reference to Figures 1, 2, and 3. There are various types of eye rotation detection devices, but here, an example of an eye rotation detection device in the form of eyewear is shown. Eyewear includes goggles and eyeglasses (sunglasses are equivalent to eyeglasses), but here, an eyeglass-type eye rotation detection device will be described. Figure 1 is a front view of an example of an eyeglass-type eye rotation detection device, and Figure 2 is a top rear view of an example of an eyeglass-type eye rotation detection device. Figure 3 is a view of a user wearing an example of an eyeglass-type eye rotation detection device, viewed from the front right.

[0013] There are two types of eye rotation: up-down and left-right. Up-down rotation includes blinking, closing the eyes, winking, etc. Left-right rotation is a slow movement in which the left and right eyeballs unconsciously rotate in the same direction. Eye movements can be broadly divided into conscious eye movements (convergence and divergence), where the left and right eyeballs rotate in the same direction, and movements where the left and right eyeballs rotate in opposite directions. Convergence is when the gaze directions of the left and right eyeballs cross, while divergence is when the gaze directions of the left and right eyeballs diverge. Eye rotation is detected based on changes in electrooculography. Electrooculography can be detected by the difference in voltage from a pair of electrodes sandwiching the eyeballs. The eyeballs can be sandwiched left and right, up and down, front and back, or even diagonally. Blinking, eye closure, winking, etc. can be detected from electrooculography detected by a pair of electrodes sandwiching the eyeballs from above and below. Blinking, eye closure, winking, slow movements, and gaze movement can be detected from electrooculography detected by a pair of electrodes sandwiching the eyeballs from above and below and left and right. Slow movements, gaze movement, and convergence and divergence can be detected from electrooculography detected by a pair of electrodes sandwiching the eyeballs from front and back and left and right.

[0014] [Electrode arrangement] The glasses include a right frame 12, a left frame 14, and a bridge 26 connecting the frames 12, 14. In this specification, right and left refer to the right and left from the perspective of the user wearing the glasses. In the front view of FIG. 1, the frames are reversed, with the frame on the right side of FIG. 1 being the left frame 14. If the device only detects electro-oculography, the right frame 12 and the left frame 14 do not need to be fitted with lenses or glass. However, if the user regularly wears glasses, lenses with a prescription suited to the user may be fitted into the right frame 12 and the left frame 14 instead of the regular glasses. If the user does not regularly wear glasses, simple glass may be fitted into the right frame 12 and the left frame 14. In a product that detects gaze movement or changes in convergence angle from electro-oculography and applies the detection results, such as a glasses-type wearable device capable of AR display, rather than a device solely for electro-oculography detection, an AR display liquid crystal panel or organic EL panel may be fitted into at least a portion of the right frame 12 and the left frame 14.

[0015] In this embodiment, in order to detect convergence and divergence, electrodes are arranged on the same plane, sandwiching the eyeballs at anterior-posterior positions that are in phase (same vector) with respect to each of the left and right eyeballs, and at lateral positions that are in opposite phase (opposite vector) with respect to each of the left and right eyeballs.

[0016] 2, in order to detect the electro-oculography of the right eyeball ER, a right temple electrode 32 is provided on the right side of the right eyeball ER, for example, on the part of the right temple 18 that rests on the ear, and a right nosepad electrode 42 is provided on the left side of the right eyeball ER, for example, on the surface that comes into contact with the nose of the right nosepad 22 that is attached near the connection point between the right frame 12 and the bridge 16. In a plan view (FIG. 2 is considered to be a plan view), the right temple electrode 32 and right nosepad electrode 42 are positioned so that a line connecting the right temple electrode 32 and the right nosepad electrode 42 passes through the right eyeball ER.

[0017] In the front view (FIG. 1 is considered to be the front view), the right temple electrode 32 is provided on the left side of the right eyeball ER, and the right nose pad electrode 42 is provided on the right side of the right eyeball ER. The right temple electrode 32 and the right nose pad electrode 42 are arranged so that a line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER. In addition, in the front view, the right nose pad electrode 42 is provided slightly above the right temple electrode 32.

[0018] In the side view, the right temple electrode 32 is provided behind the right eyeball ER, i.e., to the left of the right eyeball ER in the right side view and to the right of the right eyeball ER in the left side view, and the right nose pad electrode 42 is provided in front of the right eyeball ER, i.e., to the right of the right eyeball ER in the right side view and to the left of the right eyeball ER in the left side view. The right temple electrode 32 and right nose pad electrode 42 are positioned so that a line connecting the right temple electrode 32 and right nose pad electrode 42 passes through the right eyeball ER.

[0019] Figure 3 shows how the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in the front view, top view, and side view of the head. Note that the line connecting the two electrodes does not have to pass through the center of the right eyeball ER; it can pass through any part of the eyeball. The same is true for the left eyeball, although it is hidden by the face in Figure 3.

[0020] The right temple electrode 32 and right nose pad electrode 42, which detect the electrooculography of the right eyeball ER, are positioned so that the line connecting them passes through the right eyeball ER in any of the plan view, front view, and side view, but it is sufficient that they are positioned so that the line connecting them passes through the right eyeball ER in at least one of the plan view, front view, and side view.

[0021] Similarly, to detect the electro-oculography of the left eye EL, a left nose pad electrode 44 is provided to the right of the left eye EL in a plan view, for example, on the surface of the left nose pad 24 that comes into contact with the nose and is attached near the connection point between the left frame 14 and the bridge 16, and a left temple electrode 36 is provided to the left of the left eye EL, for example, on the part of the left temple 20 that hangs over the ear. The left nose pad electrode 44 and left temple electrode 36 are positioned so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eye EL.

[0022] The right temple electrode 32 and the left temple electrode 36 are symmetrical with respect to a line (for example, a line extending from the center of the nose to the back of the head) that is perpendicular to the midpoint of the line connecting the right frame 12 and the left frame 14.

[0023] In the front view, the left nose pad electrode 44 is provided to the left of the left eyeball EL, and the left temple electrode 36 is provided to the right of the left eyeball EL. The left nose pad electrode 44 and the left temple electrode 36 are positioned so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL. In addition, in the front view, the left nose pad electrode 44 is provided slightly above the second left electrode 42.

[0024] In the side view, the left nose pad electrode 44 is provided in front of the left eyeball EL, i.e., to the right of the left eyeball EL in the right side view and to the left of the left eyeball EL in the left side view, and the left temple electrode 36 is provided behind the left eyeball EL, i.e., to the left of the left eyeball EL in the right side view and to the right of the left eyeball EL in the left side view. The left nose pad electrode 44 and left temple electrode 36 are positioned so that a line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0025] The right temple electrode 32 is provided on both the side (contacting the temporal region) and bottom (contacting the base of the ear) of the right temple 18, so that when the eyeglasses are worn on the face, the weight of the temple 18 causes the right temple electrode 32 to come into contact with the area at the base of the ear where there is no coarse hair. The left temple electrode 36 is provided on both the side (contacting the temporal region) and bottom (contacting the base of the ear) of the left temple 20, so that when the eyeglasses are worn on the face, the weight of the temple 20 causes the left temple electrode 36 to come into contact with the area at the base of the ear where there is no coarse hair. This allows the right temple electrode 32 and left temple electrode 36 to come into close contact with the user's skin, enabling accurate sensing of the electro-oculography.

[0026] Furthermore, the left nose pad electrode 44 and left temple electrode 36, which detect the electrooculography of the left eyeball EL, may also be positioned so that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL in at least one of the plan view, front view, and side view.

[0027] A forehead pad 26 is provided inside the bridge 16 and contacts the forehead. A neutral electrode 46 is provided on the surface of the forehead pad 26 that contacts the forehead. The neutral electrode 46 is an electrode that ensures a neutral potential for electrooculography detection and is in contact with the skin, e.g., the forehead. The neutral electrode 46 is positioned so that the distance between the neutral electrode 46 and the right temple electrode 32 is equal to the distance between the neutral electrode 46 and the left temple electrode 36, and the distance between the neutral electrode 46 and the right nose pad electrode 42 is equal to the distance between the neutral electrode 46 and the left nose pad electrode 44. The neutral electrode 46 is positioned in this manner for the purpose of vergence angle detection, which will be described later. This is because the vergence angle is detected based on the results of symmetrical eyeball rotation of each eyeball as viewed from the front. For example, in an electrocardiogram, a neutral potential is obtained at a body part where the influence of eyeball rotation is negligible, such as the distal end of the right foot. Although it is somewhat affected by eye rotation, by taking a neutral potential at the center of the forehead, a location that is equally affected by both eyeballs, it is possible to equalize the influence of the eye potential that the neutral electrode receives from each eyeball.

[0028] The right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, left temple electrode 36, and neutral electrode 46 are made of metal foil such as copper, small metal pieces, metal spheres such as stainless steel, conductive silicone rubber sheets, etc. These electrodes 32, 42, 44, and 36 are electrodes for detecting the electrooculogram (EOG) as will be described later, and are therefore also referred to as EOG electrodes.

[0029] [EOG signal] As shown in FIG. 2 , a processing unit 30 for detecting electrooculography is built into or externally attached to one temple, for example, the right temple 18, near the frame 12. A battery 34 for the processing unit 30 is built into or externally attached to the other temple, for example, the left temple 20, near the frame 12. The processing unit 30 may not only detect electrooculography but also control the display in a glasses-type wearable device capable of AR display. The processing unit 30 may not be built into the glasses but may be externally attached, and the glasses and the processing unit 30 may be connected wirelessly or via a wire. In this case, the battery 34 may be built into the processing unit 30 and externally attached to the glasses. Alternatively, the processing unit 30 may be divided into two parts: a first processing unit that senses signals from the electrodes may be built into the glasses, and a second processing unit that detects electrooculography from the sensed signals and controls the glasses based on the detection results may be externally attached. A mobile device such as a smartphone can be used as the second processing unit. The second processing unit is not limited to a mobile device directly connected to the glasses, but may also include a server connected via a network.

[0030] The signal from the right temple electrode 32 is input to the negative terminal of a first analog / digital (A / D) converter 62, and the signal from the second left electrode 36 is input to the positive terminal of the first A / D converter 62, which outputs a first EOG signal ADC Ch0, which is a differential signal. Because the right temple electrode 32 and the left temple electrode 36 sandwich the eyeball from the left and right, the first EOG signal ADC Ch0 indicates the left and right rotation of the left and right eyeballs.

[0031] The signal from the right temple electrode 32 is input to the - terminal of the second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, which outputs a second EOG signal ADC Ch1, which is a differential signal. Because the right temple electrode 32 and the right nose pad electrode 42 sandwich the right eyeball from above and below and from left and right, the second EOG signal ADC Ch1 indicates the left-right and up-down rotation of the right eyeball.

[0032] The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, and the signal from the left temple electrode 36 is input to the - terminal of the second A / D converter 66, and a third EOG signal ADC Ch2, which is a differential signal, is output. Because the left nose pad electrode 44 and the left temple electrode 36 sandwich the left eyeball from above and below and from left and right, the third EOG signal ADC Ch2 indicates the left and right rotation and up and down rotation of the left eyeball.

[0033] Since the left and right positions of the two electrodes related to the second EOG signal ADC Ch1 and the left and right positions of the two electrodes related to the third EOG signal ADC Ch2 are opposite (the + / - of the A / D converter input is inverted), it is possible to detect from the waveforms of the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 whether the left and right eyeballs are rotating in the same direction or in opposite directions.

[0034] The voltage signals from the right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, and left temple electrode 36 are weak and therefore subject to significant noise. To cancel this noise, a series circuit of resistors R1 and R2 is connected between the reference analog voltage Vcc (= 3.3 V or 5.5 V) of the A / D converters 62, 64, and 66 and ground (GND), and the neutral electrode 46 is connected to the junction of the resistors R1 and R2. The resistors R1 and R2 have the same value, e.g., 1 MΩ. The A / D converters 62, 64, and 66 can detect analog voltages from 0 V (ground) to the reference analog voltage Vcc and convert the input analog voltage into a digital value in the range from 0 V to 3.3 V, centered around the midpoint of the detectable range, e.g., half a voltage of 3.3 V (referred to as the midpoint voltage). Because the connection point between resistors R1 and R2 is connected to the midpoint voltage terminal and the neutral electrode 46 is connected to the connection point between resistors R1 and R2, the midpoint voltage of the A / D converters 62, 64, 66 becomes the same as the voltage of the human body. As a result, the midpoint voltage of the A / D converters 62, 64, 66 fluctuates in conjunction with the voltage of the human body, and noise mixed in the voltage signals from the EOG electrodes 32, 42, 44, 36 does not get mixed in the digital values output by the A / D converters 62, 64, 66. This improves the S / N ratio of electrooculography detection.

[0035] 4 is a block diagram showing an example of the electrical configuration of the eyeball rotation detection device. The processing unit 30 may include A / D converters 62, 64, and 66, or the A / D converters 62, 64, and 66 may be external to the processing unit 30.

[0036] The signal from the right temple electrode 32 is input to the negative terminal of a first A / D converter 62, and the signal from the left temple electrode 36 is input to the positive terminal of the first A / D converter 62, resulting in a first-channel EOG signal ADC Ch0. The signal from the right temple electrode 32 is input to the negative terminal of a second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the positive terminal of the second A / D converter 64, resulting in a second-channel EOG signal ADC Ch1. The signal from the left nose pad electrode 44 is input to the positive terminal of a third A / D converter 66, and the signal from the left temple electrode 36 is input to the negative terminal of the second A / D converter 66, resulting in a third-channel EOG signal ADC Ch2.

[0037] The signal from the neutral electrode 46 is supplied to the midpoint voltage terminal of the A / D converters 62, 64, 66, and the midpoint voltage of the A / D converters 62, 64, 66 is taken as the voltage of the human body detected by the neutral electrode 46.

[0038] The EOG signals output from the A / D converters 62, 64, and 66 are input to an eye movement detection unit 75 that detects eye rotation (hereinafter, also referred to as eye movement). The eye movement detection unit 75 may be configured as hardware or software. In the latter case, the CPU 74, ROM 76, and RAM 78 are connected to a bus line, and the eye movement detection unit 75 is also connected to the bus line. The eye movement detection unit 75 is realized by the CPU 74 executing a program stored in the ROM 76. A wireless LAN device 80 is also connected to the bus line, and the processing unit 30 is connected to a mobile terminal 84 such as a smartphone via the wireless LAN device 80. The mobile terminal 84 may be connected to a server 88 via a network 86 such as the Internet. The eye movement detection unit 75 detects electro-oculography (EOG) signals output from the A / D converters 62 and 64, and can detect the left and right rotation (convergence and divergence) of each of the left and right eyeballs, the left and right rotation (gaze movement), and the up and down rotation (blinking and eye closure) of the eyeballs from the detected electro-oculography. Furthermore, the eye movement detection unit 75 can estimate various states of the user from the detected eye movement (for example, a state of lack of concentration and restlessness, a state of concentration, a state of tension and mental stress, or a state of fatigue and difficulty concentrating on work or tasks). The type of eye rotation to be detected and the type of state to be estimated can be changed by changing the program executed by the CPU 74. This change instruction may be given from the mobile terminal 84.

[0039] Instead of the wireless LAN device 80, ZigBee (registered trademark), Bluetooth Communication devices using communication methods such as Low Energy (registered trademark) and Wi-Fi (registered trademark) may be used. The detection results (eye movement detection results, state estimation results) of the eye movement detection unit 75 may be temporarily stored in RAM 78 and then sent to the mobile terminal 84 via a communication device such as a wireless LAN device 80. Alternatively, the detection results of the eye movement detection unit 75 may be sent to the mobile terminal 84 in real time. The mobile terminal 84 may store the detection results of the eye movement detection unit 75 in an internal memory (not shown) or may transfer the detection results to a server 88 via a network 86. The mobile terminal 84 may start some processing in response to the detection results of the eye movement detection unit 75, and may store the processing results in an internal memory or transfer the processing results to the server 88 via the network 86. The server 88 may aggregate the detection results from many eye movement detection units 75 and the processing results from many mobile terminals 84 to perform so-called big data analysis.

[0040] [Variations of electrode arrangement] 5, 6, and 7 show modified arrangements of the neutral electrode 46. In the above description, separate left and right nose pads 22, 24 are provided, but in the modified example shown in Fig. 5, an integrated, inverted-V or inverted-U shaped nose pad 52 is provided. The right nose pad electrode 42 is provided on the inside right side of the open sides of the nose pad 52, the left nose pad electrode 44 is provided on the inside left side, and the neutral electrode 46 is provided inside the apex of the V or U shape. This makes it possible to provide the neutral electrode 46 in contact with the forehead without providing a forehead pad 26.

[0041] 6 also has integrated V- or U-shaped nose pads 54. The nose pads 52 are widened downward, but the nose pads 54 are widened toward the front. The right nose pad electrode 42 is provided on the right side of the nose pad 54, the left nose pad electrode 44 is provided on the left side, and the neutral electrode 46 is provided in the center.

[0042] In the modified examples shown in Figures 5 and 6, nose pads with a larger area than normal nose pads are used, so even if the glasses-type electro-oculography detection device or glasses-type wearable terminal glasses capable of AR display, which are heavier than normal glasses, are used for a long period of time, the weight is less likely to cause pain in the nose.

[0043] 7 uses separate left and right nose pads 22, 24, but does not require the forehead pad 26. Here, a right nose pad electrode 42 and a right neutral electrode 46a are provided on the surface of the right nose pad 22 that comes into contact with the nose, and a left nose pad electrode 44 and a left neutral electrode 46b are provided on the surface of the left nose pad 24 that comes into contact with the nose. The right neutral electrode 46a and the left neutral electrode 46b are electrically shorted and are equivalent to a single neutral electrode 46.

[0044] [Relationship between eye movement and EOG signals] Referring to Figures 8 to 12, an example of changes in the waveforms of the EOG signal ADC Ch0 output from the A / D converter 62, the EOG signal ADC Ch1 output from the A / D converter 64, and the EOG signal ADC Ch2 output from the A / D converter 66 are shown when the right eyeball and left eyeball are rotated left and right from a state in which the line of sight is facing forward.

[0045] FIG. 8 shows a state in which the user's gaze direction is forward. When looking at infinity, the gaze directions of the right eyeball and the left eyeball are parallel, but when looking at a finite far point, the gaze directions of the right eyeball and the left eyeball intersect at the far point. From this state, as shown in FIG. 9, when both the right eyeball ER and the left eyeball EL rotate left (the gaze directions of the right eyeball and the left eyeball move left), the positively charged cornea of the right eyeball ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When both the left and right eyes rotate right in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 connected to the right and left temple electrodes 32, 36 is a signal with a convex waveform (a waveform that convex upward). The second EOG signal ADC Ch1 output from the second A / D converter 64 connected to the right nose pad electrode 42 and right temple electrode 32 is a signal with a convex waveform (a waveform that convex upward). The third EOG signal ADC Ch2 output from the third A / D converter 66 connected to the left nose pad electrode 44 and left temple electrode 36 is a signal with a concave waveform (a waveform that convex downward).

[0046] In this way, the left and right eyeballs rotate in the same direction (left rotation), so the second EOG signal ADC An EOG signal of opposite phase appears on Ch1 and the third EOG signal ADC Ch2. An EOG signal of the same phase as the second EOG signal ADC Ch1, which has the same + / - relationship, appears on the first EOG signal ADC Ch0.

[0047] When the right eyeball ER and the left eyeball EL both rotate to the right as shown in FIG. 10 (the gaze directions of the right eyeball and the left eyeball move to the right) from the state in which the gaze direction is forward as shown in FIG. 8, the positively charged cornea of the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When both the left and right eyes rotate left in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32, 36 are connected becomes a concave waveform (a waveform that is convex downward). The second EOG signal ADC Ch1 output from the second A / D converter 64 connected to the right nose pad electrode 42 and the right temple electrode 32 is a concave waveform (a waveform that is convex downwards). The third EOG signal ADC Ch2 output from the third A / D converter 66 connected to the left nose pad electrode 44 and the left temple electrode 36 is a convex waveform (a waveform that is convex upwards).

[0048] In this way, the left and right eyeballs rotate in the same direction (clockwise rotation), so the second EOG signal ADC Opposite phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. However, they are opposite phases when both the right eyeball ER and the left eyeball EL rotate to the left. The first EOG signal ADC Ch0 appears in phase with the second EOG signal ADC Ch1, which has the same + / - relationship. However, the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate to the right is opposite phase to the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate to the left.

[0049] As shown in FIG. 11, when the gaze direction is forward as shown in FIG. 8, the right eye ER rotates left (the gaze direction of the right eye moves left) and the left eye EL rotates right (the gaze direction of the left eye moves right), resulting in convergence, or cross-eyedness, where the gaze directions of the left and right eyes intersect. The positively charged cornea of the right eye ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of the left eye EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When the right eye rotates right and the left eye rotates right in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected remains unchanged, and neither a convex waveform nor a concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 connected to the right nose pad electrode 42 and the right temple electrode 32 is a signal with a convex waveform (a waveform that convex upward). The third EOG signal ADC Ch2 output from the third A / D converter 66 connected to the left nose pad electrode 44 and the left temple electrode 36 is a signal with a convex waveform (a waveform that convex upward).

[0050] Since the left and right eyeballs rotate in opposite directions in this way, waveforms of the same phase appear in the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2.

[0051] When the electrooculograms of the left and right eyeballs are the same and the absolute values of the rotation angles are also the same, both the positive and negative terminals of the A / D converter 62 change by the same amount in the same direction (negative direction), so no change is seen in the relative values of the two, and no change in the electrooculogram appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrodes and temple electrodes is slightly offset from the center of the left and right eyeballs, so slight changes appear according to the amount of this offset.

[0052] When the gaze direction of the right eye ER rotates to the right (the gaze direction of the right eye moves to the right) and the left eye EL rotates to the left (the gaze direction of the left eye moves to the left) as shown in FIG. 12 from the state in which the gaze direction is forward as shown in FIG. 8, i.e., the gaze directions of the left and right eyes diverge, resulting in so-called wide-set eyes. The positively charged cornea of the right eye ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of the left eye EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When the right eye rotates left in this state, the state returns to that shown in FIG. 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected remains unchanged, and neither a convex nor concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected is a concave waveform (a waveform that is convex downward). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected is a concave waveform (a waveform that is convex downward). As the left and right eyeballs rotate in opposite directions in this way, the second EOG signal ADC In-phase EOG signals appear on Ch1 and the third EOG signal ADC Ch2, but the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 for wide-set eyes are out of phase with the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 for cross-set eyes.

[0053] When the electrooculograms of the left and right eyeballs are the same and the absolute values of the rotation angles are also the same, both the positive and negative terminals of the A / D converter 62 change by the same amount in the same direction (positive direction), so no change is seen in the relative values of the two, and no change in the electrooculogram appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrodes and temple electrodes is slightly offset from the center of the left and right eyeballs, so slight changes appear according to the amount of this offset.

[0054] 13 is an electrooculogram (EOG) illustrating an example of the relationship between various eye movements of a user and the EOG signals ADC Ch0, ADC Ch1, and ADC Ch2 obtained from the A / D converters 62, 64, and 66. The vertical axis represents the sample values of the A / D converters 62, 64, and 66 (e.g., 3.3 V, 24-bit A / D converters), and the horizontal axis represents time.

[0055] As shown in FIG. 11, when neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and convex waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, the eye movement detection unit 75 detects a "cross-eyed" state in which the gaze directions of the left and right eyes converge. Although not shown in FIG. 13, as shown in FIG. 12, when neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and concave waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, the eye movement detection unit 75 detects a "wide-eyed" state in which the gaze directions of the left and right eyes diverge. As described above, convergence and divergence differ only in the convex and concave waveforms of the EOG signals ADC Ch1 and ADC Ch2, and are otherwise the same, in the following description, convergence and divergence may be collectively referred to as "vergence." The amplitude of the waveforms of the EOG signals ADC Ch1 and ADC Ch2 corresponds to the degree of convergence (convergence angle) and the degree of divergence. As will be described later with reference to Figure 14, the degree of change in amplitude increases as the distance decreases and decreases as the distance increases, so the sensitivity of detecting amplitude change increases as the distance decreases.

[0056] As shown in FIG. 9, a convex waveform appears in the EOG signal ADC Ch0, a convex waveform appears in the EOG signal ADC Ch1, and a concave waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 detects a leftward movement of the gaze direction.

[0057] As shown in FIG. 10, a concave waveform appears in the EOG signal ADC Ch0, a concave waveform appears in the EOG signal ADC Ch1, and a convex waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 detects a rightward movement of the gaze direction.

[0058] In the EOG signal ADC Ch1 and the EOG signal ADC Ch2, one blink (1), two blinks (2), and three blinks (3) are detected by convex pulse waveforms of waves 1 to 3 in phase, which momentarily rise in level and then return to their original state. The eye movement detection unit 75 detects vertical eye rotation, i.e., eye closure, by a combination of a convex waveform (upward convex waveform) when the gaze is directed upward and a concave waveform (downward convex waveform) when the gaze is directed downward in the EOG signal ADC Ch1 and the EOG signal ADC Ch2. Thus, vertical eye rotation caused by blinking and eye closure can be detected based on either the EOG signal ADC Ch1 or ADC Ch2. Therefore, in applications where only blinking and eye closure are to be detected, there is no need to provide an electrode pair for each of the left and right eyes; an electrode pair may be provided for only one of the eyes.

[0059] This figure shows an example of experimental results for detecting changes in convergence state according to an embodiment. Using the prototype electro-oculography detection device shown in Figures 1 to 4, Figure 14 shows an example of changes in the second EOG signal ADC Ch1 when a subject changes the intersection of the gaze directions of the left and right eyes from looking at a fingertip 10 cm in front of their nose to looking at a marker further away. In this case, the intersection of the gaze directions changes from near to far, resulting in a decrease in the convergence angle. The horizontal axis represents the movement distance of the intersection of the gaze directions from 10 cm away to the depth direction, and the first plot shows the EOG amplitude when the intersection of the gaze directions is moved from 10 cm away to 20 cm away. Note that 10 cm is the shortest distance at which stable gaze is possible. The minimum detection voltage of the EOG signal is set to 50 μV. In other words, the eye movement detection unit 75 can detect a change in the EOG signal when the EOG signal changes by 50 μV or more, and based on this, can detect a change in the intersection of the gaze directions, i.e., a change in the convergence angle. However, if the EOG signal does not change by 50 μV or more, the change in the EOG signal cannot be detected. If the average value obtained by integrating the measured values is used, the minimum detectable voltage of the EOG signal will be smaller, but here it is set to 50 μV. Note that the amplitude of the EOG signal depends on the contact resistance of the electrodes, and if an electrode material with low contact resistance is used, the amplitude of the EOG signal will be larger, and the minimum detectable voltage of the EOG signal will also be larger.

[0060] For example, if the subject changes the convergence angle from a state in which they are looking at a marker 30 cm in front of their nose to a more distant marker, the eye movement detection unit 75 detects a change in EOG amplitude when the EOG amplitude increases by 50 μV. The EOG amplitude obtained by adding 50 μV to the EOG amplitude when they are looking at a marker 30 cm in front of their nose corresponds to a marker 40 cm away. In other words, the eye movement detection unit 75 can detect a change in EOG amplitude when the subject changes the convergence angle from a state in which they are looking at a marker 30 cm in front of their nose to a state in which they are looking at a marker 40 cm away. The ability to detect a change in EOG amplitude corresponding to a 10 cm change in the convergence angle indicates that the detection resolution is quite good. Figure 14 shows that a change in EOG amplitude can be detected when the subject changes the convergence angle from a state in which they are looking at a marker 50 cm in front of their nose to a state in which they are looking at a marker 65 cm or further away, and a change in EOG amplitude can be detected when the subject changes the convergence angle from a state in which they are looking at a marker 70 cm in front of their nose to a marker 1.4 m or further away.

[0061] As described above, the first embodiment provides a glasses-type eye rotation position detection device that includes right and left temple electrodes, right and left nose pad electrodes, and a neutral electrode arranged so as to be equally affected by the rotation of the right and left eyeballs, and that detects convergence by independently detecting the left and right rotation of each of the right and left eyeballs. Because the neutral potential from the neutral electrode is set as the midpoint potential of the A / D converter that samples the EOG signal from the electrode, the EOG signal from the electrode is not affected by noise, allowing for accurate detection of the electrooculography, resulting in accurate detection of eye rotation. The first embodiment also makes it possible to detect left and right rotation of both eyes (left and right movement of the gaze direction) and up and down rotation of the eyeballs.

[0062] The electro-oculography detecting device of the first embodiment can detect convergence, which is a conscious eye rotation of a subject, and thereby realizes an application example in which control is performed according to the subject's intention by performing control according to the detection result. For example, in eyewear capable of displaying augmented reality (hereinafter referred to as AR), it is possible to control the AR display on / off and the display position of an AR image in a hands-free manner according to the detection of convergence.

[0063] In addition, certain tasks may require task-specific changes in convergence, and by comparing the user's pattern of convergence changes with a reference pattern, it is possible to determine the user's level of proficiency in the task and whether the task is being performed correctly.

[0064] Below, application examples of the convergence detection results are described. One application example is incorporating an electrooculography detection device into a glasses-type wearable device capable of AR display, and controlling the AR display based on the convergence detection. For example, the convergence detection can be used as a function switch. For example, as shown in FIG. 8, when the user looks at a distant object in front of them, the AR display can be turned off. When the user changes the line of sight of both eyes to look at a close distance (convergence state) as shown in FIG. 11, the display can be controlled to turn on and off so that the AR display is displayed. Since the display position of the AR image is set to a close distance, the user is in a convergence state when looking at the AR image, so the AR display continues. When the user changes the line of sight of both eyes to look at a distant object, the AR display is turned off. This enables AR display control as intended by the user. Below, as a second embodiment, a glasses-type wearable device for surgical assistance, which is an application example of this, is described.

[0065] [Second embodiment] FIG. 15 is a front view of an example of a glasses-type electro-oculography detecting device 100 according to the second embodiment. The difference from the detecting device of the first embodiment is that displays for AR display (e.g., organic EL panels or liquid crystal panels) 102, 104 are embedded in at least a part of the right frame 12 and the left frame 14. If AR images are not displayed in 3D, the displays 102 or 104 do not need to be embedded in either the right frame 12 or the left frame 14. Here, it is assumed that AR images are displayed in 3D. The arrangement of the EOG electrodes is the same as in the first embodiment. There are various application examples of the glasses-type electro-oculography detecting device 100 capable of AR display, but the second embodiment will be described using a surgery support system as an example. The glasses-type electro-oculography detecting device 100 is worn by a surgeon performing surgery and staff involved in the surgery.

[0066] 16 is a block diagram showing an example of the electrical configuration of a surgery support system including the eyeglass-type electro-oculography detecting device 100. The processing unit 30 has the same configuration as the first embodiment shown in FIG. 4, but also includes a display controller 112. The display controller 112 controls the AR display on the displays 102 and 104. The control of the AR display includes on / off control of the AR display, control of the display position of the AR image (control of the convergence angle of the AR image), etc.

[0067] In a surgery support system, multiple doctors and staff members are involved in the same surgery and multiple pairs of glasses are used, so it is preferable that the glasses processing unit 30 be connected to a control device 120 such as a personal computer with higher performance than a mobile terminal 82 such as a smartphone. Although not shown, multiple glasses processing units 30 are connected to the control device 120. The control device 120 includes a vital data memory 124, an support image memory 122, and a support information memory 126. The eye movement detection unit 75 may also be provided within the control device 120.

[0068] A vital data measurement unit 127 is connected to the control device 120, and measures the patient's electrocardiogram, blood pressure, pulse rate, cumulative amount of blood transfusion, etc., and stores the data for each patient in a vital data memory 124 within the control device 120. A support image memory 122 stores support images for surgical support. As needed, support images in a surgery support database 130 within the server 88 are downloaded to the control device 120 and stored in the support image memory 122. A support information memory 126 stores support text for surgical support. As needed, support text in the surgery support database 130 within the server 88 is downloaded to the control device 120 and stored in the support information memory 126.

[0069] An example of the operation of the eyeglass-type electro-oculography detecting device 100 will be described with reference to FIGS. 17 and 18. Here, it is assumed that the distance from the eyeball to the affected area (real world) during surgery is approximately 40 cm. When the surgeon looks at a point in front of the affected area (real world), for example, a point approximately 30 cm away from the eyeball, the eye movement detecting unit 75 detects convergence. As shown in FIG. 11, the eye movement detecting unit 75 detects convergence based on the fact that the waveform of the EOG signal ADC Ch0 does not change and the waveforms of the EOG signal ADC Ch1 and the EOG signal ADC Ch2 become upwardly convex. When the eye movement detecting unit 75 detects convergence, it requests an AR image related to surgical support from the control device 120 and causes the display controller 112 to perform AR display. An example of AR display is superimposing a semi-transparent vital data window (AR image) on the affected area (real world) during surgery, as shown in FIG. 17(a). The window includes multiple pages, and each page displays an electrocardiogram, blood pressure, pulse, cumulative blood transfusion amount, etc. The window is limited to a partial area of the screen so as not to hide the affected area. In response to a request from the processing unit 30, the control unit 120 reads the patient's vital data from the vital data memory 124 and transfers it to the processing unit 30.

[0070] When an AR image is displayed in 3D, the display position can be set arbitrarily by adjusting the convergence angle between the left and right images. Here, the display position of the vital data window is set to a position approximately 30 cm forward, which is equal to the distance at which the eye movement detection unit 75 detects convergence. Therefore, when the vital data window is displayed, the doctor is looking at a point at the same distance as the display position, allowing him or her to instantly check the contents of the window. In addition, there is no need to rotate the eyes to adjust the convergence angle in order to gaze at the window, and eye strain does not occur.

[0071] The page switching of the vital data window may be performed automatically at a fixed interval, for example, every second, or may be performed at the user's discretion based on the eyeball rotation in another direction detected by the eye movement detection unit 75. For example, when eye closure is detected for 0.5 seconds or more, the window page may be switched. Furthermore, the window page may be switched based on the direction of gaze movement. For example, the page may be switched to the next page when the gaze moves to the right, and the page may be switched to the previous page when the gaze moves to the left. Figure 17(b) shows an example of a window page being switched.

[0072] 17(a) or 17(b), when the doctor or other medical professional changes his or her line of sight to view the affected area (real world) (more than about 40 cm away) during surgery, the eye movement detection unit 75 detects that the convergence angle is decreasing (the distance to the intersection of the line of sight of both eyes is increasing), and causes the display controller 112 to stop the AR display. As a result, the doctor or other medical professional observes only the affected area during surgery as shown in FIG. 17(c) through the right frame 12 and left frame 14 of the glasses.

[0073] If a doctor or other medical professional wishes to refer to vital data in the state shown in Figure 17(c), the doctor or other medical professional changes the direction of their line of sight to look at something close to them (approximately 30 cm). The eye movement detection unit 75 detects that the convergence angle is increasing (the distance to the intersection of the line of sight of both eyes is getting closer), and causes the display controller 112 to perform AR display, resulting in the display of a vital data window such as that shown in Figure 17(a) or (b).

[0074] The increase / decrease in the convergence angle can be determined based on the relationship between the EOG amplitude and the movement distance of the intersection of the lines of sight as shown in Figure 14.If the EOG voltage decreases / increases by more than a certain voltage, it can be determined that the convergence angle has increased / decreased.

[0075] In this way, doctors can turn on and off AR images that support surgery hands-free. Since their hands are occupied during surgery, being able to turn AR displays on and off hands-free is extremely effective.

[0076] Another example of an AR display may be a support image as shown in FIG. 18(a). The support image may be an image of a previous surgical procedure performed on another patient with similar symptoms, or an image from another surgery performed on the patient. The image may be a still image or a video. For this reason, images during surgery are taken by a camera (not shown), and the taken images are uploaded to the server 88 and stored in the surgery support database 130. Another example of an AR display may be support information as shown in FIG. 18(b). The support information is various text data related to the surgery being performed.

[0077] The display position of the support information is set close to the vital data window. However, the display position of the support image may be set at a distance of approximately 40 cm, the same as the affected area (real world) during surgery. The vital data window and support information are not viewed simultaneously with the affected area during surgery, but the support image is expected to be compared side by side with the affected area. When comparing the affected area and the support image, if the display positions are different, the patient's eyes must be rotated left and right, which may cause eye strain. Therefore, unlike the above description, the support image is displayed when the eye is in the state shown in Figure 8 and is turned off when convergence is detected as shown in Figure 11. Note that if the position of the support image and the affected area (real world) during surgery are slightly misaligned, the convergence angle detected by the eye movement detection unit 75 will change slightly when comparing the images. If the display controller 112 adjusts the display position of the support image (the convergence angle of the left and right images) to minimize this change, the possibility of eye strain can be further reduced.

[0078] Switching from the vital data window to the support image and support information may be performed at the user's discretion based on the eye movement detection unit 75 detecting the rotation of the eyeball in another direction, similar to page switching in the vital data window.

[0079] In the above example, a change in the convergence angle is used to switch the AR display on and off. However, the detected convergence angle may be used for other control, and the AR display may be switched on and off based on other eye movements. For example, the AR display may be switched on and off when multiple eye closures are detected. The detected convergence angle may then be used to control the display position of the AR image. That is, the distance between the left and right images of the AR image (sometimes referred to as the image convergence angle) is adjusted based on the distance to the intersection of the line of sight of the left and right eyes at the start of displaying the AR image. The image convergence angle is adjusted by the display controller 112. This eliminates the need to rotate the eyes left and right to focus on the AR display, thereby preventing eye strain when viewing the AR display.

[0080] Another application example of the change in convergence angle and the control of AR display will be described with reference to Fig. 19. This example is an example of picking in a warehouse, and the worker wears the eyeglass-type electro-oculography detecting device 100 during the picking work.

[0081] The configuration of the processing unit 30 may be the same as that shown in FIG. 16, and is therefore not shown in the figure. The configuration of the control device 120 is the same except that the vital data, support images, and support information are changed to a pickup list, and is therefore not shown in the figure. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used, as in the first embodiment shown in FIG. 4. The mobile terminal 84 may acquire location information. The configuration of the server 88 is also the same except that the surgical support data is changed to map information regarding the positions of shelves in the warehouse, a list of products on each shelf in the warehouse, and a list of products to be picked up, and is therefore not shown in the figure. A pickup list for each worker is downloaded from the server 88 to the control device 120.

[0082] In this application example, the AR display is initially off, and the displays of the right frame 12 and left frame 14 of the glasses are transparent and show nothing. Therefore, the worker gazes at the warehouse interior as shown in FIG. 19(a) through the right frame 12 and left frame 14 of the glasses. If the mobile terminal 84 serving as the control device 120 is capable of acquiring location information, air tags may be displayed in AR on products on each shelf based on the map information in the server 88 and the acquired location information. At this time, the worker is looking several meters or even several tens of meters or more ahead.

[0083] The worker rotates his / her left and right eyes in opposite directions to each other so that convergence occurs, or so-called cross-eyedness, as shown in Figure 11, so that the worker changes from looking at a distant target to looking at a nearby target, and the eye movement detection unit 75 detects the convergence.

[0084] In response to the detection of congestion, the processing unit 30 requests the control device 120 for a pickup list showing the products to be picked up by the worker, and causes the display controller 112 to start AR display. An example of the pickup list is shown in FIG. 19(b). The display position of the pickup list is also set to nearby. When the worker rotates their left and right eyeballs from a state where they are looking at the pickup list to a state where they are looking at the distant warehouse interior as shown in FIG. 8, the eye movement detection unit 75 no longer detects congestion, and causes the display controller 112 to stop displaying the pickup list.

[0085] Pickup work, like surgery, requires constant use of your hands, so being able to turn the AR display on and off hands-free is extremely effective.

[0086] In Figure 19, the air tags are always displayed, but they can be displayed only when the user is looking at distant landmarks and turned off when the user is looking at nearby landmarks. For example, when a worker goes out to perform maintenance on a building elevator and searches for the target building during the visit, the worker looks at distant landmarks, so an air tag containing the building's name, distance, work purpose, etc. is superimposed on the target building. Here, when the worker looks down to check a map, etc., the air tag display can be turned off. When the worker arrives at the destination and stands in front of the elevator, an air tag is displayed at the inspection point while the worker is looking at the area to be inspected. When the worker looks down, an AR version of the work manual, etc. is displayed instead of the air tag. This example can also be applied to tourist guides. For example, when looking at the scenery at a tourist spot, an air tag related to that spot can be displayed, but when looking down, the air tag can be turned off and a detailed tourist guide can be displayed instead.

[0087] Another example of AR display based on convergence detection is incorporating an electrooculography detection device into AR glasses used for watching sports. The game is filmed from various angles and stored as replay footage on a server. The AR glasses are connected to the server via a network. Spectators are always focused on the game and looking at distant targets. At this time, the AR display is turned off. If a spectator in the outfield seats at a baseball stadium wants to see an enlarged image of a crossplay on home base, they can rotate their eyes to cross their eyes and activate the AR display. In the AR display, various replay images are downloaded from the server and displayed. This allows spectators to instantly enjoy replay images by rotating their eyes.

[0088] In the above explanation, AR display is started when congestion is detected and stopped when congestion is no longer detected, but the reverse is also possible: AR display may be performed when congestion is not detected, and AR display may be stopped when congestion is detected.

[0089] [Third embodiment] Another application of convergence detection is confirmation of work procedures. FIG. 20 is a block diagram showing an example of the electrical configuration of a work confirmation system including a glasses-type electro-oculography detection device. While AR display is not required for work confirmation, AR display may be used to immediately notify the worker of the confirmation results. The processing unit 30 is the same as that of the second embodiment shown in FIG. 16. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used, as in the first embodiment shown in FIG. 4. The control device 120 includes a gaze movement determination unit 148 and a gaze movement pattern memory 142. The gaze movement determination unit 148 determines gaze movement (gaze direction rightward movement, gaze direction leftward movement, or gaze direction crossing (convergence)) from the change pattern of eye rotation detected by the eye movement detection unit 75, and stores the gaze movement pattern in the gaze movement pattern memory 142. The data in the gaze movement pattern memory 142 is uploaded to the server 88 via the network 86.

[0090] The server 88 includes a standard gaze movement pattern memory 144 and a work procedure discrimination unit 146. Some tasks require task-specific gaze movements. For example, in the railway industry, pointing confirmation after departure requires checking distant and nearby targets in a predetermined order. Furthermore, visual inspection of structures such as bridges and tunnels requires specific locations to be visually inspected, requiring the gaze direction to be shifted in a predetermined pattern. The standard pattern memory 144 stores standard patterns of gaze direction movements specific to each task. The work procedure discrimination unit 146 compares the worker's gaze movement pattern uploaded from the control device 120 with the standard gaze movement pattern stored in the standard pattern memory 144 to determine whether the worker is performing the task correctly. This may be stored for each worker in a database (not shown). The worker's proficiency can be estimated based on changes in the judgment results over time. Alternatively, if the work procedure determination unit 146 determines that the worker is not performing the work correctly, it may notify the display controller 112 of the processing unit 30 via the control device 120, and cause a warning message to be displayed on the displays 102 and 104.

[0091] Another example of a standard eye movement pattern is a pattern related to driving a car. During driver's license renewal training, a video is sometimes shown showing how a good driver performs safety checks. In this case, rather than simply watching, the eye movement pattern when actually performing the same safety check can be compared with the eye movement pattern of a good driver to determine the driver's proficiency in safety checks. In the transportation industry, accidents can also be reduced by having new drivers memorize the same eye movement patterns as experienced drivers.

[0092] The above description is not limited to eyeglasses, but can also be applied to goggles. For example, electrodes may be provided on a part of the foam on the front of the goggles instead of the nose pads, and electrodes may be provided on the belt instead of the temples.

[0093] [calibration] Electrooculograms do not provide a specific convergence angle, but only the rotational direction of the eyeballs that results in a convergence state. Therefore, in the above-described embodiment, convergence is detected, and in the application example, control is performed based on a binary state of whether or not convergence is detected, regardless of the convergence angle itself. However, when there are multiple targets with known distances, by viewing the multiple targets and periodically measuring the electrooculogram at multiple distances, it is possible to obtain electrooculogram values at the multiple known distances, and to identify three or more convergence states. For example, when viewing a distant target, the AR display may be turned off, a first AR image may be displayed when viewing a medium-distance target, and a second AR image may be displayed when viewing a close-distance target.

[0094] Furthermore, because the contact resistance of the electrodes changes (decreases over time), the absolute value of the electrooculogram relative to distance changes over time, and the absolute value of distance cannot be guaranteed over the long term. For example, immediately after starting to use the electrodes, the contact resistance is high, resulting in a small electrooculogram amplitude. Over time, the resistance decreases and the electrooculogram amplitude increases. However, by periodically calibrating the electrooculogram relative to distance, it is possible to estimate the change in electrooculogram over time and guarantee the absolute value of the electrooculogram relative to distance. For example, in tasks where the focal length is constant, such as surgery or desk work, or in tasks where the work process ensures that the worker is looking at a target at a known distance (level) at a certain point in time, it is possible to perform periodic calibration with the distance known.

[0095] Furthermore, the body potential itself may fluctuate, which can also be compensated for by periodic calibration.

[0096] [Summary of effects of the embodiment] According to the above-described embodiment, the left and right rotation of the right eyeball and the left and right rotation of the left eyeball can be detected independently, thereby making it possible to detect convergence, which is the intersection of the gaze directions of the left and right eyeballs. Convergence does not occur unconsciously, but occurs due to the subject's conscious eye rotation. Therefore, by performing control in response to the detection of convergence, hands-free control can be performed according to the subject's intention. For example, in a glasses-type wearable device that displays AR, the display of an AR image can be started by crossing the eyes, and the display of the AR image can be ended by looking at a distant target.

[0097] AR display includes monocular display and binocular display. While this embodiment can be applied to monocular display, in the case of binocular display, detecting the convergence of the left and right eyes is effective. In 3D display of AR images using both eyes, the distance between the left and right images (also called the convergence angle) is set arbitrarily. However, if the convergence angle of the left and right eyes when viewing the real world differs from the convergence angle of the AR image, the convergence angle of the left and right eyes must be adjusted each time when comparing the real world and the AR image, which causes eye strain. However, if the convergence angle of the left and right images of the AR image is set to match the convergence angle of the left and right eyes, the rotation of the left and right eyes to adjust the convergence angle is unnecessary, and eye strain does not occur.

[0098] By detecting changes in eye rotation during work, determining the pattern over time, and comparing this with a standard pattern, the work content can be objectively confirmed. For example, when checking distant and nearby targets during work such as maintenance and inspection, detecting convergence during actual work can confirm whether the work is being performed according to procedure. The confirmation result may be notified to the worker as an alarm. Although not a maintenance and inspection, the embodiment can also be applied to pointing and checking after a train departs.

[0099] Comparison with standard patterns can be used not only to confirm the work content, but also to familiarize the work procedures involving vergence. For example, in driving lessons when renewing a driver's license, a video may be shown showing how a good driver performs safety checks. In this case, rather than simply watching, it is possible to compare the eye movement patterns of a good driver when actually performing the same safety checks with those of the good driver, allowing for a more detailed understanding of the ideal vergence changes.

[0100] Note that the left and right rotation of the eyeballs is not limited to the rotation of the right and left eyeballs in opposite directions (convergence), but also includes gaze shifts to the right and left, in which the right and left eyeballs rotate in the same direction, and hands-free control may be performed by combining these with convergence detection. Furthermore, nystagmus can also be detected based on the left and right gaze shifts to the right and left that occur in a state of increased drowsiness along with the occurrence of a convergence state. This makes it possible to issue a warning to a drowsy subject. Furthermore, not only horizontal rotation of the eyeballs but also vertical rotation may be detected, and hands-free control may be performed by combining these with the horizontal and vertical rotation of the eyeballs.

[0101] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0102] 32...right temple electrode, 36...left temple electrode, 42...right nose pad electrode, 44...left nose pad electrode, 46...neutral electrode, 62, 64, 66...A / D converter, 75...eye movement detection unit.

Claims

1. a frame worn by a user, the frame having a display for displaying real world and augmented reality images; a detector for detecting a gaze direction of the user; a processor for displaying on the display a first augmented reality image including a name of an object that the user is viewing in the real world based on the gaze direction detected by the detector; A wearable device comprising:

2. The wearable device according to claim 1 , wherein the processor switches between displaying and hiding the first augmented reality image in response to a change in the line of sight.

3. The wearable device according to claim 1 or 2, wherein the processor transmits a request signal to an electronic device and receives the first augmented reality image transmitted from the electronic device in response to the request signal.

4. A display method for a wearable device including a display that displays a real world and an augmented reality image, a frame worn by a user, and a detector that detects the user's line of sight, comprising: A display method that displays on the display a first augmented reality image including the name of an object that the user is viewing in the real world based on the gaze direction detected by the detector.

Images