Wearable devices, systems, and display methods

Abstract

To provide a wearable device, a system, and a display method that control the display on the basis of the results of detecting the rotation of the right eye and the rotation of the left eye.SOLUTION: The wearable device is configured to determine whether a user is looking at a first object at a first distance in the real world or looking at a second object at a second distance which is different from the first distance on the basis of the angle of convergence detected by a detector, display a first augmented reality image on a display when the user is determined to be looking at the first object, transmit a request signal to an electronic device when the user is determined to be looking at the second object, and display a second virtual reality image transmitted from the electronic device in response to the request signal on the display.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] Embodiments of the present invention relate to wearable devices, systems, and display methods.

Background Art

[0002] As one method for detecting eye rotation, there is an electro-oculogram (hereinafter also referred to as electro-oculogram: also referred to as EOG method). When electrodes are attached to the skin near the left and right eyes, the electro-oculograms of the left and right eyes can be detected. Eye rotation can be detected based on the change patterns of the electro-oculograms of the left and right eyes.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Patent Document 8

Patent Document 9

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional eye rotation detection devices cannot detect the rotation of the right eyeball and the rotation of the left eyeball individually.

[0005] The object of the present invention is to provide a wearable device, system, and display method that control the display based on the detection result of the rotation of the right eyeball and the detection result of the rotation of the left eyeball. [Means for solving the problem]

[0006] According to the embodiment, control device Wearable devices connected to this are A transparent part that is worn by the user and allows the user to see the real world, expansion A frame comprising a display that shows real-world images, A detector that detects the convergence angle between the line of sight of the user's right eye and the line of sight of the user's left eye, It comprises a processor and

[0007] The processor is Based on the convergence angle detected by the detector, it is determined whether the user is looking at a first object at a first distance in the real world, or whether the user is at a first distance in the real world. Closer distance Determine if you are looking at a second object at a second distance, If the system determines that the user is looking at the first object, The control device is requested to provide location information, and an air tag based on the location information is used as the first augmented reality image. Display it on the screen, If the system determines that the user is looking at a second object, The control device requests list information representing instructions for the user, and the list information is used as a second augmented reality image. Display it on the screen. [Brief explanation of the drawing]

[0008] [Figure 1] This is a front view of an example of a spectacle-type eye rotation detection device according to an embodiment. [Figure 2] This is a view from the rear and above of an example of a glasses-type eye rotation detection device. [Figure 3] This is a view from the front right of a user wearing an example of a glasses-type eye rotation detection device. [Figure 4]It is a block diagram showing an example of the electrical configuration of an eyeball rotation detection device. [Figure 5] It is a diagram showing a first modification of the arrangement of the neutral electrode 46. [Figure 6] It is a diagram showing a second modification of the arrangement of the neutral electrode 46. [Figure 7] It is a diagram showing a third modification of the arrangement of the neutral electrode 46. [Figure 8] It is a diagram showing an EOG signal in a state where the line of sight is facing forward. [Figure 9] It is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the left from a state where the line of sight is facing forward. [Figure 10] It is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated to the right from a state where the line of sight is facing forward. [Figure 11] It is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction where the convergence angle increases from a state where the line of sight is facing forward. [Figure 12] It is a diagram showing an example of a change in the waveform of an EOG signal when both eyeballs are rotated in a direction where the convergence angle decreases from a state where the line of sight is facing forward. [Figure 13] It is a diagram showing an example of the waveform of an EOG signal for various eye movements. [Figure 14] It is a diagram showing an example of the experimental results for detecting a change in the convergence angle. [Figure 15] It is a view of an example of the glasses-type eyeball rotation detection device according to the second embodiment as seen from the front. [Figure 16] It is a block diagram showing an example of the electrical configuration of a surgical support system including a glasses-type eyeball rotation detection device. [Figure 17] It is a diagram showing an example of the operation of a glasses-type eyeball rotation detection device. [Figure 18] It is a diagram showing another example of the operation of a glasses-type eyeball rotation detection device. [Figure 19] It is a diagram showing an example of the operation of a glasses-type eyeball rotation detection device according to a modification of the second embodiment. [Figure 20]This figure shows an example of the electrical configuration of a system including a spectacle-type eye rotation detection device according to the third embodiment. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. It should be noted that the disclosure is merely an example, and the invention is not limited to the embodiments described below. Modifications that a person skilled in the art could easily conceive are naturally included within the scope of the disclosure. For clarity, the drawings may schematically represent parts with modified sizes, shapes, etc., compared to the actual embodiments. In some cases, the same reference numerals are used for corresponding elements in multiple drawings, and detailed explanations are omitted.

[0010] This explains basic information about the eyeball. The diameter of an adult eyeball is approximately 25 mm. At birth, it is about 17 mm, and it grows larger with growth. The interpupillary distance of an adult male is approximately 65 mm. Therefore, many commercially available stereo cameras are made with a 65 mm interpupillary distance. The interpupillary distance of an adult female is several millimeters shorter than that of a male. The electrooculography (EOV) is several tens of mV. The eyeball has a positive potential on the corneal side and a negative potential on the retinal side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV (called the electrooculography).

[0011] The range of eye rotation (in a typical adult) is less than 50° to the left and less than 50° to the right in the left-right direction (also called the horizontal direction), and less than 50° downwards and less than 30° upwards in the up-down direction (also called the vertical direction). The range of vertical movement that can be controlled voluntarily is narrow upwards. This is because of the "Bell effect," in which the eyeball rotates upwards when the eyes are closed, so when the eyes are closed, the range of vertical eye movement shifts upwards. The convergence angle (the angle at which the lines of sight of the left and right eyeballs intersect) is less than 20°.

[0012] [First Embodiment] Referring to Figures 1, 2, and 3, an example of the configuration of an eye rotation detection device according to an embodiment will be described. Although there are various forms of eye rotation detection devices, here we will show an example of an eye rotation detection device in the form of eyewear. Eyewear can include goggles, glasses (sunglasses are equivalent to glasses), etc., but here we will describe a glasses-type eye rotation detection device. Figure 1 is a front view of an example of a glasses-type eye rotation detection device, and Figure 2 is a rear-upper view of an example of a glasses-type eye rotation detection device. Figure 3 is a view from the right front of a user wearing an example of a glasses-type eye rotation detection device.

[0013] There are two types of eye rotation: up-and-down rotation and left-and-right rotation. Up-and-down rotation includes blinking, closing the eyes, winking, etc. Left-and-right rotation is a slow, unconscious movement in which both eyeballs rotate in the same direction. Eye movement is broadly classified into convergence and divergence, which are conscious rotations of the left and right eyeballs in the same direction and rotations of the left and right eyeballs in opposite directions. Convergence is when the lines of sight of the left and right eyeballs intersect, and divergence is when the lines of sight of the left and right eyeballs diverge. Eye rotation is detected based on changes in electrooculography (EOE). EOE can be detected by the difference in voltage from a pair of electrodes that surround the eyeball. The direction in which the electrodes surround the eyeball can be left and right, up and down, front and back, or even diagonally. Blinking, closing the eyes, winking, etc., can be detected from EOE detected by an electrode pair positioned to surround the eyeball from above and below. Blinking, closing the eyes, winking, slow movements, and line of sight can be detected from EOE detected by an electrode pair positioned to surround the eyeball from above and below and from the left and right. Slow movements, line of sight, and convergence / divergence can be detected from EOE detected by an electrode pair positioned to surround the eyeball from the front and back and from the left and right.

[0014] [Electrode arrangement] The eyeglasses include a right frame 12, a left frame 14, and a bridge 26 connecting both frames 12 and 14. In this specification, "right" and "left" refer to the right and left as seen from the perspective of the user wearing the eyeglasses. In Figure 1, viewed from the front, this is reversed, with the frame on the right in Figure 1 being the left frame 14. If the device is solely for electrooculography detection, lenses or glass do not need to be fitted into the right frame 12 and left frame 14. However, if the user regularly wears eyeglasses, lenses with the user's appropriate prescription may be fitted into the right frame 12 and left frame 14 instead of the user's regular eyeglasses. If the user does not regularly wear eyeglasses, simple glass may be fitted into the right frame 12 and left frame 14. If the device is not solely for electrooculography detection, but also detects gaze movement or convergence angle changes from electrooculography and applies the detection results to a product such as an augmented reality (AR) display-enabled eyeglasses-type wearable device, then at least a portion of the right frame 12 and left frame 14 may be fitted with an AR display-enabled liquid crystal panel or organic EL panel.

[0015] In this embodiment, in order to detect convergence and divergence, electrodes are positioned so as to sandwich the eyeballs from an anterior-posterior position that is in phase (same vector) with respect to each eyeball, and from lateral positions that are in opposite phase (opposite vector) with respect to each eyeball, within the same plane.

[0016] To detect the electrooculography (ER) of the right eyeball, as shown in Figure 2, a right temple electrode 32 is provided on the right side of the right eyeball ER, for example, on the part of the right temple 18 that rests on the ear, and a right nose pad electrode 42 is provided on the left side of the right eyeball ER, for example, on the surface of the right nose pad 22 that is attached near the connection point between the right frame 12 and the bridge 16 and contacts the nose. In the plan view (Figure 2 is considered a plan view), the right temple electrode 32 and the right nose pad electrode 42 are positioned such that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER.

[0017] In the front view (Figure 1 is considered a front view), the right temple electrode 32 is located on the left side of the right eyeball ER, and the right nose pad electrode 42 is located on the right side of the right eyeball ER. The right temple electrode 32 and the right nose pad electrode 42 are positioned such that the line connecting them passes through the right eyeball ER. Also, in the front view, the right nose pad electrode 42 is located slightly above the right temple electrode 32.

[0018] In the side view, the right temple electrode 32 is located on the rear side of the right eyeball ER, i.e., on the left side of the right eyeball ER in the right side view and on the right side of the right eyeball ER in the left side view, and the right nose pad electrode 42 is located on the front side of the right eyeball ER, i.e., on the right side of the right eyeball ER in the right side view and on the left side of the right eyeball ER in the left side view. The right temple electrode 32 and the right nose pad electrode 42 are positioned such that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER.

[0019] Figure 3 shows how the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in the front, top, and side views of the head. Note that the line connecting the two electrodes does not need to pass through the center of the right eyeball ER; it can pass through any part of the eyeball. The same applies to the left eyeball, although it is hidden by the face in Figure 3.

[0020] The right temple electrode 32 and the right nose pad electrode 42, which detect the electrooculography of the right eyeball ER, are positioned such that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in any of the plan view, front view, or side view. However, it is sufficient that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in at least one of the plan view, front view, and side view.

[0021] Similarly, to detect the electrooculography of the left eyeball EL, a left nose pad electrode 44 is provided on the surface of the left nose pad 24 that contacts the nose, which is attached to the right side of the left eyeball EL in the plan view, for example near the connection point between the left frame 14 and the bridge 16. A left temple electrode 36 is provided on the left side of the left eyeball EL, for example on the part of the left temple 20 that rests on the ear. The left nose pad electrode 44 and the left temple electrode 36 are positioned such that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0022] The right temple electrode 32 and the left temple electrode 36 are symmetrical with respect to a line perpendicular to the midpoint of the line connecting the right frame 12 and the left frame 14 (for example, a line extending from the center of the nose to the back of the head).

[0023] In the front view, the left nose pad electrode 44 is located to the left of the left eyeball EL, and the left temple electrode 36 is located to the right of the left eyeball EL. The left nose pad electrode 44 and the left temple electrode 36 are positioned such that the line connecting them passes through the left eyeball EL. Also, in the front view, the left nose pad electrode 44 is located slightly above the second left electrode 42.

[0024] In the side view, the left nose pad electrode 44 is located on the front side of the left eyeball EL, i.e., to the right of the left eyeball EL in the right side view and to the left of the left eyeball EL in the left side view, and the left temple electrode 36 is located on the rear side of the left eyeball EL, i.e., to the left of the left eyeball EL in the right side view and to the right of the left eyeball EL in the left side view. The left nose pad electrode 44 and the left temple electrode 36 are positioned such that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0025] The right temple electrode 32 is provided across the side (contacting the temporal region) and bottom (contacting the base of the ear) of the right temple 18. When the glasses are worn on the face, the weight of the temple 18 causes the right temple electrode 32 to contact the area at the base of the ear where there is no fibrous hair. The left temple electrode 36 is provided across the side (contacting the temporal region) and bottom (contacting the base of the ear) of the left temple 20. When the glasses are worn on the face, the weight of the temple 20 causes the left temple electrode 36 to contact the area at the base of the ear where there is no fibrous hair. As a result, the right temple electrode 32 and the left temple electrode 36 are in close contact with the user's skin and can accurately sense the electrooculography.

[0026] Furthermore, the left nose pad electrode 44 and the left temple electrode 36, which detect the electrooculography of the left eyeball EL, should be positioned such that, in at least one of the plan view, front view, and side view, the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

[0027] A forehead pad 26 is provided inside the bridge 16, and a neutral electrode 46 is provided on the surface of the forehead pad 26 that contacts the forehead. The neutral electrode 46 is an electrode that ensures a neutral potential for detecting electrooculography and is in contact with the skin, for example, the forehead. The neutral electrode 46 is positioned such that the distance between the neutral electrode 46 and the right temple electrode 32 is equal to the distance between the neutral electrode 46 and the left temple electrode 36, and the distance between the neutral electrode 46 and the right nose pad electrode 42 is equal to the distance between the neutral electrode 46 and the left nose pad electrode 44. The reason for positioning the neutral electrode 46 in this way is for the purpose of detecting the convergence angle, which will be described later. The convergence angle is detected based on the results of detecting symmetrical eye rotation when viewed from the front for each eyeball. For example, in an electrocardiograph, the neutral potential is taken at a part of the body where the effect of eye rotation can be ignored, for example, at the end of the right foot. Although it is somewhat affected by eye rotation, by taking a neutral potential at the center of the forehead, a location that is equally affected by both eyeballs, the influence of the electrooculography (OE) on the neutral electrode from each eyeball can be equalized.

[0028] The right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, left temple electrode 36, and neutral electrode 46 are made of metal foil such as copper, metal pieces, metal spheres such as stainless steel, or conductive silicone rubber sheets. These electrodes 32, 42, 44, and 36 are electrodes for detecting electrooculography (EOG), as will be described later, and are therefore also called EOG electrodes.

[0029] [EOG signal] As shown in Figure 2, a processing unit 30 for detecting electrooculography (EOC) is built into or attached to the part of one temple, for example, the right temple 18, near the frame 12. A battery 34 for the processing unit 30 is built into or attached to the part of the other temple, for example, the left temple 20, near the frame 12. The processing unit 30 may not only detect EOC but also, in the case of a glasses-type wearable device capable of AR display, perform display control. The processing unit 30 may not be built into the glasses but may be provided on the outside of the glasses and connected to the glasses wirelessly or by wire. In that case, the battery 34 can be built into the processing unit 30 and provided on the outside of the glasses. Alternatively, the functions of the processing unit 30 may be divided into two parts, with only the first processing unit that senses signals from electrodes provided in the glasses, and the second processing unit that detects EOC from the sense signals and controls according to the detection results provided on the outside of the glasses. A mobile terminal such as a smartphone can be used as the second processing unit. The second processing unit is not limited to a mobile terminal directly connected to the glasses, but also includes a server connected via a network.

[0030] The signal from the right temple electrode 32 is input to the - terminal of the first analog / digital (A / D) converter 62, and the signal from the second left electrode 36 is input to the + terminal of the first A / D converter 62, and the difference signal, the first EOG signal ADC Ch0, is output. Since the right temple electrode 32 and the left temple electrode 36 sandwich the eyeball from the left and right, the first EOG signal ADC Ch0 indicates the left and right rotation of the left and right eyeballs.

[0031] The signal from the right temple electrode 32 is input to the - terminal of the second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, and the difference signal, the second EOG signal ADC Ch1, is output. Since the right temple electrode 32 and the right nose pad electrode 42 sandwich the right eyeball from above, below, and left and right, the second EOG signal ADC Ch1 indicates the left-right rotation and up-down rotation of the right eyeball.

[0032] The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, and the signal from the left temple electrode 36 is input to the - terminal of the second A / D converter 66, and the difference signal, the third EOG signal ADC Ch2, is output. Since the left nose pad electrode 44 and the left temple electrode 36 sandwich the left eyeball from above, below, and left and right, the third EOG signal ADC Ch2 indicates the left-right rotation and up-down rotation of the left eyeball.

[0033] Since the left-right positions of the two electrodes for the second EOG signal ADC Ch1 and the left-right positions of the two electrodes for the third EOG signal ADC Ch2 are opposite (the + / - inputs of the A / D converter are inverted), it is possible to detect from the waveforms of the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 whether the left and right eyeballs are rotating left and right in the same direction or in opposite directions.

[0034] The voltage signals from the right temple electrode 32, right nose pad electrode 42, left nose pad electrode 44, and left temple electrode 36 are weak and therefore highly susceptible to noise. To cancel this noise, a series circuit of resistors R1 and R2 is connected between the reference analog voltage Vcc (=3.3V or 5.5V) of the A / D converters 62, 64, and 66 and ground (GND), and the neutral electrode 46 is connected to the connection point of resistors R1 and R2. Resistors R1 and R2 have equal values, for example, 1MΩ. The A / D converters 62, 64, and 66 can detect analog voltages from 0V (ground) to the reference analog voltage Vcc, and convert the input analog voltage into a digital value in the range from 0V to 3.3V, centered around the midpoint of the detectable range, for example, half of 3.3V (referred to as the midpoint voltage). Since the connection point of resistors R1 and R2 is connected to the midpoint voltage terminal, and the neutral electrode 46 is connected to the connection point of resistors R1 and R2, the midpoint voltage of A / D converters 62, 64, and 66 becomes the same as the voltage of the human body. As a result, the midpoint voltage of A / D converters 62, 64, and 66 fluctuates in conjunction with the voltage of the human body, and noise mixed into the voltage signals from EOG electrodes 32, 42, 44, and 36 is not mixed into the digital values ​​that are the outputs of A / D converters 62, 64, and 66. This improves the signal-to-noise ratio of electrooculography detection.

[0035] Figure 4 is a block diagram showing an example of the electrical configuration of an eye rotation detection device. The processing unit 30 may include A / D converters 62, 64, and 66, or the A / D converters 62, 64, and 66 may be externally connected to the processing unit 30.

[0036] The signal from the right temple electrode 32 is input to the - terminal of the first A / D converter 62, and the signal from the left temple electrode 36 is input to the + terminal of the first A / D converter 62, obtaining the EOG signal ADC Ch0 for the first channel. The signal from the right temple electrode 32 is input to the - terminal of the second A / D converter 64, and the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, obtaining the EOG signal ADC Ch1 for the second channel. The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, and the signal from the left temple electrode 36 is input to the - terminal of the second A / D converter 66, obtaining the EOG signal ADC Ch2 for the third channel.

[0037] The signal from the neutral electrode 46 is supplied to the midpoint voltage terminals of the A / D converters 62, 64, and 66, and the midpoint voltages of the A / D converters 62, 64, and 66 are considered to be the voltage of the human body detected by the neutral electrode 46.

[0038] EOG signals output from A / D converters 62, 64, and 66 are input to an eye movement detection unit 75 that detects eye rotation (hereinafter sometimes referred to as eye movement). The eye movement detection unit 75 may be composed of hardware or software. In the latter case, the CPU 74, ROM 76, and RAM 78 are connected to a bus line, and the eye movement detection unit 75 is also connected to the bus line. The eye movement detection unit 75 is realized by the CPU 74 executing a program stored in ROM 76. A wireless LAN device 80 is also connected to the bus line, and the processing unit 30 is connected to a mobile terminal 84 such as a smartphone via the wireless LAN device 80. The mobile terminal 84 may be connected to a server 88 via a network 86 such as the Internet. The eye movement detection unit 75 detects electrooculography based on the EOG signals output from A / D converters 62 and 64, and can detect left-right rotation (convergence and divergence) of the left and right eyeballs, as well as left-right rotation (gaze movement) and up-down rotation (blinking, eye closing) of the eyeballs from the detected electrooculography. Furthermore, the eye movement detection unit 75 can estimate various user states from the detected eye movements (for example, a state of lacking concentration and being restless, a state of concentration, a state of tension and mental stress, or a state of fatigue making it difficult to concentrate on work or tasks). The type of eye rotation detected and the type of state estimated can be changed by changing the program executed by the CPU 74. This change instruction may also be given from the mobile terminal 84.

[0039] Instead of Wi-Fi device 80, use ZigBee® and Bluetooth. Communication devices using communication methods such as Low Energy® and Wi-Fi® may be used. The detection results of the eye movement detection unit 75 (eye movement detection results, state estimation results) may be temporarily stored in RAM 78 and then sent to the mobile terminal 84 via communication devices such as a wireless LAN device 80. Alternatively, the detection results of the eye movement detection unit 75 may be sent to the mobile terminal 84 in real time. The mobile terminal 84 may store the detection results of the eye movement detection unit 75 in its built-in memory (not shown), or it may transfer the detection results to the server 88 via the network 86. The mobile terminal 84 may start some processing in response to the detection results of the eye movement detection unit 75, store the processing results in its built-in memory, or transfer the processing results to the server 88 via the network 86. The server 88 may aggregate the detection results from many eye movement detection units 75 and the processing results from many mobile terminals 84 and perform so-called big data analysis.

[0040] [Variations in electrode placement] Figures 5, 6, and 7 show modified arrangements of the neutral electrode 46. In the above description, separate nose pads 22 and 24 are provided on the left and right sides, but in the modified arrangement shown in Figure 5, a single, integrated inverted V-shaped or inverted U-shaped nose pad 52 is provided. The right nose pad electrode 42 is provided on the right inner side of the open sides of the nose pad 52, the left nose pad electrode 44 is provided on the left inner side, and the neutral electrode 46 is provided on the inside of the apex of the V-shape or U-shape. This makes it possible to provide a neutral electrode 46 that contacts the forehead without providing a forehead pad 26.

[0041] In the modified example shown in Figure 6, a left and right integrated V-shaped or U-shaped nose pad 54 is provided. The difference is that the nose pad 52 widens downwards, while the nose pad 54 widens towards the front. A right nose pad electrode 42 is provided on the right side of the nose pad 54, a left nose pad electrode 44 is provided on the left side, and a neutral electrode 46 is provided in the center.

[0042] In the modified versions shown in Figures 5 and 6, a larger nose pad than a normal nose pad is used, so even when wearing glasses-type electrooculography detection devices or AR-enabled glasses-type wearable terminals for extended periods, the weight is less likely to cause nose pain.

[0043] In the modified example shown in Figure 7, separate left and right nose pads 22 and 24 are used, but the forehead pad 26 is unnecessary. Here, the right nose pad electrode 42 and the right neutral electrode 46a are provided on the surface of the right nose pad 22 that contacts the nose, and the left nose pad electrode 44 and the left neutral electrode 46b are provided on the surface of the left nose pad 24 that contacts the nose. The right neutral electrode 46a and the left neutral electrode 46b are electrically short-circuited and become equivalent to a single neutral electrode 46.

[0044] [Relationship between eye movement and EOG signal] Referring to Figures 8 to 12, an example of the waveform changes of the EOG signal ADC Ch0 output from A / D converter 62, the EOG signal ADC Ch1 output from A / D converter 64, and the EOG signal ADC Ch2 output from A / D converter 66 are shown when the right and left eyeballs are rotated left and right from a state where the line of sight is facing forward.

[0045] Figure 8 shows the state where the user's gaze direction is straight ahead. When looking at infinity, the gaze directions of the right and left eyes are parallel, but when looking at a finite far point, the gaze directions of the right and left eyes intersect at the far point. From this state, as shown in Figure 9, when both the right eyeball ER and the left eyeball EL rotate to the left (the gaze directions of the right and left eyes move to the left), the positively charged cornea of ​​the right eyeball ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. If both eyeballs rotate to the right in this state, the state returns to that shown in Figure 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected is a convex waveform (a waveform that is convex upwards). The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and right temple electrode 32 are connected is a convex waveform (a waveform that is convex upwards). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and left temple electrode 36 are connected is a concave waveform (a waveform that is convex downwards).

[0046] As the left and right eyeballs rotate in the same direction (counterclockwise), the second EOG signal ADC In Ch1 and the third EOG signal ADC Ch2, an EOG signal with the opposite phase appears. In the first EOG signal ADC Ch0, an EOG signal that is in phase with the second EOG signal ADC Ch1, which has the same + / - relationship, appears.

[0047] As shown in Figure 8, when the line of sight is facing forward, and then as shown in Figure 10, both the right eyeball ER and the left eyeball EL rotate to the right (the line of sight of both the right and left eyeballs moves to the right), the positively charged cornea of ​​the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. If both eyeballs rotate to the left in this state, the state returns to that shown in Figure 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected becomes a concave waveform (a downward-convex waveform). The second EOG signal ADC Ch1, output from the second A / D converter 64 to which the right nose pad electrode 42 and right temple electrode 32 are connected, is a concave waveform (a waveform that is convex downwards). The third EOG signal ADC Ch2, output from the third A / D converter 66 to which the left nose pad electrode 44 and left temple electrode 36 are connected, is a convex waveform (a waveform that is convex upwards).

[0048] As the left and right eyeballs rotate in the same direction (clockwise), the second EOG signal ADC In Ch1 and the third EOG signal ADC Ch2, opposite-phase EOG signals appear. However, these are opposite-phase for the case where both the right eyeball ER and the left eyeball EL rotate to the left. In the first EOG signal ADC Ch0, an EOG signal appears that is in phase with the second EOG signal ADC Ch1, which has the same + / - relationship. However, the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate to the right is opposite-phase to the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate to the left.

[0049] As shown in Figure 8, the line of sight is directed straight ahead. When the right eyeball ER rotates to the left (the line of sight of the right eyeball moves to the left) and the left eyeball EL rotates to the right (the line of sight of the left eyeball moves to the right), as shown in Figure 11, convergence occurs where the lines of sight of the left and right eyeballs intersect, resulting in what is known as crossed eyes. In this state, the positively charged cornea of ​​the right eyeball ER approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. If the right eyeball rotates to the right and the left eyeball rotates to the right in this state, the state returns to that shown in Figure 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62, to which the right and left temple electrodes 32 and 36 are connected, remains unchanged, and neither a convex nor a concave waveform appears. The second EOG signal ADC Ch1, output from the second A / D converter 64 to which the right nose pad electrode 42 and right temple electrode 32 are connected, is a convex waveform (a waveform that is convex upwards). The third EOG signal ADC Ch2, output from the third A / D converter 66 to which the left nose pad electrode 44 and left temple electrode 36 are connected, is a convex waveform (a waveform that is convex upwards).

[0050] Because the left and right eyeballs rotate in opposite directions, the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 exhibit waveforms in phase.

[0051] If the electrooculi potentials of the left and right eyeballs are identical, and the absolute value of the rotation angle is also identical, then both the + terminal and - terminal of the A / D converter 62 change by the same amount in the same direction (negative direction). As a result, no change is observed in the relative values ​​of the two, and no change in electrooculi potential appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrode and the temple electrode is slightly offset from the center of the left and right eyeballs, so slight changes appear according to the amount of these offsets.

[0052] As shown in Figure 8, the line of sight is directed straight ahead. However, as shown in Figure 12, when the right eyeball ER rotates to the right (the line of sight of the right eyeball moves to the right) and the left eyeball EL rotates to the left (the line of sight of the left eyeball moves to the left), divergence occurs where the lines of sight of the left and right eyeballs diverge, resulting in what is known as wide-set eyes. In this state, the positively charged cornea of ​​the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of ​​the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. If the right eyeball rotates to the left and the left eyeball rotates to the left in this state, the state returns to that shown in Figure 8. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62, to which the right and left temple electrodes 32 and 36 are connected, remains unchanged, and neither a convex nor concave waveform appears. The second EOG signal ADC Ch1, output from the second A / D converter 64 to which the right nose pad electrode 42 and right temple electrode 32 are connected, is a concave waveform (a downward-convex waveform). The third EOG signal ADC Ch2, output from the third A / D converter 66 to which the left nose pad electrode 44 and left temple electrode 36 are connected, is a concave waveform (a downward-convex waveform). In this way, the left and right eyeballs rotate in opposite directions, so the second EOG signal ADC In-phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. However, in the case of wide-set eyes, the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 are out of phase with respect to the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 in the case of close-set eyes.

[0053] If the electrooculi potentials of the left and right eyeballs are identical, and the absolute value of the rotation angle is also identical, then both the + terminal and - terminal of the A / D converter 62 change by the same amount in the same direction (positive direction). As a result, no change is observed in the relative values ​​of the two, and no change in electrooculi potential appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrode and the temple electrode is slightly offset from the center of the left and right eyeballs, so slight changes appear according to the amount of these offsets.

[0054] Figure 13 is an electrooculogram (EOG) illustrating an example of the relationship between various user eye movements and the EOG signals ADC Ch0, ADC Ch1, and ADC Ch2 obtained from A / D converters 62, 64, and 66. The vertical axis shows the sample values ​​of A / D converters 62, 64, and 66 (for example, 3.3V, 24-bit A / D converters), and the horizontal axis shows time.

[0055] As shown in Figure 11, when neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and convex waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, the eye movement detection unit 75 detects a "crossed-eye" state in which the lines of sight of the left and right eyeballs converge. Although not shown in Figure 13, as shown in Figure 12, when neither a convex nor concave waveform appears in the EOG signal ADC Ch0, and concave waveforms appear in the EOG signals ADC Ch1 and ADC Ch2, the eye movement detection unit 75 detects a "wide-set-eye" state in which the lines of sight of the left and right eyeballs diverge. Thus, convergence and divergence differ only in the convexity of the waveforms of the EOG signals ADC Ch1 and ADC Ch2, and are otherwise the same, so in the following explanation, convergence and divergence may be collectively referred to as convergence. The amplitude of the waveforms of the EOG signals ADC Ch1 and ADC Ch2 corresponds to the degree of convergence (convergence angle) and the degree of divergence. As will be discussed later with reference to Figure 14, the degree of amplitude change is greater at shorter distances and decreases with increasing distance. Therefore, the sensitivity for detecting amplitude changes is higher at shorter distances and decreases with increasing distance.

[0056] As shown in Figure 9, the eye movement detection unit 75 detects leftward movement in the direction of gaze when a convex waveform appears in EOG signal ADC Ch0, a convex waveform appears in EOG signal ADC Ch1, and a concave waveform appears in EOG signal ADC Ch2.

[0057] As shown in Figure 10, the eye movement detection unit 75 detects a rightward movement in the direction of gaze when a concave waveform appears in the EOG signal ADC Ch0, a concave waveform appears in the EOG signal ADC Ch1, and a convex waveform appears in the EOG signal ADC Ch2.

[0058] In the EOG signal ADC Ch1 and EOG signal ADC Ch2, one blink (1), two blinks (2), and three blinks (3) are detected by the in-phase convex pulse waveforms of waves 1 to 3 that momentarily rise in level and then return to their original level. In the EOG signal ADC Ch1 and EOG signal ADC Ch2, the eye movement detection unit 75 detects vertical eye rotation, i.e., eye closure, by the combination waveform of the convex waveform (upward-convex waveform) when the gaze is directed upward and the concave waveform (downward-convex waveform) when the gaze is directed downward. Thus, eye rotation caused by vertical blinking and eye closure can be detected based on either the EOG signal ADC Ch1 or ADC Ch2, and in applications that only detect blinking and eye closure, it is not necessary to provide an electrode pair for each of the left and right eyeballs; an electrode pair may be provided for only one of the eyeballs.

[0059] An example of experimental results for detecting changes in the convergence state according to the embodiment is shown. Using the prototype electrooculography detection device shown in Figures 1 to 4, Figure 14 shows an example of the change in the second EOG signal ADC Ch1 when the position of the intersection of the line of sight directions of the left and right eyes changes from looking at a fingertip 10 cm in front of the nose to looking at a marker further away. In this case, since the intersection of the line of sight directions changes from near to far, the convergence angle decreases. The horizontal axis is the distance the intersection of the line of sight directions moves in the depth direction from 10 cm away, and the first plot shows the EOG amplitude when the intersection of the line of sight directions moves from 10 cm away to 20 cm away. Note that 10 cm is the shortest distance at which stable gazing is possible. The minimum detection voltage of the EOG signal is set to 50 μV. That is, the eye movement detection unit 75 can detect a change in the EOG signal when the EOG signal changes by 50 μV or more, and based on that, it can detect a change in the position of the intersection of the line of sight directions, i.e., a change in the convergence angle, and if the EOG signal does not change by 50 μV or more, it cannot detect a change in the EOG signal. Note that if the average value obtained by integrating the measured values ​​is used, the minimum detectable voltage of the EOG signal will be a lower voltage, but here it was set to 50 μV. Note that the amplitude of the EOG signal depends on the contact resistance of the electrodes; if an electrode material with low contact resistance is used, the amplitude of the EOG signal will increase, and the minimum detectable voltage of the EOG signal will increase.

[0060] For example, if the convergence angle decreases so that the subject is looking at a marker 30 cm in front of their nose, and then looks at a more distant marker, the eye movement detection unit 75 detects a change in EOG amplitude when the EOG amplitude increases by 50 μV. The EOG amplitude obtained by adding 50 μV to the EOG amplitude when looking at a marker 30 cm away corresponds to 40 cm away. In other words, if the convergence angle decreases so that the subject is looking at a marker 40 cm away, the eye movement detection unit 75 can detect a change in EOG amplitude. The fact that a change in EOG amplitude corresponding to a decrease in convergence angle corresponding to a 10 cm change indicates that the detection resolution is quite good. From Figure 14, it can be seen that a change in EOG amplitude can be detected when the convergence angle decreases so that the subject is looking at a marker 65 cm or further away, and a change in EOG amplitude can be detected when the convergence angle decreases so that the subject is looking at a marker 1.4 m or further away, from a state where the subject is looking at a marker 70 cm away.

[0061] As described above, the first embodiment provides a spectacle-type ocular rotation detection device that detects convergence by independently detecting the left-right rotation of the right and left eyeballs, comprising right and left temple electrodes, right and left nose pad electrodes, and neutral electrodes arranged to equally receive the effects of the rotation of the right and left eyeballs. Since the neutral potential from the neutral electrode is used as the midpoint potential of the A / D converter that samples the EOG signal from the electrode, the EOG signal from the electrode is not affected by noise, and the electrooculography is accurately detected, thus enabling accurate detection of ocular rotation. Furthermore, according to the first embodiment, left-right rotation of both eyeballs (left-right movement in the direction of gaze) and up-down rotation of the eyeballs can also be detected.

[0062] According to the electrooculography detection device of the first embodiment, convergence, which is the conscious rotation of the subject's eyeballs, can be detected. By performing control according to the detection result, it is possible to realize applications that allow control according to the subject's intentions. For example, in eyewear capable of displaying augmented reality (AR), the AR display can be turned on / off or the display position of the AR image can be adjusted hands-free in response to the detection of convergence.

[0063] Furthermore, certain tasks may require task-specific congestion changes, and by comparing the user's congestion change patterns with a reference pattern, it is possible to determine the user's proficiency with the task and whether or not they are performing the task correctly.

[0064] The following describes examples of applications of the convergence detection results. One application example is to incorporate an electrooculography detection device into a glasses-type wearable device capable of AR display, and control the AR display based on convergence detection. For example, convergence detection can be used as a function switching switch. For example, as shown in Figure 8, the AR display can be turned off when the user is looking at something far ahead, and as shown in Figure 11, the display can be turned on and off so that the AR display is activated when the user changes the direction of their gaze to look at something close (convergence state). Since the display position of the AR image is set to a close distance, the AR display continues when the user is looking at the AR image because they are in a converged state, and the AR display is turned off when the user changes the direction of their gaze to look at something far away. This makes it possible to control the AR display as intended by the user. The following describes a second embodiment, a glasses-type wearable device for surgical assistance, which is an application example of this.

[0065] [Second Embodiment] Figure 15 is a front view of an example of a glasses-type electrooculography detection device 100 according to the second embodiment. The difference from the detection device of the first embodiment is that displays for AR display (e.g., organic EL panel or liquid crystal panel) 102 and 104 are fitted into at least a part of the right frame 12 and the left frame 14. If the AR image is not to be displayed in 3D, the display 102 or 104 does not need to be fitted into either the right frame 12 or the left frame 14. Here, it is assumed that the AR image is displayed in 3D. The arrangement of the EOG electrodes is the same as in the first embodiment. There are various applications of the AR-displaying glasses-type electrooculography detection device 100, but as an example of the second embodiment, a surgical support system will be described. The surgeon performing the surgery and the staff involved in the surgery wear the glasses-type electrooculography detection device 100.

[0066] Figure 16 is a block diagram showing an example of the electrical configuration of a surgical support system including a glasses-type electrooculography detection device 100. The processing unit 30 has a display controller 112 added to the configuration of the first embodiment shown in Figure 4. The display controller 112 controls the AR display on displays 102 and 104. Control of the AR display includes on / off control of the AR display, control of the display position of the AR image (convergence angle control of the AR image), etc.

[0067] In a surgical support system, multiple doctors and staff are involved in the same surgery, and multiple pairs of glasses are used. Therefore, it is preferable that the glasses processing unit 30 be connected to a control device 120, such as a personal computer, which is more powerful than a mobile terminal 82 such as a smartphone. Although not shown, multiple glasses processing units 30 are connected to the control device 120. The control device 120 includes a vital data memory 124, an assistive image memory 122, and an assistive information memory 126. An eye movement detection unit 75 may also be provided within the control device 120.

[0068] A vital data measurement unit 127 is connected to the control device 120, and the patient's electrocardiogram, blood pressure, pulse rate, cumulative blood transfusion volume, etc., are measured and stored for each patient in the vital data memory 124 within the control device 120. The support image memory 122 stores support images for surgical assistance. If necessary, support images from the surgical assistance database 130 in the server 88 are downloaded to the control device 120 and stored in the support image memory 122. The support information memory 126 stores support text for surgical assistance. If necessary, support text from the surgical assistance database 130 in the server 88 is downloaded to the control device 120 and stored in the support information memory 126.

[0069] Referring to Figures 17 and 18, an example of the operation of the glasses-type electrooculography detection device 100 will be explained. Here, it is assumed that the distance from the eyeball to the affected area (real world) during surgery is approximately 40 cm. When the doctor is looking at a point in front of the affected area (real world) during surgery, for example, at a distance of approximately 30 cm from the eyeball, the eye movement detection unit 75 detects convergence. As shown in Figure 11, the eye movement detection unit 75 detects convergence based on the fact that the waveform of the EOG signal ADC Ch0 does not change and that the waveforms of the EOG signals ADC Ch1 and EOG signals ADC Ch2 become upwardly convex waveforms. When the eye movement detection unit 75 detects convergence, it requests an AR image related to surgical support from the control device 120 and causes the display controller 112 to perform AR display. An example of AR display is to overlay a vital data window (AR image) semi-transparently onto the affected area (real world) during surgery, as shown in Figure 17(a). The window contains multiple pages, each page displaying an electrocardiogram, blood pressure, pulse rate, cumulative blood transfusion volume, etc. To prevent the affected area from being obscured, the window is limited to a portion of the screen. In response to a request from the processing unit 30, the control device 120 reads the patient's vital data from the vital data memory 124 and transfers it to the processing unit 30.

[0070] When AR images are displayed in 3D, the display position can be arbitrarily set by adjusting the convergence angle of the left and right images. In this case, the display position of the vital data window is set to approximately 30 cm in front, which is the same distance at which the eye movement detection unit 75 detects convergence. Therefore, when the vital data window is displayed, the physician is looking at a point at the same distance as the display position, allowing them to instantly check the contents of the window. Furthermore, there is no need to rotate the eyeballs to adjust the convergence angle in order to fixate on the window, thus preventing eye strain.

[0071] The page switching of the vital data window may be performed automatically at regular intervals, for example, every second, or it may be performed at the user's discretion based on the rotation of the eyeball in another direction detected by the eye movement detection unit 75. For example, if eye closure for 0.5 seconds or more is detected, the window page may be switched. Furthermore, the window page may be switched based on the direction of gaze movement. For example, if the gaze moves to the right, the page may be switched to the next page, and if the gaze moves to the left, the page may be switched to the previous page. Figure 17(b) shows an example of the window page being switched.

[0072] In the state shown in Figure 17(a) or Figure 17(b), when the doctor or other medical professional changes their gaze direction to look at the affected area during surgery (the real world) (at a distance greater than approximately 40 cm), the eye movement detection unit 75 detects that the convergence angle decreases (the distance to the intersection of the gaze directions of both eyes increases) and instructs the display controller 112 to stop the AR display. As a result, the doctor or other medical professional observes only the affected area during surgery through the right frame 12 and left frame 14 of the glasses, as shown in Figure 17(c).

[0073] In the state shown in Figure 17(c), if a doctor or other medical professional wants to refer to vital data, they change their gaze direction to look at something close up (approximately 30 cm). The eye movement detection unit 75 detects that the convergence angle increases (the distance to the intersection of the gaze directions of both eyes becomes shorter), and causes the display controller 112 to perform AR display, displaying a vital data window as shown in Figure 17(a) or (b).

[0074] The increase / decrease in the convergence angle can be determined based on the relationship between the EOG amplitude and the distance traveled at the intersection of the lines of sight, as shown in Figure 14. If the EOG voltage drops / rises above a certain voltage, it can be determined that the convergence angle has increased / decreased.

[0075] In this way, doctors can display or turn off AR images that assist in surgery hands-free during the procedure. Since their hands are occupied during surgery, being able to switch the AR display on and off hands-free is highly beneficial.

[0076] Another example of AR display is support images, as shown in Figure 18(a). Support images may be images of past surgeries on other patients with similar symptoms, or images from other surgeries on the patient in question. Images may be still images or videos. For this reason, images are captured during the surgery by a camera (not shown), uploaded to server 88, and stored in the surgical support database 130. Another example of AR display is support information, as shown in Figure 18(b). Support information consists of various text data related to the surgery being performed.

[0077] The display position of the support information is set close to the vital data window. However, the display position of the support image may be set at the same distance as the affected area (in the real world) during surgery, approximately 40 cm. The vital data window and support information are not viewed simultaneously with the affected area during surgery, but the support image is expected to be compared side by side with the affected area. If the display positions are different when comparing the affected area and the support image, the eyes will need to rotate left and right to compare them, which may cause eye strain. Therefore, contrary to the explanation above, the support image is displayed when the eye is in the state shown in Figure 8, and is turned off when convergence is detected as shown in Figure 11. Note that if the position of the support image and the affected area (in the real world) during surgery are slightly misaligned, the convergence angle detected by the eye movement detection unit 75 will change slightly when comparing them. If the display controller 112 adjusts the display position of the support image (convergence angle of the left and right images) so that this change is small, the possibility of eye strain can be further reduced.

[0078] Switching from the vital data window to support images and support information may be performed at the user's discretion based on the rotation of the eyeball in other directions by the eye movement detection unit 75, similar to page switching in the vital data window.

[0079] In the example above, changes in the convergence angle were used to switch the AR display on and off, but the detected convergence angle may be used for other control, and the switching of the AR display on and off may be based on other eye movements. For example, the AR display may be switched on and off when multiple eye closures are detected. Furthermore, the detected convergence angle may be used to control the display position of the AR image. That is, the distance between the left and right images of the AR image (sometimes called the image convergence angle) is adjusted based on the distance to the intersection of the line of sight directions of the left and right eyeballs at the start of AR image display. The image convergence angle is adjusted by the display controller 112. As a result, there is no need to rotate the eyeballs left and right to fixate on the AR display, so eye strain does not occur when viewing the AR display.

[0080] Other application examples of convergence angle changes and AR display control are illustrated with reference to Figure 19. This example is a picking operation in a warehouse, where the worker wears a glasses-type electrooculography detector 100 during the picking process.

[0081] The configuration of the processing unit 30 is the same as in Figure 16, so it is omitted from the illustration. The configuration of the control device 120 is the same except that vital data, support images, and support information are changed to a pickup list, so it is omitted from the illustration. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used, as in the first embodiment shown in Figure 4. The mobile terminal 84 may acquire location information. The configuration of the server 88 is also the same except that surgical support data is changed to map information regarding the location of shelves in the warehouse, a list of products on each shelf in the warehouse, and a list of products to be picked up, so it is omitted from the illustration. A pickup list for each worker is downloaded from the server 88 to the control device 120.

[0082] In this application example, the AR display is initially turned off, and the displays on the right frame 12 and left frame 14 of the glasses are transparent and do not show anything. Therefore, the worker looks at the warehouse as shown in Figure 19(a) through the right frame 12 and left frame 14 of the glasses. If the mobile terminal 84, which acts as the control device 120, can acquire location information, AR displays of air tags may be shown on the products on each shelf based on the map information in the server 88 and the acquired location information. At this time, the worker is looking several meters or even tens of meters or more ahead.

[0083] The worker rotates their left and right eyeballs in opposite directions to the right, causing convergence, or crossing of the eyes, as shown in Figure 11, and the eye movement detection unit 75 detects this convergence.

[0084] In response to the detection of congestion, the processing unit 30 requests a pickup list from the control device 120 showing the items to be picked up by the worker, and instructs the display controller 112 to start AR display. An example of a pickup list is shown in Figure 19(b). The display position of the pickup list is also set to be close. When the left and right eyeballs are rotated from looking at the pickup list to looking at the distant warehouse as shown in Figure 8, the eye movement detection unit 75 no longer detects congestion, and therefore instructs the display controller 112 to stop displaying the pickup list.

[0085] Since the pickup process, like surgery, requires hands-free operation, the ability to switch the AR display on and off hands-free is extremely beneficial.

[0086] In Figure 19, the air tag is displayed at all times, but it is also possible to display the air tag only when the user is looking at a distant target and turn it off when looking at a nearby target. For example, when a worker is going to perform maintenance on elevators in a building, and is looking for the target building on the way, the worker is looking at a distant target, so the air tag containing the building's name, distance, work purpose, etc., is displayed overlaid on the target building. At this point, if the worker looks at their device to check a map or something similar, the air tag display may be turned off. When the worker arrives at the destination and is in front of the elevator, if they are looking at the area to be inspected, the air tag will be displayed at the inspection point. If the worker looks at their device, the work manual or something similar will be displayed in AR instead of the air tag. This example can also be applied to tourist information. For example, when viewing scenery at a tourist spot, an air tag related to that spot may be displayed, and when the worker looks at their device, the air tag may be turned off and a detailed tourist guide may be displayed instead.

[0087] Another example of AR display using convergence detection is the integration of an electrooculography (EOG) detection device into AR glasses used for watching sports. The game is filmed from various angles and stored on a server as replay footage. The AR glasses connect to the server via a network. Spectators are normally focused on the game, looking at distant targets. At this time, the AR display is turned off. Spectators in the outfield seats of a baseball stadium can rotate their eyeballs to cross their eyes and activate the AR display when they want to see a magnified image of a play at home plate. In the AR display, various replay images are downloaded from the server and displayed. This allows spectators to instantly enjoy replay images by rotating their eyeballs.

[0088] In the above explanation, AR display was started when congestion was detected and stopped when congestion was no longer detected. However, the reverse is also possible. That is, AR display may be performed when no congestion is detected, and then stopped when congestion is detected.

[0089] [Third Embodiment] Another application of convergence detection is the verification of work procedures. Figure 20 is a block diagram showing an example of the electrical configuration of a work verification system including a glasses-type electrooculography detection device. While AR display is not essential for work verification, it may be used to immediately communicate the verification results to the worker. The processing unit 30 is the same as in the second embodiment shown in Figure 16. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used, as in the first embodiment shown in Figure 4. The control device 120 includes a gaze movement discrimination unit 148 and a gaze movement pattern memory 142. The gaze movement discrimination unit 148 discriminates gaze movement (rightward gaze direction, leftward gaze direction, gaze direction crossing (convergence)) from the eye rotation change pattern detected by the eye movement detection unit 75 and stores the gaze movement pattern in the gaze movement pattern memory 142. The data in the gaze movement pattern memory 142 is uploaded to the server 88 via the network 86.

[0090] Server 88 includes a standard eye movement pattern memory 144 and a work procedure determination unit 146. Some tasks require task-specific eye movement. For example, in the railway industry, pointing and confirming after departure requires checking distant and nearby targets in a predetermined order. Also, in visual inspections of structures such as bridges and tunnels, the areas to be inspected are determined, so the eye movement direction is required to move in a predetermined pattern. The standard pattern memory 144 stores these task-specific standard patterns of eye movement direction. The work procedure determination unit 146 compares the eye movement pattern of the worker uploaded from the control device 120 with the standard eye movement pattern in the standard pattern memory 144 and can determine whether the worker is performing the task correctly. This may be stored for each worker in a database (not shown). The worker's proficiency can be estimated by the change in the determination result over time. Alternatively, if the work procedure determination unit 146 determines that the worker is not performing the work correctly, it may notify the display controller 112 of the processing unit 30 via the control device 120, causing a warning message to be displayed on the displays 102 and 104.

[0091] Another example of a standard eye movement pattern is one related to driving a car. During driver's license renewal training, videos of excellent drivers performing safety checks are sometimes shown. In this case, instead of simply watching, it would be beneficial to compare the eye movement patterns of experienced drivers when performing the same safety checks to assess the level of proficiency in safety checks. In the transportation industry, accidents can be reduced if new drivers learn the same eye movement patterns as experienced drivers.

[0092] The above explanation applies not only to eyeglasses but also to goggles. For example, electrodes could be placed on a part of the foam on the front of the goggles instead of the nose pads, or on the strap instead of the temples.

[0093] [calibration] Electrooculography (ECO) does not provide a specific angle of convergence, only the direction of eye rotation at which convergence occurs. Therefore, in the above embodiment, convergence is detected, and in the application example, control is performed based on a binary state of whether or not convergence is detected, regardless of the convergence angle itself. However, if there are multiple targets of known distance, by looking at multiple targets and periodically measuring EOC for multiple distances, the electrooculogram values ​​for multiple known distances can be determined, and three or more convergence states can be identified. For example, the AR display may be turned off when looking at a distant target, the first AR image may be displayed when looking at a medium-range target, and the second AR image may be displayed when looking at a close-range target.

[0094] Furthermore, because the contact resistance of the electrodes changes (decreases over time), the absolute value of the electrooculogram (EOC) relative to distance changes over time, and the absolute value of distance cannot be guaranteed in the long term. For example, immediately after starting to use the electrodes, the contact resistance is high, so the amplitude of the EOC is small, and as time passes, the resistance decreases and the amplitude of the EOC increases. However, by periodically calibrating the EOC relative to distance, it is possible to estimate the change in EOC over time and guarantee the absolute value of the EOC relative to distance. For example, in tasks where the focal length is constant, such as surgery or desk work, or in tasks where it is guaranteed that the user is looking at a target object at a known distance (level) at a certain point in the work process, it is possible to perform periodic calibration while the distance is known.

[0095] Furthermore, the electrical potential of the human body itself can fluctuate, but this can also be compensated for by periodic calibration.

[0096] [Summary of the effects of the embodiment] According to the embodiment described above, the left-right rotation of the right eyeball and the left eyeball can be detected independently, making it possible to detect convergence, which is the intersection of the lines of sight of the left and right eyeballs. Convergence does not occur unconsciously, but is caused by the subject's conscious eye rotation. Therefore, by performing control in response to the detection of convergence, hands-free control can be performed according to the subject's intentions. For example, in a glasses-type wearable terminal that displays AR, the display of the AR image can be started by crossing the eyes, and the display of the AR image can be stopped by looking at a distant target object.

[0097] AR displays can be monocular or binocular. This embodiment can also be applied to monocular displays, but in the case of binocular displays, detecting the convergence of the left and right eyeballs is effective. In 3D display of AR images using both eyes, the distance between the left and right images (also called the convergence angle) can be set arbitrarily. However, if the convergence angles of the left and right eyeballs when viewing the real world differ from the convergence angles of the AR image, the user must adjust the convergence angles of the left and right eyeballs each time they compare the real world with the AR image, which can cause eye strain. However, if the convergence angles of the left and right images in the AR image are set to match the convergence angles of the left and right eyeballs, such rotation of the left and right eyeballs to adjust the convergence angle becomes unnecessary, and eye strain does not occur.

[0098] By detecting changes in eye rotation during work, determining the time-series pattern, and comparing it with a standard pattern, the work content can be objectively verified. For example, in maintenance and inspection work where it is necessary to check distant and near targets, detecting congestion during the actual work can confirm whether the work is being performed according to the procedure. The verification result may be notified to the worker as an alarm. Although not maintenance and inspection, this embodiment can also be applied to point-and-call confirmation after a train departs.

[0099] Comparing with standard patterns can be used not only to verify work procedures but also to master work procedures involving congestion. For example, during driver's license renewal training, videos of excellent drivers performing safety checks are sometimes shown. In this case, instead of simply watching, comparing the eye movement patterns of the excellent driver when actually performing the same safety check allows for a more detailed understanding of ideal congestion changes.

[0100] Furthermore, eye rotation is not limited to the right and left eyeballs rotating in opposite directions (convergence), but also includes rightward and leftward gaze movements where the right and left eyeballs rotate in the same direction. Hands-free control may be performed by combining these with convergence detection. In addition, nystagmus can be detected based on rightward and leftward gaze movements that occur along with convergence when drowsiness increases. This allows for the issuance of warnings to drowsy subjects. Moreover, in addition to horizontal eye rotation, vertical rotation can also be detected, and hands-free control may be performed by combining horizontal and vertical eye rotations.

[0101] It should be noted that the present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Moreover, components from different embodiments may be appropriately combined. [Explanation of Symbols]

[0102] 32...Right temple electrode, 36...Left temple electrode, 42...Right nose pad electrode, 44...Left nose pad electrode, 46...Neutral electrode, 62, 64, 66...A / D converter, 75...Eye movement detection unit.

Claims

1. A wearable device connected to a control device, A frame that is worn by a user and includes a transparent section from which the user can see the real world, and a display that shows augmented reality images, A detector that detects the convergence angle between the line of sight direction of the user's right eye and the line of sight direction of the user's left eye, A processor and, The aforementioned processor, Based on the convergence angle detected by the detector, it is determined whether the user is looking at a first object at a first distance in the real world, or whether the user is looking at a second object at a second distance that is closer than the first distance in the real world. If it is determined that the user is looking at the first object, the control device is requested to provide location information, and the air tag based on the location information is displayed on the display as the first augmented reality image. A wearable device that, when it determines that the user is looking at the second object, requests list information representing instructions for the user from the control device, and displays the list information as a second augmented reality image on the display.

2. A control device, A system comprising a wearable device connected to the control device, The aforementioned wearable device is A frame that is worn by a user and includes a transparent section from which the user can see the real world, and a display that shows augmented reality images, A detector that detects the convergence angle between the line of sight direction of the user's right eye and the line of sight direction of the user's left eye, A processor and, The aforementioned processor, Based on the convergence angle detected by the detector, it is determined whether the user is looking at a first object at a first distance in the real world, or whether the user is looking at a second object at a second distance that is closer than the first distance in the real world. If it is determined that the user is looking at the first object, the control device is requested to provide location information, and the air tag based on the location information is displayed on the display as the first augmented reality image. A system that, when it is determined that the user is looking at the second object, requests list information representing instructions for the user from the control device, and displays the list information on the display as a second augmented reality image.

3. A control device and a wearable device connected to the control device, The aforementioned wearable device is A frame that is worn by a user and includes a transparent section from which the user can see the real world, and a display that shows augmented reality images, A display method for a system comprising a detector that detects the convergence angle between the line of sight direction of the user's right eye and the line of sight direction of the user's left eye, Based on the convergence angle detected by the detector, it is determined whether the user is looking at a first object at a first distance in the real world, or whether the user is looking at a second object at a second distance that is closer than the first distance in the real world. If it is determined that the user is looking at the first object, the control device is requested to provide location information, and the air tag based on the location information is displayed on the display as the first augmented reality image. A display method comprising: determining that the user is looking at the second object; requesting list information representing instructions for the user from the control device; and displaying the list information as a second augmented reality image on the display.

Images