Binocular system for inputting commands

Abstract

A binocular system for inputting commands includes a computer and a binocular eyepiece. The computer generates a virtual graphical user interface (GUI) having one or more graphical elements, where each graphical element corresponds to a command. The binocular eyepiece comprises two eyepieces. Each eyepiece has an optical path that directs an image of an object toward a respective eye of a pair of eyes. An optical path of at least one eyepiece directs the virtual GUI towards the respective eye. At least one eyepiece is associated with an eye tracker that tracks movement of the respective eye relative to the virtual GUI to produce a tracked eye. The computer interprets motion of the tracked eye relative to the virtual GUI as interaction with a selected graphical element and initiates a command corresponding to the selected graphical element.

Description

[0001] This application is a continuation of PCT Application No. PCT / IB2019 / 055116, filed on June 18, 2019, in the name of Chinese Application No. 201980043103.7, entitled “Binocular System for Inputting Commands,” which claims priority to U.S. Provisional Patent Application No. 62 / 859, 1 10, filed on June 18, 2019, entitled “Binocular System for Inputting Commands.” TECHNICAL FIELD

[0002] The present disclosure relates generally to controlling medical systems, and more particularly to a binocular system for inputting commands. BACKGROUND

[0003] Medical devices can perform a wide variety of actions in response to commands from an operator. For example, an operator can select a command from a command panel to change the magnification, focus, and brightness of a microscope. However, inputting commands to medical devices has special problems. Touching a command panel can contaminate the panel. In addition, searching the panel for the portion used to input a certain command takes time and can distract the user. Thus, known command panels are sometimes unsuitable for certain situations. SUMMARY

[0004] In certain embodiments, a binocular system for inputting commands includes a computer and a binocular eyepiece. The computer generates a virtual graphical user interface (GUI) having one or more graphical elements, where each graphical element corresponds to a command. The binocular eyepiece includes an eyepiece. Each eyepiece has an optical path that directs an image of an object toward a respective eye of a pair of eyes. The optical path of at least one eyepiece directs the virtual GUI toward the respective eye. At least one eyepiece is associated with an eye tracker that tracks movement of the respective eye relative to the virtual GUI to produce a tracked eye. The computer interprets movement of the tracked eye relative to the virtual GUI as an interaction with a selected graphical element and initiates a command corresponding to the selected graphical element.

[0005] In certain embodiments, a method of inputting commands using a binocular system includes a computer generating a virtual graphical user interface (GUI) including one or more graphical elements. Each graphical element corresponds to a command. An optical path of each eyepiece in a binocular eyepiece directs an image of an object toward a respective eye of a pair of eyes. The optical path of at least one eyepiece directs the virtual GUI toward the respective eye. An eye tracker associated with at least one eyepiece tracks movement of the respective eye relative to the virtual GUI to produce a tracked eye. Movement of the tracked eye relative to the virtual GUI is interpreted as an interaction with a selected graphical element. A command corresponding to the selected graphical element is initiated. BRIEF DESCRIPTION OF DRAWINGS

[0006] The embodiments disclosed herein are explained in more detail by way of example with reference to the accompanying drawings, in which:

[0007] Figure 1 An embodiment of a binocular system that allows users to input commands through eye movements is shown;

[0008] Figure 2 Another embodiment of a binocular system that allows users to input commands through eye movements is shown;

[0009] Figure 3A An example of a virtual GUI is shown;

[0010] Figure 3B It shows that it can be used Figure 3A An example of an options menu displayed in a virtual GUI; and

[0011] Figure 4 It shows that it can be used with Figure 1 and Figure 2 An example of a method for inputting commands via eye movements used in systems. Detailed Implementation

[0012] Exemplary embodiments of the disclosed apparatus, system, and method are now shown in detail with reference to the specification and accompanying drawings. It will be apparent to those skilled in the art that the disclosed embodiments are exemplary and not an exhaustive list of all possible embodiments.

[0013] Figure 1 and Figure 2 An embodiment of a binocular system 10 that allows a user to input commands through eye movements is illustrated. The illustrated example system 10 includes a computer 20 and binocular eyepieces 22, which are attached to a device 32 via mounting brackets 34. The computer 20 includes an interface 40, one or more processors 42, and one or more memories 44 storing logic such as computer programs for a virtual GUI 50, eye tracking 52, and device control 54. The binocular eyepieces 22 include eyepieces 24, a display 29, an eye tracker 30, and one or more beam splitters 31. The eye tracker 30 includes lenses 60, sensors 62, and illuminators 64.

[0014] In an example of operation, binocular system 10 allows a user to input commands to any suitable device 32, such as a medical device. Computer 20 generates a virtual graphical user interface (GUI) that is an image that includes graphical elements that correspond to commands. Eyepieces 24 each have an optical path 26 that directs images of objects toward a respective eye of a pair of eyes. The optical path 26 of one or both eyepieces 24 also directs the virtual GUI toward the respective eye. One or both eyepieces 24 are associated with an eye tracker 30 that tracks movement of the respective eye relative to the virtual GUI. Computer 20 interprets movement of the tracked eye relative to the virtual GUI as an interaction with a selected graphical element and initiates a command that corresponds to the selected graphical element.

[0015] In certain embodiments, computer 20 generates a virtual GUI that is passed along optical path 26 to at least one eye. The virtual GUI includes one or more graphical elements that can have any suitable size or shape. Each graphical element corresponds to a command or instruction to device 32 that is generally used to perform an action, such as accepting a selection or setting defined by the user, performing a user-selected operation that is programmed into computer 20, displaying information requested by the user, or other appropriate action. The user can input a command by interacting with a graphical element associated with a command in a manner that signals a selection of the element. The interaction is movement of the eye (e.g., moving or directing the gaze of the eye, or blinking) relative to the graphical element that indicates, for example, a selection of the element. For example, the user can direct their gaze at the element for at least a predefined amount of time (e.g., at least 3, 5, or 10 seconds). As another example, the user can direct their gaze at the element and can blink a predetermined number of times (e.g., 1, 2, or 3 times). In certain embodiments, the interaction can be confirmed by movement of another part of the user's body. For example, the user can direct their gaze toward an element to select the element, and then confirm the selection of the element by, for example, depressing a foot pedal with their foot or pressing a physical button with their hand. In certain embodiments, the virtual GUI can indicate whether the user's gaze has interacted with or selected an element. For example, the virtual GUI can highlight (e.g., make brighter or change the color of) an element that the user's gaze has selected. The user can confirm the selection by, for example, blinking or moving a hand or foot. See Figure 3A and 3B Examples of virtual GUIs are described in more detail.

[0016] Eyepieces 24 (24a, 24b) of binocular eyepiece 22 correspond to a pair of eyes. Typically, one eyepiece 24a corresponds to one eye (e.g., the left eye), and the other eyepiece 24b corresponds to the other eye (e.g., the right eye). Eyepieces 24 can be generally tubular housings having one or more optics that define a light path 26 (i.e., a path that light can travel) through eyepiece 24. The optics can be, for example, lenses, splitters, prisms, coated glass or mirrors, or systems including multiple optics (such as a lens system). Paths 26 (26a, 26b) of eyepieces 24 (24a, 24b) generally direct images of objects toward respective eyes of the pair of eyes. For example, device 32 can be a microscope that captures images of objects, and paths 26 can direct the images of the objects toward the eyes.

[0017] Paths 26 of one or both eyepieces 24 can receive a virtual GUI from computer 20 and direct the GUI toward the respective eye. In Figure 1 , path 26a of eyepiece 24a directs the GUI, and in Figure 2 , paths 26a, 26b of eyepieces 24a, 24b direct the GUI. Virtual GUI program 50 of computer 20 generates image data that describes the virtual GUI. The image data is sent to display 29, which displays the GUI. Any suitable display can be used, for example, a display using light emitting diodes (LEDs), micro organic light emitting diodes (micro-OLEDs), liquid crystal displays (LCDs), digital light processing (DLP), liquid crystal on silicon (LCoS), or LED projectors. Display 29 can have any suitable size and shape, and can be located in any suitable location. In Figure 1 , display 29 is proximate to beamsplitter 31, which receives and reflects the virtual GUI and the images of the objects and directs them along paths 26. In Figure 2 , display 29 is proximate to beamsplitter 31, which receives and reflects the virtual GUI and the images of the objects and directs them along paths 26. In other embodiments, display 29 can be positioned toward the eye- proximate end of eyepieces 24, and the eyes can view the virtual GUI directly from display 29.

[0018] At least one eyepiece 24 is associated with eye tracker 30, which tracks motion of the eyes relative to the virtual GUI, indicating the region of the GUI that the gaze is directed toward (i.e., the eye is looking at). An eyepiece 24 is “associated with” eye tracker 30 (and vice versa) means that eye tracker 30 tracks the eye that corresponds to eyepiece 24. For example, the “right” eyepiece 24 corresponds to the right eye. Eye tracker 30 that corresponds to right eyepiece 24 tracks the right eye.

[0019] The eye tracker 30 can be placed at any suitable location that allows it to track eye movements. Figure 1 and Figure 2 In one embodiment, eyepiece 24a includes an eye tracker 30. In other embodiments, eyepiece 24b includes an eye tracker 30, or both eyepieces 24a and 24b include eye trackers 30. Furthermore, one eyepiece 24 may include an eye tracker 30 and a path 26 guiding the virtual GUI; or one eyepiece 24 may include an eye tracker 30 while the other eyepiece 24 may have a path 26 guiding the virtual GUI.

[0020] In yet another embodiment, one or more portions of the eye tracker 30 are detached from the eyepiece 24. For example, as Figure 2 The sensor 62 shown (and optionally the illuminator 64) can be attached to a ring located between the eyepiece 24a and the eye (e.g., on top of and separate from the eyepiece 24a, or directly coupled (possibly detachably) to the eyepiece 24a). Such a ring allows an operator to improve an existing binocular eyepiece 22 with an eye tracker 24, or to replace one eye tracker 24 with another. In some embodiments, such a ring allows an operator to insert the eye tracker 24 into the path 26 when eye tracking is desired, and to remove the eye tracker 24 from the path 26 when eye tracking is not desired.

[0021] The eye tracker 30 has one or more sensors 62 that detect light reflection from the eye (e.g., from the cornea (e.g., the anterior surface), the center of the pupil, the limbus, the lens (posterior surface), and / or other parts of the eye). The sensors 62 generate image data describing the light and send the image data to the computer 20. The sensors 62 can be placed at any suitable location that allows them to track eye movements. Figure 1 In this process, the reflection from the eye travels along path 26a, and the beam splitter 31 and lens 60 guide the reflection to sensor 62. Figure 2 In this sensor 62, positioned along the edge of the eyepiece 24a near the eye, it detects reflections from the eye. Any suitable sensor 62 can be used, such as a camera or charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or profilometer sensor. The profilometer sensor operates by summing the values ​​of all pixels in a row and / or column into a single value. Any suitable sampling rate can be used, for example, in the range of 30 Hz to 50 Hz, 50 Hz to 200 Hz, 200 Hz to 500 Hz, 500 Hz to 1000 Hz, or exceeding 1000 Hz.

[0022] In some embodiments, the eye tracker 30 has one or more illuminators 64 that illuminate the tracked eye to produce a reflection that can be sensed by the sensor 62. The illuminators 64 can be illuminated with any suitable light (e.g., visible or infrared light). Any suitable illuminator can be used, such as an LED illuminator, a halogen lamp, or other suitable illuminator. In other embodiments, light from the device or ambient light may be sufficient to illuminate the eye, making the illuminator 64 unnecessary.

[0023] The eye-tracking program 52 of computer 20 interprets the movement of the tracked eye relative to the virtual GUI as an interaction with a selected graphical element, and the device control program 54 initiates a command corresponding to the selected graphical element. The eye-tracking program 52 includes a known algorithm for determining the gaze direction based on image data from sensor 62. Processor 42 performs calculations based on the algorithm to determine the gaze direction. Additionally, the eye-tracking program can detect other eye movements (e.g., blinking). Given the gaze direction and the position of the virtual GUI, processor 42 determines whether the gaze has interacted with an element of the GUI in a manner indicating selection of an element. If an element is selected, device control program 54 initiates a command corresponding to the selected element.

[0024] Optical devices can be used to direct light to the eye and sensors 62. For example, in Figure 1 and Figure 2 In this process, beam splitter 31 receives a virtual GUI, receives an image of an object captured by device 32, and guides the virtual GUI and the image along the optical path toward the corresponding eye. As another example, in... Figure 1 In this process, beam splitter 31 receives light reflected from the eye and directs the light toward sensor 62 of eye tracker 30.

[0025] The binocular system 10 allows a user to input commands to any suitable device 32, such as a medical device, through eye movements. Examples of medical devices include ophthalmic surgical equipment, therapeutic equipment, or diagnostic equipment. A mounting bracket 34 can be used to connect the binocular eyepiece 22 to the device 32. The mounting bracket 34 can have any suitable size and shape to achieve the connection between the binocular eyepiece 22 and the device 32.

[0026] Figure 3A An example of the virtual GUI 80 was shown, and Figure 3BAn example of an options menu that can be displayed in a virtual GUI 80 is shown. In the example shown, the virtual GUI 80 is overlaid on an image of an object 82 being viewed through binocular eyepiece 22. The virtual GUI 80 includes graphical elements that can be selected via eye gaze, including an option element 84, an option display area 86, a previous element 88, a next element 90, and an OK element 92. The option element 84 corresponds to a command that displays one or more options menus. The menu lists options that the user can select via eye gaze. The example menu includes a surgical element 94, a lens element 96, a microscope element 98, and an on / off element 100. The surgical element 94 lists selectable surgical procedures, and the lens element 96 lists selectable lens types. The microscope element 98 lists microscope features that can be adjusted by selecting a PLUS element corresponding to an increase in feature or a MINUS element corresponding to a decrease in feature. The on / off element 100 can be selected to turn the virtual GUI on or off.

[0027] The previous element 88 corresponds to the back command (e.g., move to the previous menu, move to the previous option in the menu list, or move to the previous step in surgery). The next element 90 corresponds to the forward command (e.g., move to the next menu, or move to the next option in the menu list). The OK element 92 corresponds to accepting the command. For example, a user can select an option from a menu, and the virtual GUI 80 can display a question asking the user to confirm the selected option. The user can select the OK element 92 to confirm. Of course, the virtual GUI 80 can have any suitable graphical elements (of any suitable size or shape) capable of executing any appropriate command.

[0028] Figure 4 It shows that it can be used with Figure 1 and Figure 2 An example of a method for inputting commands via eye movements used in conjunction with System 10. The method begins at step 110, in which binocular eyepiece 22 displays a virtual GUI that can be viewed by at least one of the user's eyes. Eye movements of the user's eyes are tracked at step 112 via steps 114 and 116. At step 114, illuminator 64 illuminates the eyes. At step 116, sensor 62 detects eye movements by detecting light reflected from the eyes. Sensor 62 generates image data describing the light and sends the image data to computer 20.

[0029] At step 118, computer 20 determines whether the user has selected a command. Computer 20 determines whether eye movement relative to the virtual GUI corresponds to the selection of a graphical element in the GUI, thereby indicating whether the user has selected a command corresponding to that element. If no command has been selected, the method returns to step 112 to continue tracking eye movement. If a command has been selected, the method proceeds to step 120 to activate the selected command. At step 122, the method checks whether the user has entered a command to close the virtual GUI. If no command to close the GUI exists, the method returns to step 112 to continue tracking eye movement. If a command exists, the method proceeds to step 124 to close the GUI and the method ends.

[0030] Components of the systems and apparatuses disclosed herein (e.g., a computer) may include interfaces, logic, and / or memory, any of which may include hardware and / or software. An interface may receive inputs from a component, provide outputs to a component, and / or process inputs and / or outputs. Logic may perform operations on a component, such as executing instructions to generate outputs from inputs. Logic may be a processor, such as one or more computers or one or more microprocessors (e.g., a chip residing in a computer). Logic may be computer-executable instructions encoded in memory, which may be executed by a computer, such as a computer program or software. Memory may store information and may include one or more tangible, non-transitory, computer-readable, computer-executable storage media. Examples of memory include computer memory (e.g., random access memory (RAM) or read-only memory (ROM)), mass storage media (e.g., hard disks), removable storage media (e.g., optical discs (CDs) or digital video discs (DVDs)), and network storage devices (e.g., servers or databases).

[0031] Even though this disclosure has described certain embodiments, modifications to the embodiments (such as substitutions, additions, alterations, or omissions) will be apparent to those skilled in the art. Therefore, modifications can be made to the embodiments without departing from the scope of this disclosure. For example, modifications can be made to the systems and apparatus disclosed herein. Components of the systems and apparatus may be integral or separate, and operation of the systems and apparatus may be performed by more, fewer, or other components. As another example, modifications can be made to the methods disclosed herein. The methods may include more, fewer, or other steps, and the steps may be performed in any suitable order.

Claims

1. A binocular system for inputting commands, comprising: A computer configured to generate a virtual graphical user interface (GUI) including one or more graphical elements corresponding to one or more commands for adjusting a medical device, the one or more commands including a first command to adjust the focus of a microscope in the medical device and a second command to adjust the brightness of the microscope; as well as A binocular eyepiece, comprising a plurality of eyepieces of the microscope, each eyepiece having an optical path, each optical path being configured to guide an image of an object toward a corresponding eye in a pair of eyes: The optical path of at least one eyepiece is configured to guide the virtual GUI toward the corresponding eye, and the optical path is configured to guide the virtual GUI in the following manner: Receive the virtual GUI, the virtual GUI including the one or more graphical elements; Receive the image; and The virtual GUI and the image are guided along the light path toward the corresponding eyes; and At least one eyepiece is associated with an eye tracker configured to track the movement of the corresponding eye relative to the virtual GUI to generate a tracked eye; and The computer is further configured to adjust the microscope in the following ways: The first movement of the tracked eye relative to the virtual GUI is interpreted as an interaction with a selected first graphical element, which corresponds to a first command to adjust the focus of the microscope; The second movement of the tracked eye relative to the virtual GUI is interpreted as an interaction with a selected second graphical element, which corresponds to a second command to adjust the brightness of the microscope; and The first and second commands are initiated in response to interpreting the first and second movements of the tracked eye to adjust the focus and brightness of the microscope.

2. The system as claimed in claim 1, wherein, The eye tracker includes: One or more sensors configured to track the movement of the tracked eye, the one or more sensors including a sensor disposed between the at least one eyepiece and the tracked eye.

3. The system as described in claim 1, wherein, The eye tracker includes: One or more illuminators configured to illuminate the tracked eye; and One or more sensors configured to track the movement of the tracked eye.

4. The system as claimed in claim 1, wherein, The at least one eyepiece including the eye tracker includes a beam splitter, the beam splitter being configured to: Receive the image and guide the image toward the corresponding eye along the optical path; and The sensor receives light reflected from the corresponding eye and directs the light toward the eye tracker.

5. The system as claimed in claim 1, wherein, The eyepiece, which has the optical path configured to guide the virtual GUI, also includes the eye tracker.

6. The system as claimed in claim 1, wherein, The plurality of eyepieces includes: A first eyepiece, the first eyepiece including an eyepiece having an optical path configured to guide the virtual GUI; and The second eyepiece includes the eye tracker.

7. The system as claimed in claim 1, wherein, The optical paths of each of the plurality of eyepieces are configured to guide the virtual GUI toward the corresponding eye.

8. The system as claimed in claim 1, wherein, Each of the multiple eyepieces includes an eye tracker.

9. The system as claimed in claim 1, wherein, The graphic element includes at least one of the following: The OK element corresponding to the accepted command; The PLUS element corresponding to the increase command; and The MINUS element corresponding to the decrease command.

10. The system of claim 1, wherein, The graphic element includes at least one of the following: The previous element corresponding to the forward command; and The next element corresponding to the back command.

11. The system of claim 1, wherein, The graphical elements include option elements corresponding to commands that display one or more option menus.

12. The system of claim 1, further comprising a mounting element configured to attach to a medical device.

13. A method for inputting commands using a binocular system, the method comprising: A computer-generated virtual graphical user interface (GUI) includes one or more graphical elements corresponding to one or more commands for adjusting a medical device, the one or more commands including a first command to adjust the focus of a microscope in the medical device and a second command to adjust the brightness of the microscope; The optical path of each of the multiple eyepieces in a binocular system guides the image of an object toward the corresponding eye in a pair of eyes; The virtual GUI is guided toward the corresponding eye by the optical path of at least one eyepiece; The virtual GUI, which includes the one or more graphical elements, is received by a beam splitter. The image is received by the beam splitter; and The beam splitter guides the virtual GUI and the image along the optical path toward the corresponding eye; The movement of the corresponding eye relative to the virtual GUI is tracked by an eye tracker associated with at least one eyepiece to generate the tracked eye; The first movement of the tracked eye relative to the virtual GUI is interpreted as an interaction with a selected first graphical element, which corresponds to a first command to adjust the focus of the microscope; The second movement of the tracked eye relative to the virtual GUI is interpreted as an interaction with a selected second graphical element, which corresponds to a second command to adjust the brightness of the microscope; as well as The first and second commands are initiated in response to interpreting the first and second movements of the tracked eye to adjust the focus and brightness of the microscope.

14. The method of claim 13, further comprising: The movement of the tracked eye is tracked using one or more sensors; Illuminate the tracked eye using one or more illuminators; The movement of the tracked eye is tracked using one or more sensors. The virtual GUI is received by a beam splitter having an eyepiece configured to guide the optical path; The image is received by the beam splitter; The beam splitter guides the virtual GUI and the image along the optical path toward the corresponding eye; The image is received by a beam splitter that includes at least one eyepiece of the eye tracker and the image is guided toward the corresponding eye along the optical path; as well as The beam splitter receives light reflected from the corresponding eye and directs the light toward the sensor of the eye tracker.

15. The method of claim 13, further comprising: The movement of the tracked eye is tracked by a sensor of the eye tracker, the sensor being arranged between the at least one eyepiece and the tracked eye.

Images