Systems and methods for utilizing surgical tool instrument with graphical user interface

Through the combination of graphical user interface system and image sensors, the interaction with the GUI is solved by using surgical tools and instruments, which solves the difficulties of surgeons in operating computer systems in a sterile environment, and achieves efficient and safe surgical input, reducing errors during the operation. and distraction.

CN118975852APending Publication Date: 2024-11-19ALCON INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411052828.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-05-23
Filing Date
2019-05-22
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In a sterile surgical environment, there are difficulties in the interface between the surgeon and the computer system, which can easily lead to input errors, distractions, and affect the surgical results.

Method used

The graphical user interface system is used to display icons and cursors through the display, combined with image sensors and digital models of surgical tools and instruments, determine the location of the tool and instruments, allowing surgeons to use surgical tools and instruments as fixed-point devices to interact with the GUI, so as to realize that there is no need to directly operate the computer system. The input can be completed.

Benefits of technology

It improves the safety and efficiency of the surgical procedure, reduces the possibility of input errors, avoids distractions during the operation, and ensures the predictability of surgical results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975852A_ABST
    Figure CN118975852A_ABST
Patent Text Reader

Abstract

The present disclosure relates to systems and methods utilizing a surgical tool instrument with a graphical user interface. The present disclosure provides a system that: may display a graphical user interface (GUI) including icons via a display; a first image from an image sensor may be received; determining a first position of a surgical tool instrument within the first image based on the first image and a digital model of the surgical tool instrument; a cursor of the GUI at a second position can be displayed; receiving a second image from the image sensor; determining a third position of the surgical tool instrument within the second image based on the second image and the digital model; a cursor of the GUI at a fourth position can be displayed; a user input may be received that indicates a selection when the coordinates of the cursor include the fourth position; and may determine that the user input indicates a selection when the coordinates of the at least one icon include the fourth position.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the invention patent application PCT application PCT / IB2019 / 054241 with Chinese application number 201980033910.0 and application date May 22, 2019, and entitled “Systems and methods for surgical tools and instruments with a graphical user interface”. Technical Field

[0002] The present disclosure relates to quality assurance in medical procedures, and more particularly to systems and methods for verifying patient information utilizing a medical procedure or process elements of a medical procedure. Background Art

[0003] Computer systems can assist surgeons in performing surgical operations. Computer systems provide graphical user interfaces. However, in a sterile environment, surgeons cannot easily touch non-sterile devices, such as interfaces with computer systems. At present, there are different possibilities for surgeons to interact with the interfaces of computer systems, such as foot pedals, surgical assistants (e.g., medical staff), and disposable discards (e.g., cotton swabs) for interacting on touch screens and / or keyboards. These solutions may be prone to error and may cause input errors. For example, during interaction with a computer system, a surgeon may have to physically move his or her hands and / or head from a patient to a computer system interface to ensure that his or her computer system input is correct. This may be a potential distraction during surgery, which may lead to unforeseen and / or negative surgical results. Summary of the invention

[0004] The present disclosure provides a system capable of: displaying a graphical user interface including at least one icon via a display; receiving a first image from an image sensor; and determining a first position of the surgical tool instrument within the first image based on the first image and a digital model of the surgical tool instrument. For example, the surgical tool instrument to be used as a pointer may include a scalpel, a cotton swab, tweezers, etc. The system may include a display that displays the graphical user interface, or may be connected to the display. The system may include a microscope-integrated display containing the display, the display displaying the graphical user interface. The system may further display a cursor of the graphical user interface at a second position associated with the first position. The system may further receive a second image from the image sensor; determine a third position of the surgical tool instrument within the second image based on the second image and the digital model of the surgical tool instrument; and display a cursor of the graphical user interface at a fourth position associated with the third position. The system may further receive user input, the user input indicating a selection when the coordinates of the at least one icon include the fourth position. For example, the selection may indicate a selection of an icon. The system may further determine that the user input indicates a selection when the coordinates of the at least one icon include the fourth position; and in response to determining that the user input indicates a selection, change the data displayed by the graphical user interface. For example, the image data of the graphical user interface may be changed in conjunction with a next step of a workflow associated with a surgeon. As another example, at least a portion of the first image, the second image, or the third image may be stored in response to determining that the user input indicates a selection. The image data may be present via a storage medium.

[0005] The present disclosure may further include a non-transitory computer readable memory device having instructions that, when executed by a processor of the system, cause the system to perform the above steps. The present disclosure further includes a system or a non-transitory computer-readable memory device as described above, which has one or more of the following additional features, which can be used in combination with each other unless they are clearly mutually exclusive: i) when the processor executes the instructions, the system can be further capable of displaying a graphical user interface including at least one icon via a display; ii) when the processor executes the instructions, the system can be further capable of receiving a first image from an image sensor; iii) when the processor executes the instructions, the system can be further capable of determining a first position of the surgical tool instrument within the first image based on the first image and the digital model of the surgical tool instrument; iv) when the processor executes the instructions, the system can be further capable of displaying a cursor of the graphical user interface at a second position associated with the first position; v) when the processor executes the instructions, the system can be further capable of receiving a second image from the image sensor; vi) when the processor executes the instructions, the system can be further capable of determining a third position of the surgical tool instrument within the second image based on the second image and the digital model of the surgical tool instrument; vii) when the processor executes the instructions, the system can be further capable of viii) when the processor executes the instructions, the system may further be capable of receiving user input, the user input indicating a selection when the coordinates of the at least one icon include the fourth position; ix) when the processor executes the instructions, the system may further be capable of determining that the user input indicates a selection when the coordinates of the at least one icon include the fourth position; x) when the processor executes the instructions, the system may further be capable of changing data displayed by the graphical user interface in response to determining that the user input indicates a selection; xi) when the system receives the user input indicating a selection, the system may further be capable of receiving a third image from the image sensor; xii) when the system determines that the user input indicates a selection, the system may further be capable of determining a change in the number of pixels associated with the surgical tool instrument based at least on the second image and the third image and the digital model, the change in the number of pixels indicating a change in the distance of the surgical tool instrument from the image sensor; xiii) when the system receives the user input indicating a selection, the system may further be capable of receiving a plurality of images from the image sensor;xv) when it is determined that the user input indicates a selection, the system may be further capable of determining a movement mode associated with the movement of the surgical tool instrument that indicates the selection based at least on the digital model and the plurality of images; xvi) when the processor executes the instructions, the system may be further capable of determining a movement mode associated with the movement of the surgical tool instrument that indicates the selection based at least on the plurality of images; xvii) when the processor executes the instructions, the system may be further capable of storing at least a portion of the first image, the second image, or the third image via the memory medium in response to determining that the user input indicates a selection; xviii) when the processor executes the instructions, the system may be further capable of advancing a workflow associated with the surgeon to a next step of the workflow. ;

[0006] Any of the above systems can perform any of the above methods, and any of the above non-transitory computer-readable memory devices can cause a system to perform any of the above methods. Any of the above methods can be implemented on any of the above systems or using any of the above non-transitory computer-readable memory devices.

[0007] It should be understood that the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art through the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] For a more complete understanding of the present disclosure and its features and advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, which are not drawn to scale, in which:

[0009] Figure 1A Examples of the system are shown;

[0010] Figure 1B Examples of microscope-integrated displays were shown, as well as examples of surgical tool instruments;

[0011] Figure 2 Examples of computer systems are presented;

[0012] Figure 3A and Figure 3B Examples of graphical user interfaces are shown;

[0013] Figure 4A and Figure 4B Examples of alignment areas are shown;

[0014] Figure 4C Examples of alignment movement patterns are shown;

[0015] Figure 4D An example of receiving mobile patterns is shown;

[0016] Figure 4E Another example of receiving mobile patterns is shown;

[0017] Figure 5 An example of a method of utilizing a surgical tool instrument having a graphical user interface is presented; and

[0018] Figure 6 Another example of a method of utilizing a surgical tool instrument having a graphical user interface is presented. DETAILED DESCRIPTION

[0019] In the following description, details are set forth by way of example to facilitate discussion of the disclosed subject matter. However, it will be apparent to one of ordinary skill in the art that the disclosed embodiments are exemplary and not exhaustive of all possible embodiments.

[0020] As used herein, a reference numeral refers to a class or type of entity, and any letter following such a reference numeral refers to a specific example of a particular entity of that class or type. Thus, for example, a hypothetical entity referenced by '12A' may refer to a particular example of a particular class / type, and in general, the reference '12' may refer to a collection of examples belonging to that particular class / type, or any one example of that particular class / type.

[0021] The surgeon may be in a sterile surgical environment. The surgeon may use his or her surgical tools and instruments to control and / or guide a graphical user interface (GUI). The GUI may be used to control the workflow associated with the surgical procedure. In the process of using the surgical tools and instruments to control and / or guide the GUI, a device may determine one or more shapes of the surgical tools and instruments. For example, one or more cameras may provide one or more images to the device. The device may determine the surgical tools and instruments based on the one or more images from the one or more cameras. The device may track one or more movements of the surgical tools and instruments. For example, the device may track one or more movements of the surgical tools and instruments based at least on the one or more images from the one or more cameras.

[0022] One or more movements of the tracked surgical tool instrument can be used to interact with the GUI. In one example, the GUI can be displayed via a microscope integrated display (MID). In another example, the GUI can be displayed via a display. The surgeon can view and / or interact with the GUI via the MID. The surgeon and / or other surgical personnel can interact with the GUI via the display. The GUI can be superimposed on the surgeon's current area of ​​interest. For example, the GUI can be superimposed on the surgeon's current area of ​​interest so that the surgeon can visualize the GUI without looking elsewhere from the surgeon's current area of ​​interest. For example, the GUI can be superimposed on a live scene. Motion-based object tracking can be used to interact with the GUI. For example, using motion-based object tracking and / or object recognition, surgical tool instruments can be used as a fixed-point device for interacting with the GUI. Examples of fixed-point devices can be or include one or more of the following: a mouse, a touchpad, and a trackball, etc.

[0023] Surgical tools and instruments can be aligned with the system for use in association with the GUI. For example, surgical tools and instruments can be aligned with the system to be used as a pointing device for use in association with the GUI. Aligning surgical tools and instruments for use in association with the GUI may include: the system receives one or more images of surgical tools and instruments, and determines one or more shapes and / or one or more curves of the surgical tools and instruments that can be used to identify the surgical tools and instruments. For example, one or more machine learning processes, and / or one or more machine learning methods can be used to determine one or more shapes and / or one or more curves of the surgical tools and instruments that can be used to identify the surgical tools and instruments. The one or more machine learning processes, and / or one or more machine learning methods can generate and / or determine a digital model of the surgical tools and instruments. For example, the digital model can be used to infer one or more positions of surgical tools and instruments used in association with the GUI.

[0024] One or more movements of the surgical tool instrument can be used to determine the pointer "click". For example, one or more movements of the surgical tool instrument can be used as a mouse click. The pointer "click" can indicate the selection of one or more items displayed via the GUI. After the surgical tool instrument is aligned with the system, one or more movements of the surgical tool instrument can be determined and / or identified as a pointer "click". In one example, the first movement can be used as a left mouse button selection (e.g., "click"). In a second example, the second movement can be used as a right mouse button selection (e.g., "click"). In a third example, the third movement can be used as a left mouse button to hold the selection (e.g., hold down the left mouse button). In another example, the fourth movement can be used as a left mouse button release selection (e.g., release the left mouse button). One or more motion-based object tracking processes and / or one or more motion-based object tracking methods can be used. For example, the one or more motion-based object tracking processes and / or one or more motion-based object tracking methods can track the surgical tool instrument using one or more of background subtraction, frame difference method, and optical flow, etc.

[0025] Now turn to Figure 1A , shows an example of a system. As shown, a surgeon 110 can utilize a surgical tool instrument 120. In one example, the surgeon 110 can utilize the surgical tool instrument 120 in a surgical procedure involving a patient portion 130 of a patient 140. For example, the surgeon 110 can utilize a surgical tool instrument 120 that interacts with and / or utilizes the system 100. For example, the system 100 can be or include an ophthalmic surgical tool tracking system. As shown, the system 100 can include a computing device 150, a display 160, and a MID 170.

[0026] The computing device 150 may receive image frames captured by one or more image sensors. For example, the computing device 150 may perform various image processing on the one or more image frames. The computing device 150 may perform image analysis on the one or more image frames to identify and / or extract one or more images of the surgical tool instrument 120 from the one or more image frames. The computing device 150 may generate a GUI that may be superimposed on the one or more image frames. For example, the GUI may include one or more indicators and / or one or more icons, etc. The one or more indicators may include surgical data, such as one or more positions and / or one or more orientations. The GUI may be displayed to the surgeon 110 and / or other medical staff via the display 160 and / or the MID 170.

[0027] The computing device 150, the display 160, and the MID 170 may be implemented in separate housings that are communicatively coupled to each other or in a common console or housing. A user interface may be associated with one or more of the computing device 150, the display 160, and the MID 170, etc. For example, in addition to other input devices, the user interface may also include one or more of the following: a keyboard, a mouse, a joystick, a touch screen, an eye tracking device, a voice recognition device, a gesture control module, a dial, and / or buttons, etc. A user (e.g., a surgeon 110 and / or other medical personnel) may input desired instructions and / or parameters via the user interface. For example, a user interface may be utilized to control one or more of the computing device 150, the display 160, and the MID 170, etc.

[0028] Now turn to Figure 1B , showing examples of microscope integrated displays and examples of surgical tools and instruments. As shown, surgical tool instrument 120A can be or include a scalpel. As shown, surgical tool instrument 120B can be or include a cotton swab. As shown, surgical tool instrument 120C can be or include forceps. Other surgical tools and instruments not specifically shown can be used with one or more systems, one or more processes, and / or one or more methods described herein.

[0029] As an example, the surgical tool instrument 120 may be marked with one or more patterns. The one or more patterns may be used to identify the surgical tool instrument 120. The one or more patterns may include one or more of a hash pattern, a stripe pattern, and a fractal pattern, etc. As another example, the surgical tool instrument 120 may be marked with a dye and / or a paint. The dye and / or the paint may reflect one or more of visible light, infrared light, and ultraviolet light, etc. In one example, the illuminator 178 may provide ultraviolet light, and the image sensor 172 may receive ultraviolet light reflected from the surgical tool instrument 120. The computer system 150 may receive image data based at least on ultraviolet light reflected from the surgical tool instrument 120 from the image sensor 172; and the surgical tool instrument 120 may be identified using the image data based at least on ultraviolet light reflected from the surgical tool instrument 120, according to other image data provided by the image sensor 172. In another example, the illuminator 178 may provide infrared light, and the image sensor 172 may receive infrared light reflected from the surgical tool instrument 120. The computer system 150 can receive image data based at least on infrared light reflected from the surgical tool instrument 120 from the image sensor 172; and can use the image data based at least on infrared light reflected from the surgical tool instrument 120 and other image data provided by the image sensor 172 to identify the surgical tool instrument 120.

[0030] As shown, MID 170 may include displays 162A and 162B. For example, surgeon 110 may look into multiple eyepieces, and displays 162A and 162B may display information to surgeon 110. Although MID 170 is shown as having multiple displays, MID 170 may include a single display 162. For example, MID 170 may be implemented as having one or more displays 162. As shown, MID 170 may include image sensors 172A and 172B. In one example, image sensors 172A and 172B may acquire images. In a second example, image sensors 172A and 172B may include cameras. In another example, image sensor 172 may acquire images via one or more of visible light, infrared light, and ultraviolet light, etc. One or more image sensors 172A and 172B may provide image data to computing device 150. Although MID 170 is shown as having multiple image sensors, MID 170 may include a single image sensor 172. For example, MID 170 may be implemented with one or more image sensors 172 .

[0031] As shown, MID 170 may include distance sensors 174A and 174. For example, distance sensor 174 may determine the distance from surgical tool instrument 120. Distance sensor 174 may determine the distance associated with the Z axis. Although MID 170 is shown as having multiple image sensors, MID 170 may include a single distance sensor 174. In one example, MID 170 may be implemented with one or more distance sensors 174. In another example, MID 170 may be implemented without a distance sensor. As shown, MID 170 may include lenses 176A and 176B. Although MID 170 is shown as having multiple lenses 176A and 176B, MID 170 may include a single lens 176. For example, MID 170 may be implemented with one or more lenses 176. As shown, MID 170 may include illuminators 178A and 178B. For example, illuminator 178 may provide and / or generate one or more of visible light, infrared light, and ultraviolet light, etc. Although MID 170 is shown as having multiple illuminators, MID 170 may include a single illuminator 178. For example, MID 170 may be implemented with one or more illuminators 178.

[0032] Now turn to Figure 2, shows an example of a computer system. As shown, the computer system 150 may include a processor 210, a volatile memory medium 220, a non-volatile memory medium 230, and an input / output (I / O) device 240. As shown, the volatile memory medium 220, the non-volatile memory medium 230, and the I / O device 240 may be communicatively coupled to the processor 210.

[0033] The term "memory medium" may mean "memory", "storage device", "memory device", "computer-readable medium", and / or "tangible computer-readable storage medium". For example, the storage medium may include, but is not limited to, the following storage media: for example, direct access storage devices (including hard drives), sequential access storage devices (such as tape disk drives), compact disks (CDs), random access memories (RAMs), read-only memories (ROMs), CD-ROMs, digital versatile disks (DVDs), electrically erasable programmable read-only memories (EEPROMs), flash memory, non-volatile media, and / or one or more combinations of the above media. As shown, the non-volatile memory medium 230 may include processor instructions 232. The processor instructions 232 may be executed by the processor. In one example, one or more portions of the processor instructions 232 may be executed via the non-volatile memory medium 230. In another example, one or more portions of the processor instructions 232 may be executed via the volatile memory medium 220. One or more portions of the processor instructions 232 may be transmitted to the volatile memory medium 220.

[0034] The processor 210 may execute the processor instructions 232 to implement one or more systems, one or more flow charts, one or more processes, and / or one or more methods described herein. For example, the processor instructions 232 may be configured, decoded, and / or encoded to have instructions according to one or more systems, one or more flow charts, one or more methods, and / or one or more processes described herein. One or more of the storage medium and the memory medium may be a software product, a program product, and / or an article of manufacture. For example, the software product, the program product, and / or an article of manufacture may be configured, decoded, and / or encoded to have instructions executable by the processor according to one or more flow charts, one or more methods, and / or one or more processes described herein.

[0035] The processor 210 may include any suitable system, device, or apparatus operable to interpret and execute program instructions, process data, or both stored in a memory medium and / or received via a network. The processor 210 may further include one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), or other circuit systems configured to interpret and execute program instructions, process data, or both.

[0036] I / O devices 240 may include any one or more tools that allow, permit, and / or enable a user to interact with computer system 150 and its associated components by facilitating user input and output to the user. Facilitating user input may allow the user to manipulate and / or control computer system 150, and facilitating output to the user may allow computer system 150 to indicate the effects of the user's manipulation and / or control. For example, I / O devices 240 may allow a user to input data, instructions, or both to computer system 150 and otherwise manipulate and / or control computer system 150 and its associated components. I / O devices may include user interface devices such as a keyboard, a mouse, a touch screen, a joystick, a handheld lens, a tool tracking device, a coordinate input device, or any other I / O device suitable for use with a system, such as system 100.

[0037] The I / O device 240 may include one or more buses, one or more serial devices, and / or one or more network interfaces, etc., which may facilitate and / or allow the processor 210 to implement one or more systems, processes and / or methods described herein. In one example, the I / O device 240 may include a storage interface that may facilitate and / or allow the processor 210 to communicate with an external memory. The storage interface may include one or more of a universal serial bus (USB) interface, a SATA (serial ATA) interface, a PATA (parallel ATA) interface, and a small computer system interface (SCSI). In a second example, the I / O device 240 may include a network interface that may facilitate and / or allow the processor 210 to communicate with a network. The I / O device 240 may include one or more of a wireless network interface and a wired network interface. In a third example, the I / O device 240 may include a peripheral component interconnect (PCI) interface, a PCI Express (PCIe) interface, a serial peripheral interconnect (SPI) interface, and an internal integrated circuit (I 2 C) one or more of an interface, etc. In another example, the I / O device 240 may facilitate and / or permit the processor 210 to communicate data with one or more of the display 160 and the MID 170 , etc.

[0038] As shown, the I / O device 240 can be communicatively coupled to the display 160 and the MID 170. For example, the computer system 150 can be communicatively coupled to the display 160 and the MID 170 via the I / O device 240. The I / O device 240 can facilitate and / or permit the processor 210 to communicate data with one or more elements of the MID 170. In one example, the I / O device 240 can facilitate and / or permit the processor 210 to communicate data with one or more of the image sensor 172, the distance sensor 174, and the display 162, etc. In another example, the I / O device 240 can facilitate and / or permit the processor 210 to control one or more of the image sensor 172, the distance sensor 174, the illuminator 178, and the display 162, etc.

[0039] Now turn to Figure 3A and Figure 3B , showing an example of a graphical user interface. As shown, the GUI 310 may include icons 320A-320C. For example, the GUI 310 and / or icons 320A-320C may be superimposed on an image acquired via the image sensor 172. As shown, the GUI 310 may display a cursor 330. For example, the system 100 may track the movement of the surgical tool instrument 120 and display the cursor 330 based on one or more movements and / or one or more positions of the surgical tool instrument 120. The system 100 may track the movement of the surgical tool instrument 120. For example, the system 100 may track one or more movements and / or one or more positions of the surgical tool instrument 120 to the icon 320C.

[0040] GUI 310 may be displayed via a display. For example, GUI 310 may be displayed via one or more of displays 160, 162A, and 162B, etc. Surgeon 110 may select icon 320C. In one example, surgeon 110 may select icon 320C via a foot pedal. Actuation of the foot pedal may be used as a pointer click (e.g., a mouse click). In another example, surgeon 110 may select icon 320C via one or more movements of surgical tool instrument 120. The one or more movements of surgical tool instrument 120 may be used as a pointer click (e.g., a mouse click).

[0041] Now turn to Figure 4A and Figure 4B , shows an example of an alignment region. Figure 4A As shown, the surgical tool instrument 120B can be aligned via the alignment area 410. For example, the alignment area 410 can be displayed via the GUI 310. Figure 4BAs shown, surgical tool instrument 120A can be aligned via alignment region 410. For example, alignment region 410 can be superimposed on the acquired image. The acquired image may be acquired via image sensors 172A and 172B.

[0042] A digital model of the surgical tool instrument 120 may be determined based on one or more images from one or more image sensors 172. The digital model of the surgical tool instrument 120 may include a pattern of the surgical tool instrument 120. As an example, the digital model may be used to associate image data of the surgical tool instrument 120 within an image acquired via one or more of the image sensors 172A and 172B. The digital model may include possible relationships between image data of the surgical tool instrument within an image acquired via one or more of the image sensors 172A and 172B. For example, the digital model may include parameters that may determine the possible relationships. The learning process and / or method may fit the parameters using training data. In one example, one or more images may be used as training data. In another example, the alignment region 410 may be used to associate image data as training data. Determining the digital model of the surgical tool instrument 120 may include: training the digital model based at least on the one or more images. The digital model may be discriminative. The digital model may be generative. One or more inference processes, and / or one or more methods may use the digital model to determine image data of the surgical tool instrument 120 within an image acquired via one or more of the image sensors 172A and 172B.

[0043] Now turn to Figure 5 , showing an example of a method for utilizing a surgical tool instrument with a graphical user interface. At 510, a graphical user interface can be displayed via a display. In one example, GUI 310 can be displayed via display 160. In another example, GUI 310 can be displayed via one or more of displays 162A and 162B. At 515, a first user input can be received, indicating that a surgical tool instrument will be used as a pointer associated with the graphical user interface. In one example, the first user input can include actuating a foot pedal. In a second example, the first user input can include voice input. In another example, the first user input can include actuating a GUI icon. The surgeon 110 or other medical staff can actuate the GUI icon.

[0044] At 520, a first plurality of images from at least one sensor may be received. For example, a first plurality of images from one or more of the image sensors 172A and 172B may be received. The first plurality of images may include image data of a surgical tool instrument to be used as a pointer associated with a graphical user interface. At 525, a digital model of a surgical tool instrument including a pattern of the surgical tool instrument may be determined based on the first plurality of images. For example, the digital model may be used to associate image data of the surgical tool instrument within an image acquired via one or more of the image sensors 172A and 172B. The digital model may include possible relationships between image data of the surgical tool instrument within an image acquired via one or more of the image sensors 172A and 172B. For example, the digital model may include parameters that may determine the possible relationships. The learning process and / or method may fit the parameters using training data. For example, the first plurality of images may be used as training data. Determining the digital model of the surgical tool instrument may include: training the digital model based at least on the first plurality of images. The digital model may be discriminative. The digital model may be generative. The inference process and / or method may utilize a digital model to determine image data of a surgical tool instrument within an image acquired via one or more of the image sensors 172A and 172B.

[0045] At 530, a second plurality of images may be received via the at least one image sensor. For example, a second plurality of images may be received from one or more of the image sensors 172A and 172B. The second plurality of images may include image data of the surgical tool instrument. At 535, a movement pattern of the surgical tool instrument may be determined based on the second plurality of images and the digital model, and the movement pattern may be used to select an icon of the graphical user interface. For example, a second plurality of images may be determined based on the second plurality of images and the digital model. Figure 4C The displayed pattern 420. The pattern 420 can be used to select the icon 320, such as Figure 4D As shown. Pattern 420 can be used to select icon 320, such as Figure 4E For example, at least a portion of the pattern 420 may be superimposed on the icon 420 .

[0046] Now turn to Figure 6, another example of a method for utilizing a surgical tool instrument having a graphical user interface is shown. At 610, a graphical user interface including at least one icon can be displayed via a display. In one example, GUI 310 can be displayed via display 160. In another example, GUI 310 can be displayed via one or more of displays 162A and 162B. At 615, a first image from an image sensor can be received. For example, a first image from image sensor 172 can be received. At 620, a first position of the surgical tool instrument within the first image can be determined based on the first image and a digital model of the surgical tool instrument. For example, a first position of surgical tool instrument 120 can be determined based on the first image and the digital model of the surgical tool instrument, such as Figure 3A As an example, the digital model may be or include Figure 5 The digital model determined by method element 525. As another example, the digital model can be retrieved from a storage medium. The storage medium can store one or more digital models of surgical tools and instruments. For example, the storage medium can store a library including one or more digital models of surgical tools and instruments.

[0047] At 625, a cursor of the graphical user interface at a second position associated with the first position may be displayed. Figure 3A As shown, the cursor 330 of the GUI 310 can be displayed at a second position associated with the first position. At 630, a second image can be received from the image sensor. For example, a second image can be received from the image sensor 172. At 635, a third position of the surgical tool instrument within the second image can be determined based on the second image and the digital model of the surgical tool instrument. For example, the second position of the surgical tool instrument 120 can be determined based on the second image and the digital model of the surgical tool instrument, such as Figure 3B At 640, a cursor of the graphical user interface may be displayed at a fourth position associated with the third position. Figure 3B As shown, the cursor 330 of the GUI 310 may be displayed at a fourth position associated with the third position.

[0048] At 645, user input may be received, the user input indicating a selection when the coordinates of the at least one icon include the fourth location. In one example, the user input may include actuating a foot pedal. When the user input indicates the selection, the surgeon 110 may actuate the foot pedal. In a second example, the user input may include a movement mode. The user input may include, for example, Figure 4D and Figure 4E The movement pattern 420 is shown. The movement pattern can be approximated by Figure 4D and Figure 4E Movement mode 420 shown. Other movement modes may be configured and / or utilized. In a third example, the user input may include a change in the number of pixels associated with the surgical tool instrument, the change in the number of pixels indicating a change in the distance of the surgical tool instrument to the image sensor. If the surgical tool instrument becomes closer to the image sensor, the number of pixels associated with the surgical tool instrument may increase. As an example, a third image from the image sensor may be received from the image sensor, and a change in the number of pixels associated with the surgical tool instrument indicating a change in the distance of the surgical tool instrument to the image sensor may be determined based at least on the second image and the third image and the digital model.

[0049] At 650, it can be determined that the user input indicates a selection when the coordinates of the at least one icon include a fourth position. For example, it can be determined that the user input indicates a selection of icon 320. At 655, the data displayed by the graphical user interface can be changed. For example, the image data of GUI 310 can be changed. Changing the data displayed by the graphical user interface can be performed in response to determining that the user input indicates a selection. As an example, a workflow associated with a surgeon can proceed to the next step of the workflow. The image data of GUI 310 can be changed in conjunction with the next step of the workflow associated with the surgeon. As another example, at least a portion of the first image, the second image, or the third image can be stored in response to determining that the user input indicates a selection. The image data can be present via a storage medium.

[0050] One or more elements of these methods and / or process elements, and / or one or more parts of the methods and / or process elements may be performed in a different order, may be repeated, or may be omitted. In addition, additional, supplementary and / or reproducible methods and / or process elements may be implemented, illustrated and / or performed as needed. In addition, one or more system elements may be omitted, and / or additional system elements may be added as needed.

[0051] The storage medium may be and / or may include an article of manufacture. For example, the article of manufacture may include and / or may be a software product and / or a program product. The storage medium may be decoded and / or encoded into instructions having one or more flow charts, systems, methods, and / or processes described herein for producing the article of manufacture.

[0052] The subject matter disclosed above should be considered illustrative and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments that fall within the true spirit and scope of the disclosure. Thus, to the maximum extent allowed by law, the scope of the disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents and shall not be restricted or limited to the foregoing detailed description.

Claims

1. A system comprising: at least one processor; as well as a memory medium coupled to the at least one processor and comprising instructions that, when executed by the at least one processor, cause the system to: displaying, via the display, a graphical user interface including at least one icon; receiving a first image from an image sensor; Determine a first position of the surgical tool within the first image based on the first image and the digital model of the surgical tool; displaying a cursor of the graphical user interface at a second position associated with the first position; receiving a second image from the image sensor; determining a third position of the surgical tool within the second image based on the second image and the digital model of the surgical tool; displaying a cursor of the graphical user interface at a fourth position associated with the third position; receiving a user input indicating a selection when the coordinates of the at least one icon include the fourth location, the receiving the user input comprising receiving a third image from the image sensor; determining that the user input indicates a selection when the coordinates of the at least one icon include the fourth position, the determining that the user input indicates a selection comprising determining a change in the number of pixels associated with the surgical tool instrument indicating a change in the distance of the surgical tool instrument from the image sensor based at least on the second image and the third image and the digital model; and In response to determining that the user input indicates a selection, data displayed by the graphical user interface is changed.

2. The system of claim 1, wherein the change in the number of pixels associated with the surgical tool instrument indicating the change in distance comprises: An increase in the number of pixels associated with the surgical tool instrument indicates that the surgical tool instrument is closer to the image sensor.

3. The system of claim 1, wherein: The receiving a user input indicative of a selection further comprises receiving a plurality of images from the image sensor; and Determining that the user input is indicative of a selection further includes determining a movement pattern associated with movement of the surgical tool instrument that is indicative of a selection based at least on the digital model and the plurality of images.

4. The system of claim 1, wherein: The instructions further cause the system to: At least a portion of the first image, the second image, or a third image is stored via the memory medium in response to determining that the user input indicates a selection.

5. The system of claim 1, wherein: The instructions further cause the system to: The workflow associated with the surgical procedure is advanced to the next step in the workflow.

6. The system of claim 1, further comprising: Microscope integrated display; Wherein, the microscope integrated display includes the display.

7. A system comprising: at least one processor; as well as a memory medium coupled to the at least one processor and comprising instructions that, when executed by the at least one processor, cause the system to: displaying, via the display, a graphical user interface including at least one icon; receiving a first image from an image sensor; Determine a first position of the surgical tool within the first image based on the first image and the digital model of the surgical tool; displaying a cursor of the graphical user interface at a second position associated with the first position; receiving a second image from the image sensor; determining a third position of the surgical tool within the second image based on the second image and the digital model of the surgical tool; displaying a cursor of the graphical user interface at a fourth position associated with the third position; receiving a user input indicating a selection when the coordinates of the at least one icon include the fourth location, the receiving the user input indicating a selection comprising receiving a plurality of images from the image sensor; determining that the user input indicates a selection when the coordinates of the at least one icon include the fourth location, the determining that the user input indicates a selection comprising determining a movement pattern associated with movement of the surgical tool instrument that indicates a selection based at least on the digital model and the plurality of images; and In response to determining that the user input indicates a selection, data displayed by the graphical user interface is changed.

8. The system of claim 7, wherein the movement pattern is used as pointer clicks.

9. The system of claim 7, wherein the movement mode is used as a left mouse button selection.

10. The system of claim 7, wherein the movement mode is used as a right mouse button selection.

11. The system of claim 7, wherein the movement mode is used to select the at least one icon by at least partially superimposing on the at least one icon.

12. The system of claim 7, wherein: The receiving user input further comprises receiving a third image from the image sensor; and Determining that the user input indicates a selection further includes determining a change in the number of pixels associated with the surgical tool instrument that indicates a change in the distance of the surgical tool instrument from the image sensor based on at least the second image and the third image and the digital model.

13. The system of claim 7, wherein: The instructions further cause the system to: At least a portion of the first image, the second image, or a third image is stored via the memory medium in response to determining that the user input indicates a selection.

14. The system of claim 7, wherein: The instructions further cause the system to: The workflow associated with the surgical procedure is advanced to the next step in the workflow.

15. The system of claim 7, further comprising: Microscope integrated display; Wherein, the microscope integrated display includes the display.