Determination of surgical tool tip and orientation

Abstract

Embodiments generally relate to determining the position of a tool tip of a surgical instrument and the orientation of the surgical instrument. In some embodiments, a method includes receiving at least one image frame of a plurality of image frames. The method also includes detecting a surgical tool in at least one image frame. The method also includes determining the position of the tip of the surgical tool. The method also includes determining the orientation of the surgical tool.

Description

[0001] Related fields

[0002] Computer-assisted surgery uses computer technology to guide or perform medical procedures, such as those involving endoscopy, laparoscopy, etc. During surgery, the surgeon may need to use a variety of tools to perform the procedure. Cameras and monitors can help the surgeon operate multiple surgical tools using robotic arms. However, it is difficult to get a good view of the surgical tools on the monitor during surgery. Summary of the invention

[0003] Embodiments generally relate to determining the position of a tool tip of a surgical instrument and the orientation of the surgical instrument. In some embodiments, a system includes one or more processors and includes logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors. When executed, the logic is operable to cause the one or more processors to perform operations including receiving at least one image frame of a plurality of image frames; detecting a surgical tool in at least one image frame; determining the position of the tip of the surgical tool; and determining the orientation of the surgical tool.

[0004] Further with respect to the system, in some embodiments, the logic, when executed, is further operable to perform operations including generating a plurality of key points of a surgical tool, and wherein each of the plurality of key points corresponds to a different portion of the surgical tool. In some embodiments, the logic, when executed, is further operable to perform operations including determining a plurality of key points of a surgical tool, wherein at least one of the plurality of key points corresponds to the tip of the surgical tool; and determining the position of the tip of the surgical tool according to at least one key point corresponding to the tip of the surgical tool. In some embodiments, the logic, when executed, is further operable to perform operations including generating a plurality of key points of a surgical tool and determining the orientation of the surgical tool according to the plurality of key points. In some embodiments, the logic, when executed, is further operable to perform operations including determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real time. In some embodiments, the logic, when executed, is further operable to perform operations including generating a plurality of key points of a surgical tool and determining a key point confidence score for each key point of the surgical tool. In some embodiments, the logic, when executed, is further operable to perform operations including determining the position of the tip of the surgical tool and determining the orientation of the surgical tool based on deep learning.

[0005] In some embodiments, a non-transitory computer-readable storage medium having program instructions thereon is provided. When executed by one or more processors, the instructions are operable to cause the one or more processors to perform operations including receiving at least one image frame of a plurality of image frames; detecting a surgical tool in at least one image frame; determining a position of a tip of the surgical tool; and determining an orientation of the surgical tool.

[0006] Further with respect to the computer-readable storage medium, in some embodiments, the instructions, when executed, are further operable to perform operations including generating a plurality of key points of a surgical tool, and wherein each of the plurality of key points corresponds to a different portion of the surgical tool. In some embodiments, the instructions, when executed, are further operable to perform operations including generating a plurality of key points of a surgical tool, wherein at least one of the plurality of key points corresponds to the tip of the surgical tool; and determining the position of the tip of the surgical tool according to at least one key point corresponding to the tip of the surgical tool. In some embodiments, the instructions, when executed, are further operable to perform operations including generating a plurality of key points of a surgical tool and determining the orientation of the surgical tool according to the plurality of key points. In some embodiments, the instructions, when executed, are further operable to perform operations including determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real time. In some embodiments, the instructions, when executed, are further operable to perform operations including generating a plurality of key points of a surgical tool and determining a key point confidence score for each key point of the surgical tool. In some embodiments, the instructions, when executed, are further operable to perform operations including determining the position of the tip of the surgical tool and determining the orientation of the surgical tool based on deep learning.

[0007] In some embodiments, the method includes receiving at least one image frame of a plurality of image frames. The method also includes detecting a surgical tool in at least one image frame. The method also includes determining a position of a tip of the surgical tool. The method also includes determining an orientation of the surgical tool.

[0008] Further with respect to the method, in some embodiments, the method further includes generating a plurality of key points of the surgical tool, and wherein each of the plurality of key points corresponds to a different portion of the surgical tool. In some embodiments, the method further includes determining a plurality of key points of the surgical tool, wherein at least one of the plurality of key points corresponds to the tip of the surgical tool; and determining the position of the tip of the surgical tool based on at least one key point corresponding to the tip of the surgical tool. In some embodiments, the method further includes generating a plurality of key points of the surgical tool; and determining the orientation of the surgical tool based on the plurality of key points. In some embodiments, the method further includes determining the position of the tip of the surgical tool and determining the orientation of the surgical tool in real time. In some embodiments, the method further includes generating a plurality of key points of the surgical tool; and determining a key point confidence score for each key point of the surgical tool.

[0009] The nature and advantages of specific embodiments disclosed herein may be further understood by reference to the remainder of the specification and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A block diagram of an example operating environment is illustrated, which may be used with implementations described herein.

[0011] Figure 2 An example flow chart for determining a position of a tool tip of a surgical instrument and an orientation of the surgical instrument is illustrated in accordance with some embodiments.

[0012] Figure 3 An example image frame including a surgical tool and a bounding box is illustrated in accordance with some embodiments.

[0013] Figure 4 An example image frame including a surgical tool and key points is illustrated in accordance with some embodiments.

[0014] Figure 5 An example flow chart for determining a position of a tool tip of a surgical instrument and an orientation of the surgical instrument is illustrated in accordance with some embodiments.

[0015] Figure 6 An example flow chart for determining a position of a tool tip of a surgical instrument is illustrated in accordance with some embodiments.

[0016] Figure 7 A block diagram of an example network environment is illustrated, which may be used with some implementations described herein.

[0017] Figure 8 A block diagram of an example computing system is illustrated, which may be used with some implementations described herein. DETAILED DESCRIPTION

[0018] Embodiments generally relate to determining the position of a tool tip of a surgical instrument and the orientation of the surgical instrument. As described in more detail herein, a system receives an image frame from a camera. In various embodiments, the image frame may be one of a series of image frames of a video stream. The system detects a surgical instrument or tool in at least one image frame. The system further determines the position of the tip of the surgical tool. The system also determines the orientation of the surgical tool.

[0019] Figure 1 A block diagram of an example work environment 100 that can be used for the embodiments described herein is illustrated. A system 102 is illustrated that performs various embodiments described herein. The system 102 controls a camera 104 that captures a video of a work area 106 or surgical area. The video can be a surgical video recorded by the camera 104. The system 102 causes the camera 104 to send a video stream from the camera 104 to a viewer client 108 via a network 110. In various embodiments, the video can be used to facilitate a user, such as a surgeon during a laparoscopic procedure. For example, when a surgeon handles a surgical tool 112 shown in the work area 106 to interact with tissue, the surgeon can view a visual image of the surgical tool 112 interacting with the tissue. In various embodiments, the surgeon can also view a visual image of the surgical tool 112, whether the surgeon is directly operating the surgical tool 112 with the surgeon's hand and / or a robotic arm. As described in more detail herein, the system 102 analyzes the characteristics of the surgical tool 112 captured by the camera 104.

[0020] Figure 2 An example flow chart for determining the position of a tool tip of a surgical instrument and the orientation of a surgical instrument according to some embodiments is illustrated. As described in more detail below, the system performs various functions including determining the position of a tool tip of a surgical instrument. Figure 1 and Figure 2 Both methods begin at block 202, where a system such as system 102 receives at least one image frame. In various embodiments, the received image frame is from a sequence of image frames accessed by the system or obtained from a video stream. For example, the video stream can be a surgical video.

[0021] At block 204, system 102 detects a surgical tool, such as surgical tool 112, in an image frame. In some implementations, the system first detects one or more objects in a received image frame. System 102 may use object recognition techniques to detect specific objects in a received image.

[0022] Figure 3An example image frame 302 is illustrated including a surgical tool 112 and a bounding box 304 according to some implementations. As shown, in various implementations, the system initially detects the surgical tool 112 or target image in the scene shown in the image frame 302 within the bounding box 304.

[0023] Bounding box 304 is a visual indicator that demarcates surgical tool 112. Although bounding box 304 is shown as a rectangle, the actual shape and size of bounding box 304 may vary depending on the shape and size of the surgical tool.

[0024] At block 206, the system 102 determines the location of the tip of the surgical tool. In various implementations, the system determines the location of the tip of the surgical tool based on key points generated by the system.

[0025] Figure 4 An example image frame 302 including a surgical tool 112 and key points is illustrated in accordance with some embodiments. Key points 402, 404, 406, and 408 are illustrated at various locations on the surgical tool 112. The key points 402 and 404 are placed or positioned at the tip of the surgical tool 112. In this particular example, the surgical tool 112 includes scissors 410, and the key points 402 and 404 are located at the tip of the scissors 410.

[0026] Although four key points are shown, the specific number of key points may vary depending on the number, configuration, and shape of the components of the surgical tool. In addition, the specific number and location of the key points may vary depending on the number, configuration, and shape of the components of the surgical tool. Figure 5 Example embodiments directed to determining the position of a tip of a surgical tool are described in further detail.

[0027] At block 208, the system 102 determines the orientation of the surgical tool 112. In various embodiments, the system determines the position of the surgical tool tip based on the key points generated by the system. Figure 4 , key points 406 and 408 define the orientation 412 of surgical tool 112. Figure 6 Example embodiments directed to determining the orientation of a surgical tool tip are described in further detail.

[0028] Although steps, operations or calculations may be presented in a particular order, the order may be changed in a particular embodiment. Other orders of steps are possible depending on the particular embodiment. In some particular embodiments, multiple steps shown as sequential in this specification may be performed simultaneously. In addition, some embodiments may not have all the steps shown and / or may have other steps that replace or are in addition to those steps shown herein.

[0029] The position and orientation of the surgical tool tip are important information for surgeons and can be used in many applications. For example, for safety purposes, the surgical tool tip and orientation can provide visual feedback to the surgeon. In some embodiments, knowing the position of the surgical tool tip, the system can zoom in on the relevant working area so that the surgeon can better view the interaction between the surgical tool and the target tissue. In other words, the embodiment provides the surgeon with the best viewing angle. This minimizes or eliminates the situation where the surgical tool tip is blocked by other objects (e.g., non-target tissues) or other tools (e.g., auxiliary tools in the working or surgical area). In some embodiments, when the system tracks the movement of the surgical tool tip, the system can locally enhance the image quality around the surgical tool tip of the tool.

[0030] Embodiments also facilitate evaluation of surgeon performance during training. For example, the information may be used to determine the accuracy and / or efficiency of a surgeon during surgery. Such information may be used to compare the performance of a surgeon in training with the performance of an experienced surgeon. In addition, positioning of the surgical tool tip enables proximity estimation of a surgical target (e.g., the retina of an eye, etc.) by aligning cross-sectional views in applications such as optical coherence tomography (OCT). Other example application areas may include natural user interface design, automatic warning systems, semantic feedback systems, skill assessment systems, and the like.

[0031] Figure 5 An example flow chart for determining the position of a tool tip of a surgical instrument and the orientation of the surgical instrument according to some embodiments is illustrated. Figure 4 and Figure 5 Both, the method begins at block 502, where a system such as system 102 detects an object such as surgical tool 112 in image frame 302. In some embodiments, the system first detects one or more objects in the received image frame. System 102 can use object recognition technology to detect the object in image frame 302.

[0032] At block 504, the system determines properties or characteristics of the surgical tool 112. For example, the properties of the surgical tool 112 may include components of the surgical tool 112, a configuration of the surgical tool 112 and its components, a shape of the surgical tool 112, etc. Depending on the surgical tool or target object detected and identified in the image frame, the specific properties of the surgical tool 112 determined by the system may vary.

[0033] At block 506, the system generates key points 402, 404, 406, and 408 of the surgical tool. In various embodiments, the system generates key points based on the attributes of the surgical tool 112. In various embodiments, each key point corresponds to a different part of the surgical tool. For example, Figure 4As shown, key points 402, 404, 406, and 408 correspond to points or locations of surgical tool 112. In some embodiments, the system matches key points 402-408 to predefined key points of a known type of surgical tool, where the number of key points and the relative positions of the key points correspond to predefined points or locations of the known surgical tool.

[0034] At block 508, the system determines the location of the tip of the surgical tool 112 based on the key points 402, 404, 406, and 408. For example, at least one of the key points corresponds to the tip of the surgical tool. The system 102 determines the location of the tip of the surgical tool based on a key point corresponding to the tip of the surgical tool.

[0035] Depending on the particular implementation, the number of tool tips may vary. For example, in the illustrated implementation, surgical tool 112 includes scissors 410. The system may identify scissors 410 based on the shape of that portion of surgical tool 112. The system may also determine that key point 402 is a tool tip and key point 404 is a tool tip.

[0036] Although steps, operations or calculations may be presented in a particular order, the order may be changed in a particular embodiment. Other orders of steps are possible depending on the particular embodiment. In some particular embodiments, multiple steps shown as a sequence in this specification may be performed simultaneously. In addition, some embodiments may not have all the steps shown and / or may have other steps that replace or are in addition to those steps shown herein.

[0037] Figure 6 An example flow chart for determining the position of a tool tip of a surgical instrument according to some embodiments is illustrated. Figure 4 and Figure 6 Both, the method begins at block 602, where a system such as system 102 detects an object such as surgical tool 112 in image frame 302. In some implementations, the system first detects one or more objects in a received image frame. System 102 can detect objects in image frame 302 using object recognition techniques.

[0038] At block 604, the system determines the attributes or characteristics of the surgical tool 112. The system may be similar to the combination of Figure 5 Block 504 describes an embodiment of determining such attributes or characteristics in a manner.

[0039] At block 606, the system generates key points 402, 404, 406, and 408 based on the attributes of the surgical tool 112. The system may be similar to the combination of Figure 5Such attributes or characteristics are determined in the manner of embodiments described in block 506. In various embodiments, key points may be used as landmarks for mapping portions of a surgical tool.

[0040] At block 608, the system determines the relative positions of keypoints 402, 404, 406, and 408. For example, the system can determine the distances between different pairs of keypoints.

[0041] At block 610, the system determines the orientation of the surgical tool 112. In various embodiments, the system determines the orientation of the surgical tool based on a plurality of key points. The system can determine a straight line between key points (e.g., key point 406 and key point 406), and determine that the portion of the surgical tool between the two key points is a handle or shaft. For example, the system can determine that key point 406 is at the distal end of the shaft of the surgical tool 112, and key point 408 is at the proximal end of the shaft of the surgical tool 112.

[0042] For ease of explanation, the orientation of surgical tool 112 is generally indicated by the orientation of the axis. In various embodiments, the system also determines the orientation of multiple parts of surgical tool 112. For example, the system can determine the orientation of each blade of scissors 410 when scissors 410 are opened and closed and when surgical tool 112 moves overall.

[0043] The system can also determine a series or line of key points to determine the length of the surgical tool and / or the length of a portion of the surgical tool. If the surgical tool has multiple parts, such as a main body and a special part (e.g., scissors, graspers, etc.), the system can also determine one or more angles between two or more parts of the surgical tool.

[0044] Although steps, operations or calculations may be presented in a particular order, the order may be changed in a particular embodiment. Other orders of steps are possible depending on the particular embodiment. In some particular embodiments, multiple steps shown as a sequence in this specification may be performed simultaneously. In addition, some embodiments may not have all the steps shown and / or may have other steps that replace or are in addition to those steps shown herein.

[0045] In various embodiments, the location or position of the surgical tool tip and the orientation of the surgical tool are two-dimensional (2D) relative to the image frame and other objects (e.g., tissue, auxiliary or secondary tools, etc.). This helps the surgeon handle the surgical tool 112 and minimize or eliminate the tip of the surgical tool 112 being blocked by other objects. In some embodiments, the system can use a second camera and / or deep learning to determine the three-dimensional (3D) orientation of the target surgical tool. In various embodiments, the system 102 determines the position of the tip of the surgical tool and determines the orientation of the surgical tool in real time.

[0046] The system achieves high accuracy by using topological relationships between key points and key point validation to eliminate incorrect key point predictions. For example, in various embodiments, the system performs a validation process that is performed to eliminate incorrect key point predictions. The system can perform such a validation process based on the topological relationships between key points. For example, the system can determine the distance between two or more key points. If a given key point is substantially far away from a cluster or line of other key points, the system can treat the given key point as an outlier and can ignore the given key point. The system can apply these and similar techniques to different parts of the target surgical tool (e.g., one or more tool tips, tool components (e.g., tool shafts), etc.).

[0047] In some embodiments, the system can determine a key point confidence score for each key point of the surgical tool. In various embodiments, each key point confidence score indicates the level of accuracy of the determination of the corresponding key point. The confidence scores can range from low to high (e.g., 0.0 to 1.0, 0% to 100%, etc.). In some embodiments, key points associated with confidence scores below a predetermined confidence threshold can be considered as outliers to be ignored. Depending on the specific embodiment, the predetermined confidence threshold can vary. In some embodiments, the system can rank the confidence scores of the key points and eliminate the key points associated with confidence scores below a predetermined ranking threshold. Depending on the specific embodiment, the predetermined ranking threshold can vary.

[0048] In various embodiments, the system uses deep learning for key point estimation and tool detection on the surgical tool. Based on the information associated with the key point estimation and tool detection on the surgical tool, the system further determines the position of the tip of the surgical tool and determines the orientation of the surgical tool. In some embodiments, the position of the tip of the surgical tool and the orientation of the surgical tool can also be determined at least in part based on deep learning. In various embodiments, the system can process thousands of image frames to build a training data set to detect and identify features of the target surgical tool in real time. This achieves the best results for key point prediction verification by integrating topological information.

[0049] In various embodiments, the system utilizes a deep learning network to detect and classify objects in image frames including target surgical tools. The system can employ deep learning techniques to classify tools, which provides a lightweight model for accuracy and speed. The system can also use deep learning to distinguish between primary or dominant surgical tools (e.g., scissors) and secondary or auxiliary surgical tools (e.g., graspers). For example, a surgeon can use a grasper to grab a specific tissue to pull away from tissue to be cut by scissors or large shears.

[0050] In various embodiments, the system utilizes a deep learning network to classify objects into various tool classifications. In some embodiments, the system uses a classifier that is trained using known features learned by the deep learning network. The system uses the known features to determine the type of tool or tool classification based on features recognized by the system in the image frame.

[0051] Although various embodiments are described in the context of example surgical tools being scissors or large scissors, these embodiments and other embodiments are also applicable to various other types of tools. For example, tools may include cutting or dissecting instruments, such as scalpels, scissors, saws, etc. Tools may include bipolar forceps and irrigators. Tools may include grasping or holding instruments, such as smooth and toothed forceps, towel clips, vascular clamps, organ stents, etc. Tools may include hemostatic instruments, such as clips, hemostats, atraumatic hemostats, etc. Tools may include retractor instruments, such as C-shaped laminar flow hooks, blunt tooth hooks, sharp tooth hooks, groove probes, tamping forceps, etc. Tools may include tissue unification instruments and materials, such as needle holders, surgical needles, staplers, clips, tape, etc. The specific tools detected may vary and will depend on the specific embodiment. Although embodiments are described herein in the context of surgical tools, these embodiments and other embodiments may also be applied to other tools (e.g., non-surgical tools).

[0052] Real-time detection and identification of a dominant tool (e.g., a dominant surgical tool) facilitates various natural user interfaces. For example, given the location of the dominant tool, the camera can automatically move toward the tool to re-center the tool. This provides a better view for the user (e.g., a surgeon, etc.). In various embodiments, different types of models can be trained to locate the dominant tool in each frame of the video. Thus, all predictions can be systematically combined to produce a final result.

[0053] Embodiments described herein provide various benefits. For example, embodiments determine the position or location of one or more surgical tool tips during surgery. Embodiments also determine the orientation of surgical tools, including the orientation of various components of surgical tools during surgery.

[0054] The embodiments described herein can be used for various applications in computer-based assistance for different types of surgery. For example, the embodiments can be applied to procedures involving endoscopy and laparoscopy. The embodiments can be used to automatically adjust the camera during surgery (e.g., place the surgical tool tip in the center of the image for autofocus, etc.). The embodiments can be used with a robotic arm during surgery (e.g., a camera holder, a tool holder, etc.). The embodiments can also be used to provide warnings (e.g., if the surgical tool tip is too close to a specific part of an organ, etc.).

[0055] Figure 7A block diagram of an example network environment 700 is illustrated, which can be used for some embodiments described herein. In some embodiments, the network environment 700 includes a system 702, which includes a server device 704 and a network database 706. The network environment 700 also includes client devices 710, 720, 730, and 740, which can communicate with each other directly or through the system 702. The client devices 710, 720, 730, and 740 can include viewer clients, cameras, and other surgery-related devices. The network environment 700 also includes a network 750.

[0056] For ease of explanation, Figure 7 One block for each of the system 702, server device 704, and network database 706 is illustrated, and four blocks are shown for client devices 710, 720, 730, and 740. Although some embodiments are described in the context of one client device being used to view a video of a surgical procedure (e.g., one surgeon viewing the video), these and other embodiments may be applied to multiple client devices. For example, there may be other doctors, and / or other clinicians, and / or students viewing the video.

[0057] Blocks 702, 704 and 706 may represent multiple systems, server devices and network databases. In addition, there may be any number of client devices. In other embodiments, network environment 700 may not have all of the components shown and / or may have other elements including other types of elements (instead of those components shown herein, or in addition to those components shown herein). In various embodiments, users U1, U2, U3 and U4 may interact with each other or with system 702 using respective client devices 710, 720, 730 and 740.

[0058] In various implementations described herein, a processor of system 702 and / or a processor of any of client devices 710, 720, 730, and 740 causes elements (eg, information, etc.) described herein to be displayed in a user interface on one or more display screens.

[0059] The embodiments may be applied to any network system and / or may be applied locally to a single user. For example, the embodiments described herein may be implemented by the system 702 and / or any of the client devices 710, 720, 730, and 740. The system 702 may execute the embodiments described herein on a stand-alone computer, a tablet computer, a smart phone, etc. Any of the system 702 and / or the client devices 710, 720, 730, and 740 may execute the embodiments described herein alone or in combination with other devices.

[0060] Figure 8A block diagram of an example computer system 800 is shown, which may be used with some of the embodiments described herein. For example, the computer system 800 may be used to implement Figure 1 System 102 and / or Figure 7 , and perform the embodiments described herein. In some embodiments, the computer system 800 may include a processor 802, an operating system 804, a memory 806, and an input / output (I / O) interface 808. In various embodiments, the processor 802 can be used to implement the various functions and features described herein, as well as to perform the method embodiments described herein. Although the processor 802 is described as performing the embodiments described herein, any suitable component or combination of components of the computer system 800 or any suitable processor or multiple processors associated with the computer system 800 or any suitable system may perform the described steps. The embodiments described herein may be performed on a user device, on a server, or a combination of both.

[0061] The computer system 800 also includes software applications 810, which may be stored in the memory 806 or any other suitable storage location or computer-readable medium. The software applications 810 provide instructions that enable the processor 802 to perform the embodiments described herein and other functions. The software applications may also include an engine (such as a network engine for performing various functions associated with one or more networks and network communications). The components of the computer system 800 may be implemented by any combination of one or more processors or hardware devices, as well as any combination of hardware, software, firmware, etc.

[0062] For ease of explanation, Figure 8 A block is shown for each of processor 802, operating system 804, memory 806, I / O interface 808, and software application 810. These blocks 802, 804, 806, 808, and 810 may represent multiple processors, operating systems, memories, I / O interfaces, and software applications. In various implementations, computer system 800 may not have all of the components shown and / or may have other elements including other types of components (instead of or in addition to those shown herein).

[0063] Although the specification has been described with respect to specific embodiments thereof, these specific embodiments are merely illustrative and not restrictive. The concepts illustrated in the examples may be applied to other examples and implementations.

[0064] In various embodiments, the software is encoded in one or more non-transitory computer readable media for execution by one or more processors. The software, when executed by one or more processors, is operable to perform the embodiments described herein and other functions.

[0065] Any suitable programming language (including C, C++, Java, assembly language, etc.) can be used to implement the routine of a specific embodiment. Different programming techniques can be adopted, such as process-oriented or object-oriented. The routine can be executed on a single processing device or multiple processors. Although the steps, operations or calculations can be presented in a specific order, the order can be changed in different specific embodiments. In some specific embodiments, multiple steps shown as successive in this specification can be executed simultaneously.

[0066] Particular embodiments may be implemented in a non-transitory computer-readable storage medium (also referred to as a machine-readable storage medium) for use with or in conjunction with an instruction execution system, apparatus, or device. Particular embodiments may be implemented in the form of control logic in software or hardware or a combination of both. When executed by one or more processors, the control logic is operable to perform the embodiments described herein and other functions. For example, a tangible medium such as a hardware storage device may be used to store the control logic, which may include executable instructions.

[0067] Particular embodiments may be implemented using programmable general purpose digital computers and / or by using application specific integrated circuits, programmable logic devices, field programmable gate arrays, optical, chemical, biological, quantum or nano-engineered systems, components and mechanisms. In general, the functionality of a particular embodiment may be implemented by any means known in the art. Distributed, networked systems, components and / or circuits may be used. The communication or transmission of data may be wired, wireless or any other means.

[0068] A "processor" may include any suitable hardware and / or software system, mechanism or component that processes data, signals or other information. A processor may include a system having a general-purpose central processing unit, multiple processing units, a dedicated circuit or other system for implementing a function. Processing does not have to be limited to a geographic location or have time constraints. For example, a processor may perform its functions in "real time", "offline", "batch mode", etc. Parts of the processing may be performed by different (or the same) processing systems at different times and in different locations. A computer may be any processor that communicates with a memory. The memory may be any suitable data storage, memory and / or non-transitory computer-readable storage medium, including electronic storage devices such as random access memory (RAM), read-only memory (ROM), magnetic storage devices (hard drives, etc.), flash memory, optical storage devices (CDs, DVDs, etc.), magnetic disks or optical disks, or other tangible media suitable for storing instructions (e.g., programs or software instructions) executed by a processor. For example, a tangible medium such as a hardware storage device may be used to store control logic, which may include executable instructions. The instructions may also be contained in and provided as electronic signals, such as in the form of software as a service (SaaS) delivered from a server (eg, a distributed system and / or a cloud computing system).

[0069] It should also be understood that one or more elements depicted in the drawings / figures may also be implemented in a more separate or integrated manner, or even removed or rendered inoperable in some cases (depending on the needs of a particular application). It is also within the spirit and scope to implement a program or code that can be stored in a machine-readable medium to allow a computer to perform any of the above methods.

[0070] As used in the specification and claims that follow, “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Also, as used in the specification and claims that follow, the meaning of “in” includes “in” and “on” unless the context clearly dictates otherwise.

[0071] Thus, although specific embodiments have been described herein, a range of modifications, various changes and substitutions are contemplated in the foregoing disclosure, and it should be understood that in some cases some features of specific embodiments will be employed without corresponding use of other features without departing from the scope and spirit set forth. Thus, many modifications may be made to adapt a particular situation or material to the basic scope and spirit.

Claims

1. A system, include: one or more processors; and Logic encoded in one or more non-transitory computer-readable storage media for execution by the one or more processors and, when executed, operable to perform operations including: receiving at least one image frame of a plurality of image frames; detecting a surgical tool in the at least one image frame; determining one or more properties of the surgical tool, wherein the one or more properties include one or more components of the surgical tool, a configuration of the surgical tool, a configuration of the one or more components of the surgical tool, and a shape of the surgical tool; generating a plurality of key points based on the one or more attributes of the surgical tool, wherein each key point in the plurality of key points corresponds to a different portion of the surgical tool, wherein at least one key point in the plurality of key points corresponds to a tip of the surgical tool; determining a position of a tip and a shaft of a surgical tool based on the plurality of key points; and determining an orientation of the tip and shaft of the surgical tool based on the plurality of key points, wherein the orientation includes a distal end of the shaft and a proximal end of the shaft, and wherein one of the key points is at the distal end of the shaft of the surgical tool and another of the key points is at the proximal end of the shaft of the surgical tool, The operations further include eliminating incorrect keypoint predictions by using a topological relationship between the keypoints and a validation process, and wherein the validation process is based on the topological relationship between the keypoints.

2. The system of claim 1 , wherein the logic, when executed, is further operable to perform operations comprising: The position of the tip of the surgical tool is determined based on at least one key point corresponding to the tip of the surgical tool.

3. The system of claim 1, wherein the logic, when executed, is further operable to perform operations including determining a position of the tip of the surgical tool and determining an orientation of the surgical tool in real time.

4. The system of claim 1 , wherein the logic, when executed, is further operable to perform operations comprising: A keypoint confidence score is determined for each keypoint of the surgical tool.

5. The system of claim 1, wherein the logic, when executed, is further operable to perform operations including determining a position of a tip of the surgical tool and determining an orientation of the surgical tool based on deep learning.

6. A non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions being operable when executed by one or more processors to perform operations comprising: receiving at least one image frame of a plurality of image frames; detecting a surgical tool in the at least one image frame; determining one or more properties of the surgical tool, wherein the one or more properties include one or more components of the surgical tool, a configuration of the surgical tool, a configuration of the one or more components of the surgical tool, and a shape of the surgical tool; generating a plurality of key points based on the one or more attributes of the surgical tool, wherein each key point in the plurality of key points corresponds to a different portion of the surgical tool, wherein at least one key point in the plurality of key points corresponds to a tip of the surgical tool; determining a position of a tip and a shaft of a surgical tool based on the plurality of key points; and determining an orientation of the tip and shaft of the surgical tool based on the plurality of key points, wherein the orientation includes a distal end of the shaft and a proximal end of the shaft, and wherein one of the key points is at the distal end of the shaft of the surgical tool and another of the key points is at the proximal end of the shaft of the surgical tool, The operations further include eliminating incorrect keypoint predictions by using a topological relationship between the keypoints and a validation process, and wherein the validation process is based on the topological relationship between the keypoints.

7. The computer-readable storage medium of claim 6, wherein the instructions, when executed, are further operable to perform operations comprising: The position of the tip of the surgical tool is determined based on at least one key point corresponding to the tip of the surgical tool.

8. The computer-readable storage medium of claim 6, wherein the instructions, when executed, are further operable to perform operations including determining a position of the tip of the surgical tool and determining an orientation of the surgical tool in real time.

9. The computer-readable storage medium of claim 6, wherein the instructions, when executed, are further operable to perform operations comprising: A keypoint confidence score is determined for each keypoint of the surgical tool.

10. The computer-readable storage medium of claim 6, wherein the instructions, when executed, are further operable to perform operations including determining a position of a tip of the surgical tool and determining an orientation of the surgical tool based on deep learning.

Images