Computer-aided annotation of subsurface structures onto the surface
By generating a projection on the surface of the anatomical structure that most closely resembles the subsurface structure, the problem of inaccurate subsurface structure projection in existing technologies is solved, improving the precision and safety of surgery and optimizing the use of surgical resources.
Patent Information
- Application Number
- CN202580026190.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-01
- Filing Date
- 2025-03-25
- Publication Date
- 2026-11-10
AI Technical Summary
In the prior art, the rendering or annotation of subsurface structures is often projected onto a non-optimal location on the surface of the anatomical structure, making it difficult for surgeons to clearly visualize and determine the best path to the subsurface structure.
By receiving data captured by imaging devices using different imaging modalities, the point on the surface of the anatomical structure that is closest to the subsurface structure is determined, and a projection of the subsurface structure is generated at that point. The projection characteristics, such as color and blur, are used to indicate distance, providing a more accurate entry point.
It provides surgeons with better access to subsurface structures, improving surgical precision and safety, reducing errors and recovery time, and optimizing the use of surgical resources.
Smart Images

Figure CN123076356A_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This application claims the benefit of co-pending U.S. Provisional Patent Application Serial No. 63 / 572,406, filed April 1, 2024. All of the aforementioned related patent applications are incorporated herein by reference in their entirety. Technical Field
[0003] This disclosure generally relates to medical systems. Specifically, this disclosure relates to using medical systems to annotate subsurface structures onto surfaces. Background Technology
[0004] During surgery, it is crucial for surgeons to adequately visualize the subject's anatomy. Since surgical targets are often located within the subsurface, a range of visualization techniques have been investigated. This includes visualization of nerves, blood vessels, and tumors. Ultrasound has been used to provide registration visualization of tumors in robotic surgery, but it suffers from noise, poor sensitivity, and specificity, and is primarily used to locate large tumors deeply embedded below the surface or to guide instruments to said large tumors. Therefore, even with ultrasound employing stereoscopic endoscopy, it is difficult to clearly visualize subsurface structures at different depths. Stereoscopic endoscopy utilizes a stereoscopic imaging device. A stereoscopic imaging device comprises two imaging units (e.g., a left imaging unit and a right imaging unit). Triangulation techniques can be performed using images from both imaging units to determine the depth of different points in the field of view, assisting the medical system in projecting renderings or annotations of subsurface structures onto the surface of the anatomical structure. However, the renderings or annotations of subsurface structures are often projected onto suboptimal locations within the surface of the anatomical structure. Summary of the Invention
[0005] This disclosure describes a system and method for annotating a subsurface structure onto the surface of a surface structure. According to an embodiment, a system for annotating a subsurface structure onto the surface of a surface structure includes a memory and a controller communicatively coupled to the memory. The system receives first imaging data captured using a first imaging device in a first imaging mode, the first imaging data illustrating the surface of the surface structure; receives second imaging data captured using a second imaging device in a second imaging mode, the second imaging data illustrating the first surface structure; determines a first point on the surface based at least on where the first surface structure is closest; generates a first projection of the first surface structure onto the surface at the first point; and displays the first projection at the first point on the surface of the surface structure on a display device.
[0006] According to another embodiment, a method for annotating a subsurface structure onto the surface of a surface structure includes: receiving first imaging data captured by a first imaging device using a first imaging mode, the first imaging data showing the surface of the surface structure; receiving second imaging data captured by a second imaging device using a second imaging mode, the second imaging data showing the first surface structure; determining a first point on the surface based at least on where on the surface is closest to the first surface structure; generating a first projection of the first surface structure onto the surface at the first point; and displaying the first projection at the first point on the surface of the surface structure on a display device.
[0007] The foregoing general description and the following detailed description are exemplary and illustrative in nature and are intended to provide an understanding of this disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description
[0008] Figure 1A An example medical system is shown.
[0009] Figure 1B and Figure 1C It shows Figure 1A Example components in the system.
[0010] Figure 2A An example medical system is shown.
[0011] Figure 2B and Figure 2C It shows Figure 2A Example medical device systems in the system.
[0012] Figure 3 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0013] Figure 4 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0014] Figure 5 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0015] Figure 6 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0016] Figure 7 It is by Figure 1A or Figure 2A A flowchart of an example method executed by the system.
[0017] Figure 8 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0018] Figure 9 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0019] Figure 10 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0020] Figures 11A to 11C It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0021] Figure 12 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0022] Figure 13 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0023] Figure 14 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0024] Figure 15 It shows the result of Figure 1A or Figure 2A The example operation performed by the system.
[0025] Figure 16 It is by Figure 1A or Figure 2A A flowchart of an example method executed by the system.
[0026] Figure 17 It shows the result of Figure 1A or Figure 2A The example operation performed by the system. Detailed Implementation
[0027] Endoscopy is a medical procedure that visualizes internal organs or natural cavities of the human body. An endoscope is a rigid or flexible tubular device that allows direct observation of the body. Endoscopic systems can be constructed as purely optical devices containing lenses, transparent rods, or fibers, or in combination with integrated or additional imaging devices. The endoscope tip can be inserted through a small incision or through the body's natural lumens. A light source on the endoscope provides sufficient illumination to the cavity being examined. The imaging device on the endoscope provides a field of view that can be visualized on a screen and recorded for subsequent diagnosis or documentation. Two imaging devices (e.g., a left and right camera) can be placed side-by-side on the endoscope to form a stereoscopic imaging setup, which provides a stereoscopic view and allows for depth perception. For example, the two cameras can capture images from slightly different perspectives, similar to how the human eye perceives depth. These dual images are then processed to create 3D images or videos, providing surgeons and medical professionals with improved spatial awareness during the procedure. For example, triangulation techniques can be performed using images from two camera devices to determine the distances (e.g., depth of the points) between different points in the view and the camera devices, as well as the distances between these points. This allows surgeons to better understand the relative distances between different structures or instruments in the view. This enhanced spatial awareness can facilitate the performance of fine and precise manipulations.
[0028] In existing systems, subsurface anatomical structures are projected onto a non-optimal location on the surface, which is along the perspective or line of sight of the imaging device (e.g., the lines of sight of the two cameras). Such a perspective may not indicate to the surgeon the optimal path to access the subsurface structures. Therefore, such projection onto a non-optimal location on the surface does not provide the surgeon with the most desired surgical entry point. However, enhanced spatial awareness can be achieved by projecting anatomical structures onto the surface that is physically closest to or most proximate to the subsurface structures. This new projection site or location provides the surgeon with a better path to access the subsurface structures of interest.
[0029] This disclosure describes a medical system for annotating subsurface structures onto the surface of an anatomical structure. The medical system creates a projection (e.g., contour, colored shape, etc.) of the subsurface structure onto the surface of the anatomical structure. The location on the surface of the anatomical structure for projection is determined as the location on the surface that is physically closest to or closest to the subsurface structure. Depth maps and 3D reconstructions of the anatomical structure and subsurface structure are used to determine the point closest to the subsurface structure. The projection representing the subsurface structure is displayed at the closest point on the surface of the anatomical structure, or the projection representing the subsurface structure is projected. The depth of the subsurface structure is indicated using characteristics of the projection (e.g., color, blur, transparency, etc.). In one example, the appearance of the projection (e.g., color, transparency, etc.) can be varied to indicate the distance of the medical device from the projection.
[0030] In some implementations, the system offers several technical advantages. For example, it can provide surgeons with better access points to subsurface structures. More accurate access points allow surgeons to precisely locate anatomical structures, lesions, or target areas within the patient's body. This precision is crucial for performing delicate and targeted surgical procedures, minimizing errors. Surgeons can also rely on this information to navigate through complex anatomy and avoid unintended damage to surrounding tissues. Surgeons can use this information to plan and execute procedures with a high level of confidence, ensuring that critical structures are properly identified and treated. More accurate spatial information also aids in guiding instrument placement and allows for efficient navigation through narrow and confined spaces, contributing to the efficient use of surgical resources, including time and equipment. Therefore, the system can improve patient health and safety and reduce recovery time and postoperative complications.
[0031] In some examples, one or more components of a medical system may be implemented as a computer-assisted surgical system. However, it should be understood that the medical system can be implemented as any type of medical system, such as a digital benchmark system, an anatomical structure detection system, and a clinical guidance system. Figure 1A and Figure 2A Example computer-assisted surgical systems 100 and 200 are shown that can implement some of the features described herein.
[0032] Surgical system 100 includes a manipulator assembly 102, a user control device 104, and an auxiliary device 106, all of which are communicatively coupled to each other. A medical team utilizes surgical system 100 to perform computer-assisted medical procedures or other similar operations on the body of patient 108 or on any other body, such as that which may serve a particular implementation. The medical team includes a first user 110-1 (e.g., a surgeon during the procedure), a second user 110-2 (e.g., a patient-side assistant), a third user 110-3 (e.g., another assistant, nurse, intern, etc.), and a fourth user 110-4 (e.g., an anesthesiologist during the procedure), all of whom are collectively referred to as users 110, and each of them may control surgical system 100, interact with surgical system 100, or otherwise become a user of surgical system 100. During a medical procedure, there may be more, fewer, or alternative users, as may be required to serve a particular implementation. For example, the team composition for different medical procedures or for non-medical procedures may differ and include users with different roles.
[0033] although Figure 1AThe illustration depicts a minimally invasive medical procedure in progress, such as a minimally invasive surgical procedure; however, it will be understood that the surgical system 100 can be similarly used to perform open medical procedures or other types of operations. For example, it can also perform operations such as exploratory imaging, simulated medical procedures for training purposes, and / or other operations.
[0034] Manipulator assembly 102 includes one or more instruments that can be coupled to one or more manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4). These instruments are used for computer-assisted surgical procedures performed on patient 108 (e.g., by being at least partially inserted into and manipulated within patient 108). Although manipulator assembly 102 is depicted and described herein as comprising four manipulator arms 112, manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1 shows manipulator arms 112 as robotic manipulator arms, one or more instruments may be partially or fully manually controlled, for example by being held and manually controlled by a person. These partially or fully manually controlled instruments are used in conjunction with or as alternative instruments to computer-assisted instruments coupled to manipulator arms 112.
[0035] During medical procedures, the user control device 104 facilitates remote operation control of the manipulator arm 112 and instruments attached to it by the user 110-1. To this end, the user control device 104 provides the user 110-1 with images of the operating area associated with the patient 108, such as images captured by an imaging device. The manipulator arm 112 or any instrument coupled to it mimics the dexterity of the user 110-1's hand, wrist, and fingers across multiple degrees of freedom. In this way, the user 110-1 can visually perform procedures (e.g., incision procedures, suture procedures, etc.) using one or more of the manipulator arms 112 or any instruments coupled to them.
[0036] The auxiliary device 106 includes one or more computing devices that perform auxiliary functions to support the process, such as providing airflow, electrocautery energy, illumination, or other energy for the imaging device, image processing, or coordinating components of the surgical system 100. The auxiliary device 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information, to support the process. In some cases, the display monitor 114 is a touchscreen display that provides user input functionality. Enhanced content provided by a region-based enhancement system may be similar to or different from the content associated with the display monitor 114 or one or more display devices (not shown) in the operating area.
[0037] The controller assembly 102, user control device 104, and auxiliary device 106 are communicatively coupled to each other in any suitable manner. The controller assembly 102, user control device 104, and auxiliary device 106 may be communicatively coupled via control line 116, which represents any wired or wireless communication link that can serve a particular implementation. For this purpose, the controller assembly 102, user control device 104, and auxiliary device 106 may each include one or more wired or wireless communication interfaces, such as one or more LAN interfaces, Wi-Fi network interfaces, cellular interfaces, etc.
[0038] Figure 1B An example manipulator component 102 is shown. (See example...) Figure 2A As seen, the manipulator assembly 102 includes a base 118, manipulator arms 112-1, 112-2, 112-3, and 112-4. Each manipulator arm 112-1, 112-2, 112-3, and 112-4 is pivotally coupled to the base 118. Although the base 118 may include casters to allow for easy movement, in some embodiments, the manipulator assembly 102 is fixedly mounted to the floor, ceiling, console, structural frame, etc.
[0039] In a typical procedure, two of the manipulator arms 112-1, 112-2, 112-3, or 112-4 hold the surgical instruments, and a third manipulator arm holds the stereoscopic endoscope. The remaining manipulator arms are available to allow the introduction of other instruments at the work site. Alternatively, the remaining manipulator arms can be used to introduce another endoscope or another image-capturing device, such as an ultrasound transducer, into the work site.
[0040] Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 is formed by a linkage manipulated and coupled together by an actuable joint. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device such that a pivot point is located at its access port into the patient's body. The device manipulator can then manipulate its held device such that the held device can pivot about the pivot point, be inserted into and retracted from the access port, and rotate about its axis. Each of manipulator arms 112-1, 112-2, 112-3, and 112-4 may include sensors (e.g., joint sensors, position sensors, accelerometers, etc.) for detecting or tracking movement of manipulator arms 112-1, 112-2, 112-3, and 112-4. For example, these sensors can detect how far or how fast the manipulator arms 112-1, 112-2, 112-3, or 112-4 move in a certain direction.
[0041] Figure 1C An example user control device 104 is shown. User control device 104 includes a stereoscopic vision display 120, allowing a user to stereoscopically observe the surgical site from images captured by a stereoscopic camera device of manipulator assembly 102. A left eyepiece 122 and a right eyepiece 124 are provided in the stereoscopic vision display 120, allowing the user to view the left and right display screens within the display 120 with their left and right eyes, respectively. While typically observing images of the surgical site on a suitable observer or display, the surgeon performs the surgical procedure on the patient by manipulating a main control input device, which in turn controls the movement of robotic instruments.
[0042] User control device 104 also includes a left input device 126 and a right input device 128, which the user grasps with his / her left and right hands, respectively, to manipulate a device (e.g., a surgical instrument) held by manipulator arms 112-1, 112-2, 112-3, and 112-3 of manipulator assembly 102, preferably in six or more degrees of freedom (“DOF”). A foot pedal 130 with toe control and heel control is provided on user control device 104, allowing the user to control the movement and / or actuation of devices associated with the foot pedal.
[0043] A processing unit 132 is provided in the user control device 104 for control and other purposes. The processing unit 132 performs various functions within the surgical system 100. One function performed by the processing unit 132 is to convert and transmit the mechanical movements of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1, 112-2, 112-3, and 112-4, enabling the surgeon to effectively manipulate devices such as surgical instruments. Another function of the processing unit 132 is to implement the methods, cross-coupled control logic, and controllers or processors described herein. The auxiliary device 106 may include the processing unit 132 performing the functions or actions described herein. The processing unit 132 includes a controller and memory that perform the functions described herein. The controller may include one or more processors.
[0044] The controller may include any electronic circuit system, including but not limited to one or a combination of a microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory and controlling the operation of user control device 104 and / or auxiliary device 106. The controller may be 8-bit, 16-bit, 32-bit, 64-bit, or any other suitable architecture. The controller may include: an arithmetic logic unit (ALU) for performing arithmetic and logical operations; registers for supplying operands to the ALU and storing the results of ALU operations; and a control unit for fetching instructions from memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. The controller may include other hardware that operates software to control and process information. The controller executes software stored in memory to perform any of the functions described herein. The controller controls the operation and management of user control device 104 or auxiliary device 106 by processing information (e.g., information received from user control device 104, manipulator assembly 102, auxiliary device 106, and / or memory). A controller is not limited to a single processing device and may include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the controller is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions within that set.
[0045] Figure 2A An example computer-assisted surgical system 200 implementing some of the features described herein is shown. The surgical system 200 can be used for, for example, surgery, diagnosis, treatment, biopsy, or non-medical procedures. Figure 2AAs shown, a surgical system 200 (which may be a robot-assisted surgical system) includes one or more manipulator assemblies 202 for operating one or more medical device systems 204 while performing various procedures on a patient P positioned on a table T in a medical environment. For example, the manipulator assembly 202 may drive the movement of a catheter or end effector, apply treatment to target tissue, and / or manipulate control components. The manipulator assembly 202 may be a teleoperated assembly, a non-teleoperated assembly, or a hybrid teleoperated and non-teleoperated assembly, having selectable degrees of freedom of motion that can be motorized and / or teleoperated, and selectable degrees of freedom of motion that can be de-motorized and / or non-teleoperated. An operator input system 206, which may be inside or outside the medical environment, typically includes one or more control devices for controlling the manipulator assembly 202. The manipulator assembly 202 supports the medical device system 204 and may optionally include multiple actuators or motors that drive inputs on the medical device system 204 in response to commands from the control system 212. The actuator may optionally include a drive system that, when coupled to the medical device system 204, can advance the medical device system 204 into a natural orifice or surgically created anatomical orifice. Other drive systems can enable the distal end of the medical device to move in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the x, y, and z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the x, y, and z Cartesian axes). The manipulator assembly 202 may support various other systems for irrigation, treatment, or other purposes. Such systems may include fluid systems (e.g., reservoirs, heating / cooling elements, pumps, and valves), generators, lasers, interrogators, and ablation components.
[0046] The surgical system 200 also includes a display system 210 for displaying images or representations of the surgical site, and a medical device system 204. The image or representation is generated by an imaging system 209, which may include an endoscopic imaging system. The display system 210 and operator input system 206 can be configured such that an operator can control the medical device system 204 and operator input system 206 with a sense of telepresence. A graphical user interface can be displayed on the display system 210 and / or on a separately planned workstation display system.
[0047] In some examples, imaging system 209 includes an endoscopic imaging system having components integrally or removably coupled to medical device system 204. However, in some examples, a separate imaging device, such as an endoscope, attached to a separate manipulator assembly may be used with medical device system 204 to image a surgical site. Imaging system 209 may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include controller 214 of control system 212.
[0048] The surgical system 200 also includes a sensor system 208. The sensor system 208 may include a position / positioning sensor system (e.g., an actuator encoder or electromagnetic (EM) sensor system) and / or a shape sensor system (e.g., a fiber optic shape sensor) for determining the position, orientation, rate, velocity, pose, and / or shape of the medical device system 204. These sensors may also detect the position, orientation, or pose of the patient P on the table T. For example, the sensors may detect whether the patient P is facing down or up. As another example, the sensors may detect the direction in which the patient P's head is pointing. The sensor system 208 may also include temperature, pressure, force, or contact sensors, etc.
[0049] The surgical system 200 may also include a control system 212, which includes at least one memory 216 and at least one controller 214 (which may include a processor) for controlling the medical device system 204, the operator input system 206, the sensor system 208, and the display system 210. The control system 212 includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to perform procedures using the surgical system 200, including for navigation, steering maneuvers, imaging, engagement feature deployment or retraction, application of treatment to target tissue (e.g., via energy application), etc.
[0050] The control system 212 may also include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device system 204 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on reference to acquired preoperative or intraoperative datasets of the anatomical pathway. The virtual visualization system processes images of the surgical site, for example, using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The control system 212 uses preoperative images to locate target tissue (using visual imaging techniques and / or by receiving user input) and create a preoperative plan that includes an optimal first positioning for performing the treatment. The preoperative plan may include, for example, planned dimensions for expanding the expandable device, treatment duration, treatment temperature, and / or multiple deployment positions.
[0051] Controller 214 is any electronic circuit system, including but not limited to: one or a combination of a microprocessor, microcontroller, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), and / or state machine, communicatively coupled to memory 216 and controlling the operation of control system 212. Controller 214 may be 8-bit, 16-bit, 32-bit, 64-bit, or have any other suitable architecture. Controller 214 may include: an arithmetic logic unit (ALU) for performing arithmetic and logical operations; a processor register for supplying operands to the ALU and storing the results of ALU operations; and a control unit for fetching instructions from memory and executing the instructions by directing the coordinated operation of the ALU, registers, and other components. Controller 214 may include other hardware that operates software to control and process information. Controller 214 executes software stored on memory 216 to perform any of the functions described herein. Controller 214 controls the operation and management of control system 212 by processing information (e.g., information received from manipulator component 202, operator input system 206, and memory 216). The controller 214 is not limited to a single processing device, and may include multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. If multiple processing devices jointly perform a set of functions or actions, the controller 214 is considered to perform that set of functions or actions, even if different processing devices perform different functions or actions in that set.
[0052] Memory 216 may permanently or temporarily store data, operating software, or other information of controller 214. Memory 216 may include any or a combination of volatile or non-volatile local or remote devices suitable for storing information. For example, memory 216 may include random access memory (RAM), read-only memory (ROM), magnetic storage devices, optical storage devices, or any other suitable information storage devices or combinations thereof. Software refers to a collection of any suitable instructions, logic, or code contained in a computer-readable storage medium. For example, software may be contained in memory 216, a disk, CD, or flash drive. In certain embodiments, software may include an application executable by controller 214 to perform one or more of the functions described herein. Memory 216 is not limited to a single memory and may include multiple memories contained in the same device or computer or distributed across multiple devices or computers. Memory 216 is considered to store a collection of data, operating software, or information if multiple memories jointly store a collection of data, operating software, or information, even if different memories store different portions of the data, operating software, or information in that collection.
[0053] Figure 2B An example medical device system 204 in a surgical system 200 is shown. In some embodiments, the medical device system 204 is used for image-guided medical procedures. For example, the medical device system 204 may be used for non-teleoperable exploratory procedures or procedures involving conventionally manually operated medical devices (e.g., endoscopes).
[0054] The medical device system 204 includes an elongated flexible device 220, such as a flexible catheter or endoscope (e.g., a gastroscope, bronchoscope), coupled to a drive unit 222. The elongated flexible device 220 includes a flexible body 224 having a proximal end 226 and a distal or tip portion 228. In some embodiments, the flexible body 224 has an outer diameter of approximately 14 mm to 20 mm. Other flexible bodies may have larger or smaller outer diameters. The flexible body 224 is of appropriate length to reach portions of anatomical structures, such as the lungs, sinuses, throat, or upper or lower gastrointestinal region, when inserted into a patient's mouth or nasal cavity.
[0055] The medical device system 204 includes a tracking system 230 for determining the position, orientation, rate, velocity, pose, and / or shape of the distal end 228 and / or one or more segments 232 along the flexible body 224 using one or more sensors and / or imaging devices. The entire length of the flexible body 224 between the distal end 228 and the proximal end 226 is effectively divided into segments 232. The tracking system 230 is implemented as hardware, firmware, software, or a combination thereof, and interacts with or is otherwise performed by one or more computer controllers, which may include controller 214 of the control system 212.
[0056] Tracking system 230 uses shape sensor 234 to track one or more of the distal end 228 and / or segment 232. In some embodiments, tracking system 230 uses position sensor system 236, such as an electromagnetic (EM) sensor system, to track the distal end 228. In some examples, position sensor system 236 measures six degrees of freedom (e.g., three position coordinates x, y, and z, and three orientation angles indicating pitch, yaw, and roll of the reference point) or five degrees of freedom (e.g., three position coordinates x, y, and z, and two orientation angles indicating pitch and yaw of the reference point).
[0057] The flexible body 224 includes one or more channels 238, which are sized and shaped to receive one or more medical devices 240. In some embodiments, the flexible body 224 includes two channels 238 for a single device 240; however, a different number of channels 238 may be provided. Figure 2C It shows Figure 2B Example section of medical device system 204. (e.g.) Figure 2CAs shown, medical device 240 extends through flexible body 224. In some embodiments, medical device 240 can be used for procedures and aspects of procedures, such as surgery, biopsy, ablation, mapping, imaging, illumination, irrigation, or aspiration. Medical device 240 unfolds through channel 238 of flexible body 224 and is used at a targeted location within an anatomical structure. Medical device 240 includes, for example, image capturing devices, biopsy instruments, ablation instruments, catheters, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. Medical tools include end effectors with a single working component, such as scalpels, blunt blades, lenses, optical fibers, electrodes, etc. Other end effectors include, for example, forceps, graspers, balloons, needles, scissors, applicators, etc. Other end effectors also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, imaging devices, etc. Medical device 240 is advanced from the opening of channel 238 to perform the procedure and then retracts into the channel when the procedure is complete. The medical device 240 is removed from the proximal end 226 of the flexible body 224 or along another optional device port (not shown) of the flexible body 224. The medical device 240 can be used in conjunction with an image capturing device (e.g., an endoscopic camera device) also located within the elongated flexible device 220. Alternatively, the medical device 240 itself can be an image capturing device.
[0058] The medical device 240 additionally accommodates cables, linkages, or other actuation controls (not shown) extending between the proximal and distal ends to controllably bend the distal end of the medical device 240. The flexible body 224 also accommodates cables, linkages, or other steering control (not shown) extending between the drive unit 222 and the distal end 228 to controllably bend the distal end 228, as illustrated by the dashed line 242 depicting the distal end 228. In some examples, at least four cables are used to provide independent “up-down” steering operations for controlling the pitch motion of the distal end 228 and “left-right” steering maneuvers for controlling the yaw motion of the distal end 228. In embodiments where the medical device system 204 is actuated by a robot-assisted component, the drive unit 222 may include a drive input removably coupled to and receiving power from a drive element, such as an actuator, of the teleoperation component. In some embodiments, the medical device system 204 includes a gripping feature, a manual actuator, or other components for manually controlling the movement of the medical device system 204. Information from the tracking system 230 can be sent to the navigation system 244, where it is combined with information from the visualization system 246 and / or a preoperatively acquired model to provide real-time location information to the physician or other operator.
[0059] Figures 3 to 17 This shows the medical system (e.g., Figure 1A Surgical system 100 or Figure 2AThe example operations performed by the computer system in the surgical system 200 are as follows. Generally, the computer system (which may be implemented in the user control device 104 and / or auxiliary device 106 of the surgical system 100 using processing device 132, and / or in the control system 212 of the surgical system 200 using controller 214 and memory 216) determines the optimal path for the surgeon to access one or more subsurface structures of interest.
[0060] Figure 3 An example operation 300 performed by a computer system is illustrated. The computer system receives a first video captured by a camera device of an imaging apparatus 310 (e.g., an endoscope). The camera device is positioned at the front of the imaging apparatus 310. The video can be captured from a viewpoint at the front of the imaging apparatus 310. First imaging data 316 from the video can show the surface 318 of an anatomical structure. The anatomical structure may also be referred to as a surface structure. The computer system also receives second imaging data 326 from a second imaging apparatus 320 (e.g., an ultrasound probe). The second imaging data 326 can show the subsurface structure 328 beneath the surface 318 of the anatomical structure.
[0061] The first imaging device 310 provides a first imaging mode. For example, the first imaging device 310 may be an endoscope with a camera. The camera provides visible light imaging data, such as video. The second imaging device 320 provides a second imaging mode different from the first imaging mode. For example, the second imaging device may be an ultrasound probe that captures ultrasound images.
[0062] First imaging data 316 and second imaging data 326 are used to determine the point 330 on the surface 318 of the anatomical structure that is closest to or closest to the subsurface structure 328. Once the point 330 is determined, a projection 340 of the subsurface structure 328 is generated at the point 330 on the surface 318. For example, the computer system can determine the point 330 on the surface 318. The computer system can use a three-dimensional (3D) model of the subsurface structure 328 to determine the location or position of the subsurface structure 328 below the surface 318. The computer system can then determine the point 330 on the surface 318 that is physically closest to the subsurface structure 328. The computer system then generates a projection 340 of the subsurface structure 328 at the point 330. When viewed along a line intersecting the subsurface structure 328 and the point 330, the boundary or size of the projection 340 can indicate the boundary of the subsurface structure 328.
[0063] Figure 4 Example operation 400 performed by a computer system is shown. Generally, Figure 4The diagram illustrates a computer system generating and / or using depth maps to determine the location of surfaces of an anatomical structure. The computer system receives a first video captured by a camera device of imaging apparatus 310. First imaging data 316 from the video can show the surface 318 of the anatomical structure. The first imaging data 316 may include visible light imaging data 402, such as video.
[0064] The computer system uses depth map 410 to determine depth measurements of points on the anatomical structure. The computer system can generate depth map 410 by capturing and processing information representing one or more distances to points within the imaging or surgical field. For example, the first imaging device 310 may include multiple cameras capturing video of the anatomical structure from an offset viewpoint. Using these videos, the computer system can determine the coordinates of points on the anatomical structure and perform depth triangulation on the depths of these points. The computer system then generates depth map 410, which includes the coordinates of the points and their corresponding depths.
[0065] For a point on surface 318, the computer system determines the coordinates 412 of that point. Coordinates 412 indicate the location 414 of the point on surface 318 of the anatomical structure. The computer system can also use a depth map 410 to determine the depth corresponding to coordinates 412. The computer system can then determine this depth as the depth 416 of the point. The computer system can use this process to determine the location 414 and depth 416 of any number of points on surface 318.
[0066] Figure 5 Example operation 500 performed by a computer system is shown. Generally, Figure 5 The diagram illustrates how a computer system generates a 3D model 510 and uses this 3D model 510 to determine the location of the subsurface structure 328.
[0067] The computer system receives second imaging data 326, which may include ultrasound imaging data 404 (e.g., an ultrasound image) and may show the subsurface structure 328. The computer system can generate a 3D model 510 of the subsurface structure 328 based on the ultrasound imaging data 404.
[0068] Ultrasound imaging data 404 can show the size, location, and / or orientation of the subsurface structure 328 beneath the surface 318 of the anatomical structure. A computer system can generate a 3D model 510 of the subsurface structure 328 with the same size, location, and / or orientation shown in the ultrasound data 404. Therefore, the 3D model 510 has the size of the subsurface structure 328 and is located and oriented as the subsurface structure 328. Thus, the 3D model 510 shows the location, size, and orientation of the subsurface structure 328 beneath the surface 318 of the anatomical structure.
[0069] A computer system can create a 3D model 510 by merging two-dimensional (2D) ultrasound images of the subsurface structure 328 or by stitching together two-dimensional (2D) ultrasound images of the subsurface structure 328. Each of the ultrasound images can be a 2D image showing a slice of the subsurface structure 328. The computer system can apply volumetric reconstruction to the ultrasound images to arrange the slices into a 3D volume. The computer system can then convert the 3D volume into a mesh representation, where each voxel in the volume becomes a vertex in the mesh. The mesh representation forms the 3D model 510.
[0070] Figure 6 An example operation 600 performed by a computer system is shown. Generally, the computer system performs operation 600 to generate a projection of the subsurface structure at the point closest to the subsurface structure on the surface of the anatomical structure.
[0071] Since the depth map 410 indicates the depth of points on the surface 318 of the anatomical structure and the 3D model 510 indicates the location, size, and orientation of the subsurface structure 328, the computer system can use the depth map 410 and the 3D model 510 to determine the point 330 on the anatomical structure that is closest to the 3D model 510. For example, combining the depth map 410 with the 3D model 510 can show the relative positioning of the surface 318 of the anatomical structure and the subsurface structure 328. Using the depth of points in the depth map 410 and the 3D model 510, the computer system can calculate the distance between points on the surface 318 of the anatomical structure and the surface of the 3D model 510. The computer system can then select the point 330 on the surface 318 of the anatomical structure with the smallest such distance as the point 330 closest to the subsurface structure 328 on the anatomical structure.
[0072] A computer system can generate a projection 340 of the anatomical structure at point 330. When viewed along a line intersecting the subsurface structure 328 and point 330, the boundary or size of the projection 340 can indicate the boundary of the subsurface structure 328. The depth of the subsurface structure 328 can be indicated using properties of the projection 340, such as color, blurriness, transparency, etc. In one example, the appearance of the projection 340 (e.g., color, transparency, etc.) can be varied to indicate the distance of the medical device from the projection 340. Furthermore, the color, transparency, blurriness, or size of the projection 340 at least partially indicates the depth of the subsurface structure 328 from the surface 318 of the surface structure. The color, transparency, or blurriness can vary across the projection 340 to indicate that the subsurface structure 328 occupies multiple depths from the surface 318 of the surface structure.
[0073] Figure 7 It is by Figure 1A System 100 or Figure 2AThe flowchart illustrates an example method 700 executed by system 200. In a particular embodiment, the computer system executes method 700. By executing method 700, the computer system implements certain features that assist a user in annotating subsurface structures onto the surface of a surface structure (e.g., an anatomical structure). Thus, the subsurface structure is projected onto the point on the surface of the surface structure closest to the subsurface structure, which can indicate to the user where the closest entry path is for accessing the subsurface structure (e.g., to successfully treat a patient's health or medical condition).
[0074] In block 710, the computer system receives first imaging data captured by a first imaging device using a first imaging mode. For example, the first imaging device may be an endoscope with a camera positioned near a surface structure. The first imaging data may include visible light imaging data (e.g., video) captured by the camera device and may show the surface of the surface structure. In block 720, the computer system uses the first imaging data to generate a depth map. The depth map may include the coordinates of points on the surface of the surface structure. Each coordinate may indicate the location and depth of the point on the surface.
[0075] In block 730, the computer system receives second imaging data captured by a second imaging device using a second imaging modality. For example, the second imaging device may be an ultrasound probe, and the second imaging data may include ultrasound imaging data (e.g., 2D ultrasound images) showing a first surface structure beneath the surface of the anatomical structure. In block 740, the computer system uses the second imaging data to generate a 3D model of the first surface structure. The ultrasound imaging data and / or the 3D model may indicate the location of the subsurface structure. The 3D model may also indicate the size and orientation of the subsurface structure.
[0076] In box 750, the computer system uses a depth map and a 3D model to determine the first point on the surface of the surface structure that is closest to the primary surface structure. Since the depth map indicates the depth of points on the surface of the anatomical structure and the 3D model indicates the location, size, and orientation of the subsurface structure, the computer system can use the depth map and the 3D model to determine the point on the anatomical structure that is closest to the 3D model. For example, combining the depth map with the 3D model can show the relative positioning of the surface of the anatomical structure and the subsurface structure. Using the depth of points in the depth map and the 3D model, the computer system can calculate the distance between points on the surface of the anatomical structure and the surface of the 3D model. The computer system can then select the point on the surface of the anatomical structure with the smallest such distance as the point on the anatomical structure that is closest to the subsurface structure.
[0077] The distance or depth between the first point on the surface and the first surface structure can be referred to as the closest path, shortest path, or entry path. The closest path may not be aligned with the observation axis of the first imaging device. The computer system can adjust the entry path between the first surface structure and the first point on the surface structure to avoid one or more protected structures located between the first surface structure and the first point on the surface.
[0078] In box 760, the computer system generates and displays a first projection of the first surface structure onto the surface at a first point. The first projection of the first surface structure forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure. The boundary of the first projection is aligned with the boundary of the first surface structure or the centerline of the subsurface structure. The color, transparency, blurriness, or size of the first projection at least partially indicates the depth of the first surface structure from the surface of the surface structure. The color, transparency, or blurriness may vary across the first projection to indicate that the first surface structure occupies multiple depths from the surface of the surface structure.
[0079] Figure 8 An example operation 800 performed by a computer system is illustrated. The computer system annotates a subsurface structure 328 onto a point 825 on the surface 318 of an anatomical structure within the imaging device 310 (e.g., an endoscope capturing video of the surface 318) or in the user's line of sight. For example, the computer system may define point 825 as a point on the surface 318 on a line (e.g., an observation axis) between the imaging device 310 and the subsurface structure 328. The computer system may use a 3D model 510 of the subsurface structure 328 to determine the location or position of the subsurface structure 328 beneath the surface 318. The computer system may then determine point 825 on the surface 318 on the line between the imaging device 310 and the subsurface structure 328. The computer system then generates a projection 826 of the subsurface structure 328 at point 825. When viewed from the imaging device 310, the boundaries or dimensions of the projection 826 may indicate the boundaries of the subsurface structure 328. If the imaging device 310 moves, the computer system can move the point 825 and the projection 826 to keep the point 825 and the projection on the line between the imaging device 310 and the subsurface structure 328. The computer system can also adjust the size of the projection 826 for changes in the viewing angle of the subsurface structure 328.
[0080] However, rendering or annotating the subsurface structure 328 in this manner may result in suboptimal positioning of the projection onto the surface 318 of the anatomical structure. For example, such a perspective view of the subsurface structure 328 may not indicate the shortest path from the surface 318 to the subsurface structure 328.
[0081] Figure 9An example operation 900 performed by a computer system is illustrated. The computer system annotates a subsurface structure 328 onto a point 330 on a surface 318 of the anatomical structure, which may be the closest point 330 on surface 318 to the subsurface structure 328. As previously described, the computer system may have used a depth map 410 and a 3D model 510 of the subsurface structure 328 to determine the point 330 on surface 318 with the shortest distance to the subsurface structure 328. The computer system then generates a projection 911 of the subsurface structure 328 at point 330 (which may be aligned with or not aligned with the observation axis of the imaging device 310). When viewed along a line intersecting the subsurface structure 328 and point 330, the boundary or size of the projection 911 may indicate the boundary of the subsurface structure 328.
[0082] Boundary lines 916 and 918 are shown to visualize how the boundary of projection 911 aligns with the boundary of subsurface structure 328. Boundary lines 916 and 918 can extend from the opposite edges or surfaces of the boundary of projection 911. Since point 330 is closest to subsurface structure 328, point 330 indicates the shortest surgical path into subsurface structure 328. The determination of point 330, the generation of the projection, and the display of the projection are independent of the movement of the imaging device. For example, the positioning of point 330 and projection 911 on point 330 can remain unchanged regardless of where the imaging device 310 is located or where it may be moved.
[0083] The computer system can depict the boundary of projection 911 as solid or dashed lines with different colors and intensities. The computer system can also depict projection 911 as a single point (zero-dimensional annotation) or a single line (one-dimensional annotation). Figure 9 In this context, projection 911 is depicted as a two-dimensional annotation with solid lines. The computer system can adjust or change the color, transparency, blur, shadow, line type, size, etc. of projection 911 to indicate any characteristics or information about the subsurface structure 328 (e.g., the depth of the subsurface structure or the distance between the surgical instrument and the subsurface structure).
[0084] Figure 10 An example operation 1000 performed by a computer system is illustrated. The computer system creates a heat map for the subsurface structure 328. In a particular embodiment, the computer system may display the heat map to provide an indication of where the subsurface structure 328 is located relative to the surface 318 of the anatomical structure. The heat map may be included on the surface 318 of the anatomical structure as part of a projection 911, or the heat map may be projected onto the surface 318 together with the projection 911.
[0085] A heatmap is a graphical representation that uses a color-coding system to represent different values. Color coding can include other visual attributes, such as, but not limited to, transparency and blur. Figure 10 The thermal image includes a first ring 1010, a second ring 1012, and a third ring 1014. The first ring 1010 is formed directly around the subsurface structure 328 and can be shown in a first color with a first intensity. The second ring 1012 is formed directly around the first ring 1010 and can be shown in a first color with a second intensity. The third ring 1014 is formed directly around the second ring 1012 and can be shown in a first color with a third intensity. The first intensity can be greater than the second intensity, and the second intensity can be greater than the third intensity. Therefore, the color intensity can decrease as the distance from the subsurface structure 328 increases. The color intensity provides the user with an indication of the distance of the subsurface structure 328 from the surface 318 of the anatomical structure. Therefore, when a user attempts to access the subsurface structure 328 with a surgical instrument, based on the color intensity, the user can infer how far or near the tip of the surgical instrument is from the subsurface structure 328.
[0086] Figure 11A , Figure 11B and Figure 11C Example operations 1100A, 1100B, and 1100C performed by a computer system are illustrated. The computer system annotates a subsurface structure 328 onto the surface 318 of the anatomical structure. In a particular embodiment, the computer system may project the subsurface structure 328 onto the surface 318 that is physically closest to the subsurface structure 328, such that the subsurface structure 328 forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure.
[0087] exist Figure 11A In this process, the computer system projects the subsurface structure 328 as a single point 1110 (zero-dimensional annotation) on the surface 318. Point 1110 can be located on the point 330 on the surface 318 that is closest to the subsurface structure 328.
[0088] exist Figure 11BIn this diagram, the computer system projects the subsurface structure 328 as a line 1120 (one-dimensional annotation) on the surface 318. Line 1120 can be aligned with line 822 of the subsurface structure 328. Line 822 can extend across the center point of the subsurface structure 328. Therefore, line 1120 can represent or indicate the length or width of the subsurface structure 328. The computer system can depict line 1120 as a solid or dashed line of varying thickness to indicate the proximity of the subsurface structure 328 to the anatomically shaped surface 318. A dashed line can indicate a first distance between the subsurface structure 328 and the surface 318. A thin solid line can indicate a second distance between the subsurface structure 328 and the surface 318 greater than the first distance. A thick solid line can indicate a third distance between the subsurface structure 328 and the surface 318 greater than both the first and second distances.
[0089] exist Figure 11C In this context, the computer system projects the subsurface structure 328 using boundary lines 1130 (two-dimensional annotations) on surface 318. Boundary lines 1130 can be aligned with boundary lines 1135 of the subsurface structure 328. These boundary lines can also be referred to as perimeter lines. Therefore, boundary lines 1130 can represent the perimeter of the subsurface structure 328. The computer system can depict boundary lines 1130 as solid or dashed lines of varying thicknesses to indicate the proximity of the subsurface structure 328 to the anatomically shaped surface 318. Dashed lines can indicate a first distance between the subsurface structure 328 and surface 318. Thin solid lines can indicate a second distance between the subsurface structure 328 and surface 318 greater than the first distance. Thick solid lines can indicate a third distance between the subsurface structure 328 and surface 318 greater than both the first and second distances.
[0090] In one example, the user can pre-select how the annotations of the subsurface structure 328 are projected. The user can choose whether to present the subsurface structure 328 as a zero-dimensional, one-dimensional, or two-dimensional annotation on the anatomical surface 318.
[0091] Figure 12An example operation 1200 performed by a computer system is illustrated. The computer system determines a point 1205 in a projection onto a surface 318 of the anatomical structure. The computer system also determines the depth 1201 of a point on a subsurface structure corresponding to point 1205. The computer system can then set different colors 1210, transparency 1220, or blur 1230 for point 1205 based on the depth 1201. Therefore, the color 1210, transparency 1220, or blur 1230 of a point in the projection 1205 can indicate the depth 1201 of the point on the subsurface structure corresponding to that point 1205 (or the distance between point 1205 and its corresponding point on the subsurface structure). The computer system can use any number of colors, transparency levels, or blur levels to indicate any number of depths 1201. The user can choose whether to use different colors, different transparency levels, or different blur levels to indicate the depth 1201.
[0092] In some cases, the projection size 1240 can also indicate the depth of the subsurface structure 328. For example, since the boundary of the projection can be aligned with the boundary of the subsurface structure 328, the projection boundary can be smaller if the subsurface structure 328 is far below the surface 318 of the anatomical structure. If the same subsurface structure 328 is close to the surface 318 of the anatomical structure, the projection boundary will be larger. Therefore, the projection size 1240 can also provide an indicator of the depth 1201 of a point on the subsurface structure 328.
[0093] Figure 13 An example operation 1300 performed by a computer system is illustrated. The computer system determines the distance 1310 between a surgical instrument and a subsurface structure 328. For example, the computer system may determine the coordinates of the surgical instrument and then determine the distance between these coordinates and a 3D model of the subsurface structure 328. The computer system then adjusts or sets the color, transparency level, blur level, or size of the projection to indicate the distance 1310. For example, as the surgical instrument is brought closer to or further away from the subsurface structure, the computer system may change the color 1210, transparency 1220, blur 1230, or size 1240 of the projection to indicate the changing distance between the surgical instrument and the subsurface structure. The user can choose whether to use color, transparency level, blur level, or projection size as an indicator of the distance between the surgical instrument and the subsurface structure.
[0094] Figure 14An example operation 1400 performed by a computer system is illustrated. The computer system can determine that the path between a point 330 on the surface 318 of the anatomical structure and the subsurface structure 328 is blocked by an object 1405. In response, the computer system can determine a point 1420 on the surface of the anatomical structure that provides the shortest, unobstructed path 1410 to the subsurface structure. The computer system can then generate a projection 1415 at point 1420 to indicate how to enter the subsurface structure 328 while avoiding the object 1405.
[0095] Object 1405 may be a protected structure (e.g., a tumor, vein, artery, or another anatomical object) that should be avoided when surgery enters the subsurface structure 328. The computer system can determine the location, orientation, and / or orientation of object 1405 based on ultrasound imaging data. For example, object 1405 may appear in an ultrasound image as a second imaging device (e.g., an ultrasound probe) scans across the surface 318 of the anatomical structure. 3D model 510 may include a model of object 1405 indicating its location, orientation, and / or orientation.
[0096] The computer system can determine, based on the model of object 1405, that object 1405 is positioned on the path from point 330 to subsurface structure 328. Therefore, object 1405 blocks this path, and entering subsurface structure 328 from point 330 may damage object 1405. In response, the computer system uses depth map 410 and 3D model 510 to determine other points on surface 318 that are not blocked by object 1405 for paths between these points and the subsurface structure. The computer system determines the distances between these points and subsurface structure 328 and selects the point with the shortest distance as point 1420. The computer system then generates a projection 1415 of subsurface structure 328 on surface 318 at point 1420. Figure 14 As shown, the path 1410 from point 1420 to the subsurface structure is not blocked by object 1405.
[0097] In some implementations, when the computer system determines that object 1405 is blocking the path between point 330 and subsurface structure 328, the computer system generates a warning or alarm. The user can then instruct the computer system to determine an alternative point or access path. In response to this instruction, the computer system determines point 1420 and generates projection 1415.
[0098] In one embodiment, the computer system determines point 1420 in response to determining that point 330 is hidden outside the field of view or inaccessible by surgical instruments. Since point 330 is hidden outside the field of view or inaccessible by surgical instruments, attempting to enter the subsurface structure 328 from point 330 may be impractical for the user. In response, the computer system determines point 1420 on the surface 318 of the anatomical structure that is closest to the subsurface structure, visible, and accessible by surgical instruments. The computer system then generates a projection 1415 at point 1420 to instruct the user on how to enter the subsurface structure 328.
[0099] Figure 15 An example operation 1500 performed by a computer system is illustrated. The computer system allows a user to place a marker on surface 318. In a particular embodiment, the computer system may provide the user with the ability to place virtual markers in any location relative to or adjacent to the projection.
[0100] After the computer system detects the subsurface structure 328, it provides a projection 1525 on the surface 318. A center point 1505 of the projection is also provided. The user can apply different types of marks, markers, or indicators on or near the projection 1525. For example, the user can apply marker 1510 along the boundary line of the projection 1525. In another example, the user can apply marker 1520 near the center point 1505. The user can apply any type of mark with any color, transparency, blur, or size at any location near the projection 1525. Markers can represent potential entry points or entry paths for guiding surgical instruments to the subsurface structure 328. The user can also save such marks for future reference. The marks provide the user with different entry options based on the surgical procedure. In one example surgical procedure, the user may need to enter the entire subsurface structure 328 for removal purposes. Therefore, marker 1510 provides the boundary or outline of the subsurface structure 328, which can guide the user when cutting or removing the entire subsurface structure 328. In another example surgical procedure, the user may only need to access a center point of the subsurface structure 328 to, for example, place a staple or suture. Therefore, marker 1520 provides a center point relative to the subsurface structure where the computer system can allow the user to apply staples or sutures.
[0101] Figure 16 This is a flowchart of example method 1600 executed by a computer system. By executing method 1600, the computer system annotates subsurface structures onto the surface of a surface structure. These annotations can indicate visible and reachable points that provide unobstructed paths to the subsurface structures.
[0102] In block 1610, the computer system receives first imaging data captured by a first imaging device using a first imaging mode. For example, the first imaging data may include video captured by a camera device of an endoscope. The first imaging data may show the surface of a surface structure. The computer system uses the first imaging data to generate a depth map having coordinates indicating the position and depth of points on the surface of the surface structure.
[0103] In box 1620, the computer system receives second imaging data captured by a second imaging device using a second imaging modality. For example, the second imaging data may include ultrasound images captured by an ultrasound probe scanning the surface of a surface structure. The second imaging data may show a subsurface structure beneath the surface of the surface structure. The computer system uses the second imaging data to generate a 3D model of the subsurface structure. The 3D model may indicate the location, position, and / or orientation of the first surface structure.
[0104] In box 1630, the computer system determines a first point on the surface of the surface structure that is closest to the subsurface structure. The computer system can use a depth map and a 3D model to determine the first point. For example, the depth map can indicate the location and depth of a point on the surface of the surface structure, and the 3D model can indicate the location and position of the subsurface structure. Using this information, the computer system can determine the distance between a point on the surface of the surface structure and the subsurface structure. The computer system can select the point on the surface of the surface structure that has the shortest distance to the subsurface structure as the first point.
[0105] In box 1640, the computer system generates a first projection of the subsurface structure onto the surface at a first point. The first projection can be a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure. The computer system can set the color, transparency level, or blur level of the first projection to indicate the depth of the subsurface structure from the surface of the surface structure. The color, transparency level, or blur level can vary across the first projection to indicate that the subsurface structure occupies multiple depths from the surface of the surface structure.
[0106] In box 1650, the computer system determines that the path from the first point to the subsurface structure is blocked. For example, the computer system may determine that an object (e.g., a tumor, vein, artery, or anatomical feature) is blocking the path. To avoid damaging the object, the user can prompt the computer system to calculate a second shortest path or distance to avoid the object.
[0107] In box 1660, the computer system uses a depth map and a 3D model to determine a second point on the surface of the subsurface structure. The second point can have the shortest, unobstructed path to the subsurface structure.
[0108] In box 1670, the computer system generates and displays a second projection of the subsurface structure onto the surface of the surface structure at the second point. The second projection indicates an unobstructed path to the subsurface structure.
[0109] Figure 17 An example operation 1700 performed by a computer system is illustrated. The computer system identifies multiple subsurface structures of interest. In a particular implementation, the computer system may provide various indicators for the subsurface structures.
[0110] The second imaging device 320 captures second imaging data 326 (e.g., ultrasound image). The second imaging data 326 may show the first surface structure 328. The computer system determines a first point 330 on the surface of the anatomical structure closest to the first surface structure 328. The computer system generates and displays a first projection 340 of the first surface structure 328 onto the surface at the first point 330. The first projection 340 may be marked or identified by a first indicator 1726. The first indicator 1726 may be color, transparency level, blur level, or projection size. The user can choose whether to use color, transparency level, blur level, or projection size as an indicator of the distance between the second imaging device 320 and the first surface structure 328.
[0111] The second imaging data 326 can also show a second surface structure 1730 near the first surface structure 328. The computer system determines a second point 1732 on the surface of the anatomical structure that is closest to the second surface structure 1730. The computer system generates a second projection 1734 of the second surface structure 1730 onto the surface at the second point 1732. The second projection 1734 can be marked or identified by a second indicator 1736. The second indicator 1736 can be color, transparency level, blur level, or projection size. The user can choose whether to use color, transparency level, blur level, or projection size as an indicator of the distance between the second imaging device 320 and the second surface structure 1730.
[0112] The computer system can display the first projection 340 and the second projection 1734 simultaneously or concurrently on the display 1740. Therefore, the display 1740 can provide an indication of the first surface structure 328 relative to the second surface structure 1730 on the surface of the surface structure. Additionally, the first surface structure 328 can be displayed simultaneously with the first projection 340. Similarly, the second surface structure 1730 can be displayed simultaneously with the second projection 1734. Furthermore, in one example, the first projection 340 can be represented as a first color on the surface of the surface structure, and the second projection 1734 can be represented as a second color.
[0113] In another example, the user can switch between the first projection 340 and the second projection 1736. The user can observe the first projection 340 independently of the second projection 1736 after the first projection 340 and the second projection 1736 have been generated.
[0114] In summary, the medical system annotates subsurface structures onto the surface of anatomical structures. The medical system creates projections of the subsurface structures (e.g., contours, colored shapes, etc.) onto the surface of the anatomical structure. The location on the surface of the anatomical structure used for projection is determined as the location on the surface that is physically closest to or closest to the subsurface structure. Depth maps and 3D models of the anatomical and subsurface structures are used to determine the point closest to the subsurface structure. The projection representing the subsurface structure is displayed at the closest point on the surface of the anatomical structure. The depth of the subsurface structure is indicated using properties of the projection (e.g., color, blurriness, transparency, etc.). In one example, the appearance of the projection (e.g., color, transparency, etc.) can be varied to indicate the distance of the medical device from the projection. The system can provide surgeons with better access points to subsurface structures. More accurate access points allow surgeons to precisely locate anatomical structures, lesions, or target areas within the patient's body. This precision is important for performing delicate and targeted surgical procedures, minimizing errors. Surgeons can also rely on this information to navigate through complex anatomical structures and avoid unintended damage to surrounding tissues. Surgeons can use this information to plan and execute procedures with a high level of confidence, ensuring that critical structures are properly identified and treated. More accurate spatial information also helps guide instrument placement and enables effective navigation through narrow and confined spaces, contributing to the efficient use of surgical resources, including time and equipment. Therefore, the system can improve patient health and safety, and reduce recovery time and postoperative complications.
[0115] The accompanying drawings, illustrating various aspects, embodiments, or modules, and this specification should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure other features. Identical numbers in two or more figures denote identical or similar elements.
[0116] In this specification, specific details are set forth in relation to some embodiments consistent with this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art can implement other elements that, while not specifically described herein, are within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments, unless otherwise specifically described, or if one or more features would render an embodiment inoperable.
[0117] Furthermore, the terminology used in this specification is not intended to be limiting. For example, spatially relative terms such as “below,” “under,” “lower,” “above,” “upper,” “near,” “far”, etc., may be used to describe the relationship of one element or feature to another element or feature as shown in the figures. In addition to the positions and orientations shown in the figures, these spatially relative terms are intended to cover different positions (i.e., positioning) and orientations (i.e., rotational placement) of elements or their operations. For example, if one of the contents in the figures is flipped, an element described as being “below” or “under” other elements or features will then be “above” or “on” other elements or features. Thus, the exemplary term “below” can include both above and below positions and orientations. Devices may be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein are interpreted accordingly. Similarly, descriptions of movement along and around various axes include various specific element positions and orientations. Additionally, unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are also intended to include the plural forms. Furthermore, the terms "comprising," "having," "including," etc., specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly electrically or mechanically coupled, or they may be indirectly coupled via one or more intermediate components.
[0118] Where feasible, elements described in detail with reference to one embodiment or module may be included in other embodiments or modules in which such elements are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may still be required to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment or application may be incorporated into other embodiments or aspects unless specifically described otherwise, unless one or more elements would render one or more embodiments inoperable, or unless two or more of the elements provide conflicting functionality.
[0119] In some cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the implementation.
[0120] This disclosure describes the various devices, elements, and portions of computer-aided devices and elements in terms of their state in three-dimensional space. As used herein, the term "position" refers to the location of an element or portion of an element in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an element or portion of an element (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "shape" refers to a set of positions or orientations measured along the element. As used herein, and for devices with repositionable arms, the term "proximal end" refers to a direction along its kinematic chain toward the base of the computer-aided device, and "distal end" refers to a direction along the kinematic chain away from the base.
[0121] Various aspects of this disclosure are described with reference to computer-aided systems and apparatuses, which may include teleoperated, remotely controlled, autonomous, semi-autonomous, robotic, and other such systems and apparatuses. Furthermore, various aspects of this disclosure are described according to embodiments using medical systems, such as the da Vinci Surgical System or Ion System, commercially available from Intuitive Surgical Inc. of Sunnyvale, California. However, those skilled in the art will understand that the aspects disclosed herein can be implemented and carried out in various ways, including robotic and non-robotic implementations (where applicable). The techniques described with reference to surgical instruments and methods can be used in other contexts. Therefore, the instruments, systems, and methods described herein can be used for humans, animals, parts of human or animal anatomy, industrial systems, general-purpose robotic or teleoperation systems. As another example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general-purpose robotic applications, sensing or manipulating non-tissue artifacts, cosmetic enhancements, imaging of human or animal anatomy, acquiring data from human or animal anatomy, setting up or dismantling systems, training medical or non-medical personnel, etc. Additional example applications include procedures for removing tissue from human or animal anatomy (with or without returning it to the anatomy) and procedures for human or animal cadavers. Furthermore, these techniques can also be used in medical treatments or diagnostic procedures, with or without surgical involvement.
[0122] Although exemplary embodiments have been shown and described, extensive modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, certain features of the embodiments may be employed without corresponding use of other features. Many variations, alternatives, and modifications will be recognized by those skilled in the art. Therefore, the scope of this disclosure should be limited only by the appended claims, and it is appropriate that the claims be interpreted broadly and in a manner consistent with the scope of the embodiments disclosed herein.
Claims
1. A system for annotating subsurface structures onto the surface of a surface structure, the system comprising: Memory; as well as A controller communicatively coupled to the memory, the controller being configured to: Receive first imaging data captured by a first imaging device using a first imaging mode, the first imaging data illustrating the surface of the surface structure; Receive second imaging data captured by a second imaging device using a second imaging mode, the second imaging data showing the first surface structure; The first point on the surface is determined based at least on where on the surface it is closest to the first surface structure; At the first point, a first projection of the first surface structure onto the surface is generated; as well as The first projection at the first point on the surface of the surface structure is displayed on the display device.
2. The system according to claim 1, wherein, The first imaging data includes visible light imaging data, or the second imaging data includes ultrasound imaging data.
3. The system according to claim 1, wherein, The controller is also configured to generate a three-dimensional (3D) model of the first surface structure based on the second imaging data.
4. The system according to claim 3, wherein, The controller is also configured to generate a depth map that includes the coordinates of the surface of the surface structure, wherein each coordinate of the depth map indicates the position and depth of a portion of the surface of the surface structure.
5. The system according to claim 4, wherein, Determining the first point on the surface that is closest to the first surface structure includes merging the depth map with the 3D model.
6. The system according to claim 1, wherein, The first projection of the first surface structure forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure.
7. The system according to claim 1, wherein, The boundary of the first projection is aligned with the boundary of the first surface structure or the centerline of the second surface structure.
8. The system according to claim 1, wherein, The color, transparency, blur, heatmap, or size of the first projection at least partially indicates the depth of the first surface structure from the surface of the surface structure.
9. The system according to claim 8, wherein, The depth represents the closest distance between the first surface structure and the surface of the surface structure.
10. The system according to claim 8, wherein, The color, transparency, heatmap, or blur varies across the first projection to indicate that the first surface structure occupies multiple depths from the surface of the surface structure.
11. The system according to claim 1, wherein, The controller is also configured to determine the distance between the instrument and the first surface structure; and The color, transparency, blur, heatmap, or size of the first projection at least partially indicates the determined distance.
12. The system according to claim 1, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is not aligned with the observation axis of the first imaging device.
13. The system according to claim 12, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is further defined to avoid one or more protected structures located between the first surface structure and the first point on the surface.
14. The system according to claim 13, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is determined based on a mark made by the user.
15. The system according to claim 1, wherein, The controller is also configured to: A second point on the surface is determined based at least on where on the surface it is closest to the first surface structure, and the distances from the first point to the first surface structure and from the second point to the first surface structure are within a predetermined range; At the second point, a second projection of the first surface structure onto the surface is generated; as well as The second projection at the second point on the surface of the surface structure is displayed on the display device.
16. The system according to claim 15, wherein, The second projection occurs when the first point is determined to be hidden outside the field of vision or inaccessible by instruments.
17. The system according to claim 1, wherein, The determination of the first point, the generation of the first projection, and the display of the first projection are independent of the movement of the first imaging device.
18. The system according to claim 1, wherein, The first surface structure is displayed simultaneously with the first projection.
19. The system according to claim 1, wherein, The second imaging data also shows a second surface structure, and the controller is further configured to: The second point on the surface is determined based at least on where on the surface it is closest to the second surface structure; At the second point, a second projection of the second surface structure onto the surface is generated; as well as The second projection at the second point on the surface of the surface structure is displayed on the display device.
20. The system according to claim 19, wherein, The first projection and the second projection are simultaneously displayed on the surface of the surface structure.
21. The system according to claim 20, wherein, On the surface of the surface structure, the first projection is represented as a first color, and the second projection is represented as a second color.
22. A method for annotating subsurface structures onto the surface of a surface structure, the method comprising: Receive first imaging data captured by a first imaging device using a first imaging mode, the first imaging data illustrating the surface of the surface structure; Receive second imaging data captured by a second imaging device using a second imaging mode, the second imaging data showing the first surface structure; The first point on the surface is determined based at least on where on the surface it is closest to the first surface structure; At the first point, a first projection of the first surface structure onto the surface is generated; as well as The first projection at the first point on the surface of the surface structure is displayed on the display device.
23. The method according to claim 22, wherein, The first imaging data includes visible light imaging data, or the second imaging data includes ultrasound imaging data.
24. The method according to claim 22, wherein, The controller is also configured to generate a three-dimensional (3D) model of the first surface structure based on the second imaging data.
25. The method according to claim 24, wherein, The controller is also configured to generate a depth map that includes the coordinates of the surface of the surface structure, wherein each coordinate of the depth map indicates the position and depth of a portion of the surface of the surface structure.
26. The method according to claim 25, wherein, Determining the first point on the surface that is closest to the first surface structure includes merging the depth map with the 3D model.
27. The method according to claim 22, wherein, The first projection of the first surface structure forms a zero-dimensional, one-dimensional, or two-dimensional annotation on the surface of the surface structure.
28. The method according to claim 22, wherein, The boundary of the first projection is aligned with the boundary of the first surface structure or the centerline of the second surface structure.
29. The method according to claim 22, wherein, The color, transparency, blur, heatmap, or size of the first projection at least partially indicates the depth of the first surface structure from the surface of the surface structure.
30. The method according to claim 29, wherein, The depth represents the closest distance between the first surface structure and the surface of the surface structure.
31. The method according to claim 29, wherein, The color, transparency, heatmap, or blur varies across the first projection to indicate that the first surface structure occupies multiple depths from the surface of the surface structure.
32. The method according to claim 22, wherein, The controller is also configured to determine the distance between the instrument and the first surface structure; and The color, transparency, blur, heatmap, or size of the first projection at least partially indicates the determined distance.
33. The method according to claim 22, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is not aligned with the observation axis of the first imaging device.
34. The method according to claim 33, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is further defined to avoid one or more protected structures located between the first surface structure and the first point on the surface.
35. The method according to claim 34, wherein, The entry path between the first surface structure and the first point on the surface of the surface structure is determined based on a mark made by the user.
36. The method according to claim 22, wherein, The controller is also configured to: A second point on the surface is determined based at least on where on the surface it is closest to the first surface structure, and the distances from the first point to the first surface structure and from the second point to the first surface structure are within a predetermined range; At the second point, a second projection of the first surface structure onto the surface is generated; as well as The second projection at the second point on the surface of the surface structure is displayed on the display device.
37. The method of claim 36, wherein, The second projection occurs when the first point is determined to be hidden outside the field of vision or inaccessible by instruments.
38. The method according to claim 22, wherein, The determination of the first point, the generation of the first projection, and the display of the first projection are independent of the movement of the first imaging device.
39. The method according to claim 22, wherein, The first surface structure is displayed simultaneously with the first projection.
40. The method according to claim 22, wherein, The second imaging data also shows a second surface structure, and the method further includes: The second point on the surface is determined based at least on where on the surface it is closest to the second surface structure; At the second point, a second projection of the second surface structure onto the surface is generated; and The second projection at the second point on the surface of the surface structure is displayed on the display device.
41. The method according to claim 40, wherein, The first projection and the second projection are simultaneously displayed on the surface of the surface structure.
42. The method according to claim 41, wherein, On the surface of the surface structure, the first projection is represented as a first color, and the second projection is represented as a second color.
43. A non-transitory machine-readable medium storing instructions for annotating a subsurface structure onto a surface, the instructions causing the controller, when executed by the controller, to: Perform the method according to any one of claims 22 to 42.
