System and method for persistent marking

Abstract

A system may include a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions may cause the system to receive three-dimensional primary image data from an imaging system having a field of view and generate a surface representation of a tissue surface in the field of view. The instructions may also cause the system to identify a region in the field of view with a fiducial marker, associate the fiducial marker with the surface representation, and move the surface representation and the fiducial marker in response to movement of the tissue surface in the field of view.

Description

[0001] Cross-referenced applications This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 415,538, filed on October 12, 2022, and entitled “Systems and Methods for Persistent Markers,” which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure is directed to systems and methods for robotically assisted medical procedures, and more particularly, to systems and methods for generating fiducial markers and causing the fiducial markers to persist and / or move with anatomical images. Background Art

[0003] Minimally invasive medical techniques are intended to reduce the amount of extraneous tissue that is damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments such as endoscopic instruments that provide users with images of the field of view within the patient's anatomy.

[0004] Some minimally invasive medical tools may be robotically assisted, including teleoperated, remotely opertated, or otherwise computer-assisted. During a medical procedure, a clinician may view an image of a field of view of a patient's anatomy on a display, which may include one or more minimally invasive medical tools. Video telestrations visible on the display may be generated to mark, annotate, identify, or otherwise provide graphical or alphanumeric information associated with items visible in the field of view. Improved systems and methods are needed to present video telestrations based on knowledge of the field of view. Summary of the invention

[0005] Embodiments of the invention are best summarized by the claims following the description.

[0006] In one example, a system may include a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions may cause the system to receive three-dimensional primary image data from an imaging system having a field of view and generate a surface representation of a tissue surface in the field of view. The instructions may also cause the system to identify an area in the field of view with a fiducial marker, associate the fiducial marker with the surface representation, and move the surface representation and the fiducial marker in response to movement of the tissue surface in the field of view.

[0007] In another example, a system may include a processor and a memory having computer readable instructions stored thereon. When executed by the processor, the computer readable instructions may cause the system to receive three-dimensional primary image data from an imaging system having a field of view, receive enhanced image data of the field of view from a secondary imaging modality, identify an area on a tissue surface in the enhanced image data of the field of view, and generate a fiducial marker associated with the identified area.

[0008] It should be understood that both the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart illustrating a method for associating fiducial markers with surface representations of structures in a field of view, according to some examples.

[0010] Figure 2 A display system for displaying an image of a field of view is shown according to some examples.

[0011] Figure 3A A mesh model of a structure in a field of view is shown according to some examples.

[0012] Figure 3B shows some examples from Figure 3A Continuous surface model of the structure.

[0013] Figure 4A is a flow chart illustrating a method for identifying a region in a field of view with a fiducial marker, according to some examples.

[0014] Figure 4B is a flow chart illustrating a method for identifying a region in a field of view with a fiducial marker, according to some other examples.

[0015] Figure 5A An anatomy is shown in a field of view of a fluorescence imaging system according to some examples.

[0016] Figure 5B According to some examples Figure 5A Surface representation of the anatomical body in the field of view.

[0017] Fig. 6A According to some examples Figure 3A Deformation of the mesh model.

[0018] Figure 6B According to some examples Figure 3B The deformation of the surface representation.

[0019] Fig. 7A An anatomy is shown in a field of view of a fluorescence imaging system according to some examples.

[0020] Figure 7B According to some examples Fig. 7A Surface representation of the anatomical body in the field of view.

[0021] Figure 7C FIG. 1 shows a schematic diagram of a 3D image sensor having a reference marker according to some examples. Fig. 7A anatomy of the field of view.

[0022] Fig. 8A Surface representations of anatomy according to some examples are shown.

[0023] Figure 8B FIG. 1 shows a visible light imaging system according to some examples. Fig. 8A A surface representation of an anatomical body is shown.

[0024] Fig.9A is a flow chart illustrating a method of associating subsurface fiducial markers with an image or representation of a field of view, according to some examples.

[0025] Fig. 9B An anatomy of a field of view with subsurface fiducial markers is shown according to some examples.

[0026] Fig.10 Schematic diagrams of medical systems are shown according to some examples.

[0027] Fig.11 is a perspective view of a manipulator assembly of the medical system of FIG. 9 , according to some examples.

[0028] Fig.12 is a front view of an operator console in a robotic-assisted medical system according to some examples.

[0029] Embodiments of the present disclosure and their advantages are best understood by referring to the detailed description below. It should be understood that like reference numerals are used to identify like elements shown in one or more figures, where the illustrations therein are for the purpose of illustrating embodiments of the present disclosure, rather than for the purpose of limiting them. DETAILED DESCRIPTION

[0030] In a robotic-assisted medical procedure, an endoscopic image of the surgical environment can provide the clinician with an image of the anatomical region of the patient's anatomy and the field of view of any medical tool located in the anatomical region. During the procedure, various factors, such as the skill level of the clinician, environmental interference, displaced tissue and abnormal anatomy, rapid movement of the camera to different areas of the scene, and the dissipation of specific injection signals over time, may detract the clinician's ability to maintain spatial positioning of structures in the anatomical region. Identifying and tracking key structures (such as vasculature, organ surfaces, ductal structures, or abnormal tissues) can help clinicians maintain spatial positioning of structures during the procedure. In some cases, the clinician may physically mark the tissue (e.g., by burning marks or deposition of marking materials on anatomical structures). In some cases, virtual markers that do not physically contact or change the tissue can be used to mark and track the tissue. Virtual markers can include, for example, freehand video dynamic drawings drawn by the clinician, graphic symbols, alphanumeric characters, geometric shapes, or other indicators generated by the clinician, imaging system, or control system of the robotic-assisted medical system. These virtual markers may be displayed as an overlay or integrated with the display of the field of view, and may persist under various illumination modes of the field of view, and may move with the anatomy to which the marker is associated.

[0031] Figure 1 is a flow chart illustrating a method 100 for associating a fiducial marker with a surface representation of a structure in a field of view. The methods described herein are illustrated as a set of operations or processes and continue to be described with reference to the attached figures. Not all of the illustrated processes may be performed in all embodiments of the method. Additionally, one or more processes not explicitly illustrated therein may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more of the illustrated processes may be omitted. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., a processing unit of a control system such as control system 720), may cause the one or more processors to perform the one or more processes. In one or more embodiments, the processes may be performed by the control system.

[0032] At process 102, three-dimensional primary image data of a field of view may be received, for example, by a control system. Figure 2 A display system 200 is provided, which includes a display area 201 for displaying an image 202 of a field of view. The field of view image 202 can be generated by image data from an imaging instrument (e.g., an endoscopic imaging system 715 that can produce a stereoscopic endoscopic imaging stream) within an anatomical environment in a patient's anatomy, which can include tissue surfaces, tools, suture materials, and / or other naturally occurring or clinically introduced items. In this example, the image 202 can be a three-dimensional stereoscopic image, but in other examples, the image can be a two-dimensional image. The image 202 can be, for example, an intraoperative real-time video endoscopic image. The primary image data used to generate the image 202 can be obtained when the field of view is illuminated with visible spectrum (e.g., white) light in a standard imaging mode. The image 202 of the field of view can have, for example, an image reference system X based on the distal end of the endoscopic imaging system. I , Y I , Z I .

[0033] At process 104, a surface representation may be generated for the tissue surface in the field of view. The surface representation may be, for example, a mesh model, a dense point cloud model, or a continuous surface model. The surface representation may be generated from the 3D primary image data by any of one or more surface or volume rendering techniques, including, for example, simultaneous localization and mapping ("SLAM") methods that use a mesh representation or neural radiance fields ("NeRF") to capture a volumetric scene. Figure 3A An example of a surface representation 250 is provided, which may be a mesh model generated from 3D imaging data of a field of view, and Figure 3B An example of a surface representation 280 is provided, which can be a continuous surface model generated from 3D imaging data of the same field of view used to generate the surface representation 250. A plurality of points 282 can be associated with locations (e.g., pixels or voxels) on the surface representation 280. The points 282 can correspond to, for example, lines or other markings created with reference to corresponding image data, as described in more detail below. In some examples, the multiple surface representations 250, 280 can be generated from the same imaging data of the field of view. In some examples, the generated surface representations can be displayed, for example, on the display system 200.

[0034] At process 106, areas in the field of view may be identified with fiducial markers. As an example, Figure 4AA method 300 for identifying a region with a fiducial marker is shown. At process 302, 3D primary image data of a field of view may be generated therefrom and may be displayed on a display system. At process 304, a region on a surface representation of the field of view may be identified. For example, a user may interact with the displayed image via a user interface (e.g., a touch screen, a mouse, a keyboard, a manipulator of a robot-assisted medical system) to indicate a region in the field of view. For example, the region may be a region including structures of interest to the procedure, such as vasculature, organ surfaces, ductal structures, and / or tumors or other abnormal tissues. The region may represent tissues relevant to the procedure, including boundaries, proposed treatment locations, proposed resection lines, or other tissue regions that may be involved or avoided during the procedure. In some examples, the region may be identified based on a computational evaluation, such as by a recognition system or a control system based on image processing, reference to a model of a similar anatomical region, or other analysis of the 3D primary image data. At process 306, a fiducial marker may be generated. The fiducial marker may be a freehand video telestration (e.g., drawn by a user), a graphic symbol, an alphanumeric character, a geometric shape, or other indicator. The fiducial markers may be marked on the surface of the grid (ie the tissue / organ surface), but may also be located above or below the surface. The markers may be registered with nearby areas of the surface via a vector field representing the spatial relationship between the marked surfaces.

[0035] As another example of a method for identifying a region with a fiducial marker, Figure 4B Method 310 is shown. At process 312, enhanced image data of the field of view can be received from a secondary imaging modality. For example, the enhanced image data can be generated from a secondary, co-registered imaging modality (such as an imaging modality using additional or alternative wavelengths of light, or such as an image processing modality different from the primary image processing modality). The co-registered imaging modality can include, for example, fluorescence imaging (e.g., "firefly" fluorescence imaging available on some systems provided by Intuitive Surgical, Inc.), hyperspectral imaging, laser speckle contrast imaging, oxygenated / deoxygenated hemoglobin concentration imaging, Raman spectroscopy, and / or other analytical light imaging modalities.

[0036] An image from a co-registered imaging system may look very different than a color image from a visible spectrum imaging system. Tracking / mapping systems (such as SLAM) can handle the two different types of images by using color invariant features. For example, all images can be changed to grayscale to allow focus on features / shapes rather than the color of the image. This can allow a user to create a marker in fluorescence imaging and then track the created marker in visible light imaging.

[0037] In some examples, the enhanced image data of the field of view can be image data from a near-infrared fluorescence imaging system. The fluorescence imaging system can be integrated with the visible light endoscope imaging system (e.g., integrated into the imaging system 715), and can provide fluorescence imaging to visualize, for example, blood vessels, blood flow, bile ducts, and related tissue perfusion. In order to generate enhanced image data and evaluate tissues using the fluorescence imaging system, a fluorescent dye (e.g., indocyanine green "ICG") can be injected intravenously into the patient. ICG can bind to plasma proteins (such as albumin) in the blood and emit infrared signals when excited by a laser (e.g., at a wavelength of 803 nm) in situ. The laser can be emitted by an infrared excitation laser separated from the visible light endoscope, or can be emitted by an illuminator light emitting diode with an infrared excitation laser that is integrated into an endoscope that also images with visible light. The luminescent fluorescent dye can be detected by the control system, and a software algorithm can be used to color the fluorescent signal for display on the display system. The control system can allow the user to switch between visible light and fluorescence imaging modes from the operator console. As a more specific example, during a cholecystectomy, ICG provided intravenously may concentrate in the patient's bile and may become fluorescent under near-infrared spectrum light, allowing clinicians to identify bile duct structures that are critical in cholecystectomy. However, over time, ICG may dissipate, making continuous tracking of the bile duct difficult. Marking anatomical structures with a persistent marker at peak or near-peak fluorescence can allow continued identification even after the dye dissipates or in situations where high levels of fluorescence in background structures obscure the identified area.

[0038] At process 314, an area on the tissue surface can be identified in the enhanced image data of the field of view. The identification can be performed by a clinician or other user viewing the enhanced image data, or by a control system that uses image analysis techniques to recognize structures in the enhanced data. For example, at process 316, a user viewing the enhanced image data on a display can identify and thus identify an area of ​​the field of view corresponding to an area of ​​peak fluorescence. Alternatively or additionally, at process 318, an identification system (e.g., an image processing system of the control system 720 or an image processing system that communicates with the control system 720) can identify an area of ​​peak fluorescence. In some examples, the identification system can use an artificial intelligence system and machine learning to identify structures based on the characteristics of the enhanced image data. In some examples, the identification system can provide a recommended identification that can be confirmed, rejected, or modified by the user. The structures of interest identified in process 314 can depend on the type of procedure. For example, for a cholecystectomy procedure, structures such as the bile duct, the cystic artery, and the cystic duct can be identified. For a prostatectomy, a prostate tumor can be identified. For a urology or gynecology procedure, a ureter can be identified. For colorectal surgery, an area of ​​high perfusion can be identified.

[0039] At process 320 , fiducial markers may be generated based on the regions identified in the enhanced image data. Figure 5A A field of view 500 of a patient's anatomy captured by a fluorescence imaging system is shown, with an image reference frame X I , Y I , Z I . Structures in field of view 500 may include tissue 501 and tools 503. Tissue structures in region 502, such as blood vessels and perfused tissue, may glow under near-infrared light provided by the imaging system. To track the fluorescing structures, a user may create a fiducial marker 504, such as a curved line, that tracks the vascular structure even after the fluorescent dye begins to fade. The line may be manually generated with a user interface device, such as a touch screen, stylus, mouse, or manipulator of a robotic-assisted medical system. In some examples, the user interface device may constrain the input to two dimensions (e.g., X and Y directions). In some examples, if the region is identified based on a recognition system, the control system may generate line 504 to correspond to the pattern of identified peak fluorescence.

[0040] Reference again Figure 1 , method 100 may proceed to process 108, where the fiducial marker is associated with the surface representation. For example, the fiducial marker may be placed on a three-dimensional surface representation (e.g., a mesh model or a continuous surface model) of a region of the anatomy in the field of view. Portions of the fiducial marker may be placed at three-dimensional locations corresponding to the identified region. For example, Figure 5B shows a reference frame X R , YR , Z R A surface representation 510 is provided that corresponds to (e.g., is registered to) and represents tissue in the field of view 500. The surface representation 510 can be generated based on a depth map of the structure in the field of view 500. A sequence of points or a point cloud corresponding to the graphical marker 504 can be projected onto the surface representation 510 as a sequence of points or a point cloud 512 to mark the area 502. Each point 512 can correspond to an X, Y, Z coordinate on the surface representation 510. Therefore, even though the markers can be generated using a two-dimensionally constrained user interface, each point 512 can have a Z depth value along the surface representation 510. In some examples, the markers 504 can be sampled at a fixed pixel spacing to generate the points 512 projected onto the surface representation 510. For example, using a SLAM process, the depth map and stereo image calibration can be used to place the points 512 on a 3D vector map.

[0041] In some examples, the marker 504 can be rendered in the image reference system as a two-dimensional marker extending over the identified area. In other examples, based on the registered surface representation 510 and the projection point 512, the marker 504 can be rendered in the image reference system as a three-dimensional marker that fits the surface contour of the tissue in the field of view. By projecting the marker in the stereoscopic image, the marker 504 can be displayed as a three-dimensional overlay on the surface field of view 500. In various examples, the three-dimensional marker can be displayed as a wavy graphic line (e.g., a continuous line or a dashed line), a series of discrete symbols, or alphanumeric / character text. The marker can be drawn freehand or can be selected from pre-established graphic options. The properties of the marker 504 can be changed based on user selection, proximity of the tool in the field of view, or other user or system selection criteria. For example, the display of the marker can be turned on or off, the color of the marker can be changed, and / or the style of the marker (e.g., a continuous line or a dashed line) can be changed.

[0042] At optional process 110, the surface representation and fiducial markers may move in response to changes in the tissue surface in the field of view. Tissue changes may be due to motion, such as breathing, cardiac activity, surgical tool intervention, or other forces during a medical procedure. The appearance of changes in tissue in the field of view may also or alternatively be caused by changes in the position and / or orientation of the imaging system (e.g., an endoscope). Fig. 6A and Figure 6B As shown, the surface representations 250, 280 can be represented as captured tissue based on their movement in the field of view (relative to their position in the field of view). Figure 3A and Figure 3BThe configuration in the image reference system changes. For example, the surface representation can be updated at an interactive frame rate as part of a SLAM process that uses a real-time 3D vector map to track tissue deformation and endoscope movement. Changes in tissue in the field of view may be caused by tissue movement or changes in the orientation and / or orientation of the imaging system. As the surface representation 280 moves, a marker or point 282 associated with a position, pixel, or voxel on the surface representation moves with the representation. Therefore, when the tissue associated with the point 282 is displaced, stretched, occluded, dissected, or otherwise changed, the point 282 and the marker in the image reference system associated with the point are similarly changed. By using information from SLAM, markers (or any type of graphics or annotations) can be placed during deformation of the surface and / or during camera movement. This can allow real-time placement and visual feedback during initial digital fiducial marking. In some examples, the SLAM process can be a deformable SLAM process that performs simultaneous positioning and mapping of deformable or non-rigid surfaces, volumes, structures, and environments.

[0043] The display of the fiducial marker may also change as the surface representation and the point associated with the fiducial marker change. Figure 5A compared to, Fig. 7A A field of view 500 is shown with tissue 501 and tool 503 in different locations and orientations. Figure 7B Shown according to Fig. 7A The surface representation 510 is changed with the change of the field of view 500 in the image. The position of a point 512 in the surface representation reference frame associated with the position on the surface representation 510 is likewise changed. Figure 7C A field of view 500 is shown in which a fiducial marker 504 is displayed in an updated configuration corresponding to a change in point 512. Thus, as structures in the field of view 500 move, the fiducial marker 504 persists and moves with the identified anatomical structure. Thus, while the fluorescent dye dissipates, the associated critical anatomical structure can be tracked. Tracking the fiducial marker 504 can include updating the displacement, orientation, configuration, and / or deformation of the marker when the associated pixel, voxel, or other associated graphical element moves. The marker 504 can be stretched or bent so that the distal portion of the marker moves relative to the proximal portion of the marker when the associated tissue moves. The surface representation 510 can be updated using a surface and / or volume rendering technique (such as SLAM) applied to a stereoscopic endoscopic video stream, and thus the position of the point 512 and the marker 504 can be updated. The SLAM process can include a deformable SLAM process that can be used to generate updated surface and volume renderings in response to image system motion or tissue deformation while tracking the fiducial marker under the surface relative to the updated rendering.

[0044] At optional process 112, the fiducial markers may be displayed with the other co-registered image data. For example, the fiducial markers may be displayed with the 3D primary image data. Fig. 8A Surface representation 510 and points 512 are shown in a modified configuration. Figure 8B A field of view 500 is shown that is viewed with only visible light and without near infrared light to highlight key structures. Based on the registration of the field of view 500 in the image reference frame with the surface representation 510 in the representation reference frame, a line 504 corresponding to a point 512 can be displayed as a three-dimensional overlay or otherwise integrated with the field of view 500 illuminated by visible light. The three-dimensional nature of the marker 504 can be achieved by presenting a stereoscopic image of the marker and the field of view illuminated by visible light to the user. Thus, the fiducial marker 504 can persist even in the absence of a fluorescent image to allow the clinician to track key structures.

[0045] In some examples, the endoscopic camera can be moved so that the new camera field of view does not include structures marked with fiducial markers. In this example, rendering technology (such as SLAM) can create a map of the entire surgical field or only critical areas. Therefore, when the camera is moved back to visualize the field of view where the digital fiducials were originally drawn, the system will automatically recognize and reposition to that area and display the digital fiducials in the correct location.

[0046] In some examples, the secondary imaging modality provides subsurface image data and selected structural images, and video dynamic rendering, labels, markers, or other information associated with the subsurface structure can be displayed with the primary image data or a surface representation of the primary image data. In some examples, the information associated with the subsurface structure may persist with the displayed primary image data as the primary imaging system moves.

[0047] Fig.9A6 is a flow chart showing a method 600 of associating a fiducial marker corresponding to a subsurface structure with an image or representation of a field of view. At process 602, for example, primary image data of a field of view may be received by a control system. The primary image data may be received from a primary imaging modality and may be three-dimensional image data. The process may be substantially similar to process 102 described above. At process 604, secondary image data may be received from a secondary imaging modality having a secondary field of view. For example, the image data may be generated from a secondary imaging modality, such as an imaging modality that uses high-frequency sound waves (e.g., ultrasound imaging) to image subsurface tissue structures. The secondary image data may, for example, provide a subsurface image of tissue within the field of view of the primary image data. In some examples, the secondary image data may be generated before receiving the primary image data. For example, prior to an endoscopic imaging procedure, an ultrasound scan may be performed to generate the secondary image data. In other examples, the secondary image data may be ultrasound imaging data generated by an ultrasound probe within or extending along an endoscope for generating the primary image data.

[0048] At process 606, the secondary image data can be registered with the primary image data. Registering the primary image dataset and the secondary image dataset can include spatially aligning the two image datasets using any of a variety of registration techniques, including feature matching and / or image system tracking, or transforming one of the datasets into the coordinate system of the other. In some examples, the registration process 606 can be performed at different stages of the method (e.g., after process 608 or after process 614).

[0049] At process 608, a subsurface region may be identified in the secondary imaging data. For example, the identified region may be a structure such as a tumor, a tumor boundary, an anatomical duct (e.g., a ureter), a vasculature, or another anatomical structure of interest or related to a medical procedure located below the surface of the tissue visible in the field of view of the primary image data. Secondary imaging data (such as ultrasound imaging data) may visualize subsurface structures. Identification may be performed by a clinician or another user viewing the secondary image data, or by a control system using image analysis techniques to identify structures in the secondary image data. For example, at process 610, a user viewing the secondary image data on a display may identify and thus identify a region corresponding to a structure of interest. Alternatively or additionally, at process 612, an identification system (e.g., an image processing system of the control system 720 or an image processing system in communication with the control system 720) may identify a region of interest. In some examples, the identification system may use an artificial intelligence system and machine learning to identify structures based on characteristics of the secondary image data. In some examples, the identification system may provide a recommended identification that may be confirmed, rejected, or modified by the user. The structure of interest identified at process 608 may depend on the type of procedure. For a prostatectomy, the borders of a prostate tumor can be identified. For a urology or gynecology procedure, the ureters can be identified.

[0050] At process 614, a fiducial marker may be generated based on an area identified in the secondary image data. The fiducial marker may be, for example, a video telestration that marks a tissue boundary, outlines a structure, indicates fluid flow, or otherwise marks an area of ​​interest. In some examples, the fiducial marker may be a portion of the secondary image data (such as a portion depicting a tumor). In some examples, the fiducial marker may be a sign or label indicating a clinical intervention or an area of ​​interest. In some examples, the fiducial marker may include a number or text character, symbol, line, shape, or other graphical representation. In some examples, the fiducial marker may be manually generated with a user interface device (such as a touch screen, stylus, mouse, or manipulator of a robotic-assisted medical system).

[0051] At process 616, the fiducial markers can be displayed with the primary image data. For example, the subsurface fiducial markers can be displayed as an underlying layer beneath a see-through window, a semi-transparent window, or other type of opaque or visually distinct rendering of the tissue surface in the field of view of the primary image data. In other examples, the fiducial markers can be integrated with the primary image data, superimposed on the primary image data, or placed on a three-dimensional surface representation (e.g., a mesh model or a continuous surface model) of a region of the anatomical structure in the field of view generated from the primary image data. For example, Fig. 9BA field of view 630 captured by primary image data is shown. The field of view 630 includes tissue 632 and a tool 634. A fiducial marker 636 associated with an identified subsurface region of interest from the secondary ultrasound imaging data is shown with the field of view 630 of the primary image data. In this example, the fiducial marker 636 may mark a border or margin of a tumor and may be visible as an underlying layer beneath a semi-transparent surface 638 of the tissue 632 in the field of view 630.

[0052] At process 618, the fiducial markers and can persist as the main imaging system changes position or orientation, and can move with the tissue in the field of view, or can move within the field of view along with the anatomical structure associated with the marker. Tissue movement may be due to, for example, breathing, cardiac activity, surgical tool intervention or other forces during a medical procedure. Tissue in the field of view may also or alternatively move due to changes in the position and / or orientation of an imaging system (e.g., an endoscope). For example, fiducial markers can be added to an interactive frame rate and updated at an interactive frame rate as part of a SLAM process that uses a real-time 3D vector map to track tissue deformation and endoscope motion. Changes in tissue in the field of view may be caused by changes in the position and / or orientation of the tissue movement or the imaging system. As the surface representation of the tissue or tissue moves, the fiducial markers associated with the position, pixel or voxel representing the tissue may also have corresponding motion. For example, a deformable SLAM process can be used to generate updated surface and volume renderings in response to image system motion or tissue deformation while tracking subsurface fiducial markers. Therefore, when the tissue associated with the fiducial marker is displaced, stretched, blocked, dissected or otherwise changed, the fiducial marker in the field of view is similarly changed. As the surface representation and the points associated with the fiducial marker change, the display of the fiducial marker may also be changed. Therefore, as the structure in the field of view 630 moves, the fiducial marker 636 persists and moves with the superimposed anatomical structure. Therefore, even in the absence of current secondary image data, the associated key anatomical structure can be tracked. Tracking the fiducial marker 636 can include updating the displacement, orientation, configuration and / or deformation of the marker as the associated pixel, voxel or other associated graphic element moves. The marker 636 can be stretched or bent. The fiducial marker 636 can be updated using surface and / or volume rendering techniques (such as SLAM applied to stereoscopic endoscopic video streams).

[0053] Figures 10 to 12 Together, an overview of a medical system 710 that can be used for medical procedures including, for example, diagnostic, therapeutic, or surgical procedures is provided. The fiducial marker generation and tracking examples provided above can be used in the context of a medical system 710. The medical system 710 is located in a medical environment 711. In Fig.10, medical environment 711 is depicted as an operating room. In other embodiments, medical environment 711 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may be performed. In still other embodiments, medical environment 711 may include an operating room and a control area located outside of the operating room.

[0054] In one or more embodiments, the medical system 710 can be a robotic-assisted medical system under teleoperation control of an operator (e.g., a surgeon, clinician, physician, etc.). In alternative embodiments, the medical system 710 can be under partial control of a computer programmed to perform a medical procedure or subroutine. In still other alternative embodiments, the medical system 710 can be a fully automated medical system under full control of a computer programmed to perform a medical procedure or subroutine with the medical system 710. An example of a medical system 710 that can be used to implement the systems and techniques described in the present disclosure is the da Vinci® surgical system manufactured by Intuitive Surgical, Inc. of Sunnyvale, California.

[0055] like Fig.10 As shown, medical system 710 generally includes component 712, which can be mounted to or positioned near an operating table T, on which a patient P is positioned. Component 712 can be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, component 712 can be a teleoperated component. The teleoperated component can be referred to as, for example, a teleoperated arm cart. A medical device system 714 and an endoscopic imaging system 715 are operably coupled to component 712. An operator input system 716 allows an operator O or other type of clinician to view an image of a surgical site or an image representing a surgical site, and to control the operation of the medical device system 714 and / or the endoscopic imaging system 715.

[0056] The medical device system 714 may include one or more medical devices. In embodiments where the medical device system 714 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 715 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0057] The operator input system 716 may be located at an operator's control console, which may be located in the same room as the operating table T. In some embodiments, the operator O and the operator input system 716 may be located in a different room from the patient P or in a completely different building. The operator input system 716 typically includes one or more control devices for controlling the medical instrument system 714. The control device(s) may include one or more of any number of various input devices, such as a handle, a joystick, a trackball, a data glove, a trigger-gun, a foot pedal, a hand controller, a voice recognition device, a touch screen, a body motion or presence sensor, and other types of input devices.

[0058] In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical device(s) of the medical device system 714 to provide telepresence to the operator, which is a perception that the control device(s) are integral to the instrument, giving the operator a strong sense of direct control of the instrument, as if present at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the operator. In some embodiments, the control device(s) are manual input devices that are capable of moving in six degrees of freedom, and which may also include an actuatable handle for actuating the instrument (e.g., for closing a gripping jaw end effector, applying electrical potentials to electrodes, delivering drug therapy, and actuating other types of instruments).

[0059] When the operator O views the surgical site through the operator input system 716, the assembly 712 can support and manipulate the medical instrument system 714. The image of the surgical site can be obtained by the endoscopic imaging system 715, which can be manipulated by the assembly 712. The assembly 712 can include the endoscopic imaging system 715, and can also similarly include multiple medical instrument systems 714. The number of medical instrument systems 714 used at one time will generally depend on factors such as the diagnostic or surgical procedure to be performed and the space constraints in the operating room. The assembly 712 can include one or more non-servo control links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as a setting structure) and the kinematic structure of the manipulator. When the manipulator takes the form of a teleoperated manipulator, the assembly 712 is a teleoperated assembly. The assembly 712 includes multiple motors that drive inputs on the medical instrument system 714. In one embodiment, these motors move in response to commands from a control system (e.g., control system 720). The motor includes a drive system that, when coupled to the medical device system 714, can advance the medical device into a natural or surgically created anatomical orifice. Other electric drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation around the X, Y, Z Cartesian axes). In addition, the motor can be used to actuate an articulated end effector of the medical device to grasp tissue in the jaws of a biopsy device or the like. The medical device of the medical device system 714 can include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, tweezers, a grasper, scissors, or a clip applier.

[0060] The medical system 710 also includes a control system 720. The control system 720 includes at least one memory 724 and at least one processor 722 for implementing control between the medical device system 714, the operator input system 716, and other auxiliary systems 726, which may include, for example, imaging systems, image recognition systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician can circulate within the medical environment 711 and can access, for example, the components 712 during a setup procedure, or view a display of the auxiliary system 726 (e.g., display system 200) from the patient's bedside.

[0061] Although depicted as being external to component 712 in FIG. 9 , in some embodiments, control system 720 may be fully contained within component 712. Control system 720 also includes programming instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. Although control system 720 is shown as a single block in the simplified schematic diagram of FIG. 9 , control system 720 may include two or more data processing circuits, with a portion of the processing optionally being performed on or near component 712, another portion of the processing being performed at operator input system 716, and the like.

[0062] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programming instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into a variety of other aspects of the systems described herein (including teleoperation systems). In one embodiment, the control system 720 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0063] In some embodiments, the control system 720 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 714. In response to the feedback, the servo controller transmits a signal to the operator input system 716. The servo controller(s) may also transmit a signal that instructs the assembly 712 to move the medical device system(s) 714 and / or the endoscopic imaging system 715, which extend to an internal surgical site within the patient's body via an opening within the body. Any suitable conventional or specialized servo controller may be used. The servo controller may be separate from or integrated with the assembly 712. In some embodiments, the servo controller and assembly 712 are provided as part of a teleoperated arm cart positioned near the patient's body.

[0064] The control system 720 can be coupled to the endoscopic imaging system 715 and can include a processor for processing the captured images for subsequent display, such as to an operator on an operator's console or on another suitable display located locally and / or remotely. For example, in the case of a stereoscopic endoscope, the control system 720 can process the captured images to present a coordinated stereoscopic image of the surgical site to the operator. Such coordination can include alignment between relative images and can include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0065] In alternative embodiments, the medical system 710 may include more than one component 712 and / or more than one operator input system 716. The exact number of components 712 will depend on factors such as the surgical procedure and the space constraints within the operating room. The operator input systems 716 may be collocated, or they may be located in separate locations. Multiple operator input systems 716 allow more than one operator to control one or more components 712 in various combinations. The medical system 710 may also be used to train and rehearse medical procedures.

[0066] Fig.11 712, which may be referred to as a patient side cart, surgical cart, teleoperated arm cart, manipulator assembly, or surgical robot. The illustrated assembly 712 provides manipulation of three surgical tools 730a, 730b, and 730c (e.g., medical instrument system 714) and an imaging device 728 (e.g., endoscopic imaging system 715) (such as a stereo endoscope for capturing images of the procedural site). The imaging device may transmit signals to the control system 720 via cable 756. Manipulation is provided by a teleoperated mechanism having multiple joints. The imaging device 728 and surgical tools 730a-c may be positioned and manipulated through an incision of the patient such that a kinematic remote center is maintained at the incision to minimize the size of the incision. The image of the surgical site may include an image of the distal end of the surgical tools 730a-c (when the distal end of the surgical tools 730a-c is positioned within the field of view of the imaging device 728).

[0067] Assembly 712 includes a drivable base 758. Drivable base 758 is connected to a telescoping column 757, which allows the height of arm 754 to be adjusted. Arm 754 may include a rotary joint 755, which rotates and moves up and down. Each arm 754 can be connected to an orientation platform 753. Arm 754 can be marked to facilitate troubleshooting. For example, each arm 754 can be decorated with different numbers, letters, symbols, other identifiers or combinations thereof. Orientation platform 753 can be capable of rotating 760 degrees. Assembly 712 can also include a telescopic horizontal cantilever 752 for moving orientation platform 753 in a horizontal direction.

[0068] In this example, each arm 754 is connected to a manipulator arm 751. The manipulator arm 751 can be directly connected to a medical instrument (e.g., one of the surgical tools 730a-c). The manipulator arm 751 can be teleoperable. In some examples, the arm 754 connected to the orientation platform 753 may not be teleoperable. Instead, such arms 754 can be positioned as needed before the operator O begins to operate using the teleoperated components. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with arms may change during the procedure.

[0069] Endoscopic imaging systems (e.g., endoscopic imaging system 715 and imaging device 728) can be provided in various configurations (including rigid or flexible endoscopes). Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end of the endoscope. Flexible endoscopes use one or more flexible optical fibers to transmit images. Digital image-based endoscopes have a "chip-on-the-tip" design in which a distal digital sensor (such as one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a more accurate depth perception to the viewer. Stereoscopic endoscopic instruments employ stereo cameras to capture stereoscopic images of a patient's anatomy. Endoscopic instruments may be fully sterilizable assemblies in which the endoscope cable, handle, and shaft are all rigidly coupled and hermetically sealed.

[0070] Fig.12 7 is a perspective view of an embodiment of an operator input system 716 at an operator's console. The operator input system 716 includes a display system (e.g., display system 200) having a left eye display 732 and a right eye display 734 for presenting a coordinated stereoscopic view of a surgical environment that enables depth perception to the operator O. The left eye display 732 and the right eye display 734 can be components of a display system 735 (e.g., display system 200). In other embodiments, the display system 735 can include one or more other types of displays. The display system 735 can present, for example, images captured by the imaging system 715 to display the endoscopic field of view to the operator. The endoscopic field of view can be enhanced by virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the viewer.

[0071] The operator input system 716 also includes one or more input control devices 736, which in turn cause the assembly 712 to manipulate one or more instruments of the endoscopic imaging system 715 and / or the medical instrument system 714. The input control device 736 can provide the same degrees of freedom as the instrument with which it is associated to provide telepresence to the operator O, or provide a sense that the input control device 736 is integral with the instrument so that the operator has a strong sense of direct control of the instrument. To this end, position, force, and tactile feedback sensors (not shown) can be used to transmit position, force, and tactile sensations from medical instruments (e.g., surgical tools 730a-c or imaging devices 728) back to the operator's hand through the input control device 736. The input control device 739 is a foot pedal that receives input from the user's foot. Aspects of the operator input system 716, assembly 712, and auxiliary system 726 can be adjustable and customizable to meet the physical needs, skill level, or preferences of the operator O.

[0072] Elements described in detail with reference to one embodiment, implementation, or application may optionally be included (where feasible) in other embodiments, implementations, or applications that are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless otherwise specifically described, unless one or more elements will render an embodiment or implementation inoperative, or unless two or more elements provide conflicting functions.

[0073] Any changes and further modifications to the described devices, systems, apparatuses, methods, and any further applications of the principles of the present disclosure are fully contemplated as would be normally thought of by a person skilled in the art to which the present disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described according to an illustrative embodiment may be used or omitted according to the applicable circumstances of other illustrative embodiments. For the sake of brevity, multiple iterations of these combinations will not be described separately.

[0074] Various systems and portions of systems have been described in terms of their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or portion of an object (three rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or portion of an object in at least one translational degree of freedom and the orientation of the object or portion of an object in at least one rotational degree of freedom (up to six total degrees of freedom).

[0075] Although some of the examples described herein relate to surgical procedures or instruments, or medical procedures and medical instruments, the disclosed techniques are optionally applicable to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts. Other example applications relate to cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, and training medical or non-medical personnel. Additional example applications include procedures for tissue removed from human or animal anatomical structures (without returning to the human or animal anatomical structures), and procedures performed on human or animal corpses. In addition, these techniques can also be used for surgical and non-surgical medical treatment or diagnostic procedures.

[0076] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programmed instructions, input information, and output information. The term "computer" and similar terms such as "processor" or "controller" or "control system" are similar.

[0077] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the broad invention rather than limiting thereof, and that the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art.

Claims

1. A system comprising: processor; and A memory having computer readable instructions stored thereon, which, when executed by the processor, cause the system to perform the following operations: receiving three-dimensional primary image data from an imaging system having a field of view; generating a surface representation of a tissue surface in the field of view; identifying an area in the field of view with a fiducial marker; associating the fiducial marker with the surface representation; and The surface representation and the fiducial marker are moved in response to movement of the tissue surface in the field of view.

2. The system of claim 1, wherein the three-dimensional primary image data has a field of view illuminated by visible spectrum light. The system of claim 1 , wherein the surface representation comprises a mesh model. The system of claim 1 , wherein the surface representation comprises a point cloud model. The system of claim 1 , wherein the surface representation comprises a continuous surface model.

6. The system of claim 1, wherein the surface representation is generated by a simultaneous positioning and mapping rendering technique.

7. The system of claim 1, wherein the surface representation is generated by a neural radiance field rendering technique.

8. The system of claim 1, wherein identifying an area in the field of view with a fiducial marker comprises receiving enhanced image data of the field of view from a secondary imaging modality.

9. The system of claim 8, wherein the enhanced image data is received from a co-registered secondary imaging modality.

10. The system of claim 8, wherein the secondary imaging modality comprises a fluorescence imaging modality.

11. The system of claim 8, wherein the secondary imaging modality is at least one of: a hyperspectral imaging modality, a laser speckle contrast imaging modality, an oxygenated / deoxygenated hemoglobin concentration imaging modality, or a Raman spectroscopy imaging modality.

12. The system of claim 8, wherein identifying an area comprises receiving a user indication of a structure visible in the enhanced image data.

13. The system of claim 8, wherein identifying regions comprises receiving identification of structures visible in the enhanced image data from a trained recognition system.

14. The system of claim 8, wherein the fiducial marker is visible when the enhanced image data is suppressed or dissipated.

15. The system of claim 1, wherein the computer readable instructions, when executed by the processor, further cause the system to display the fiducial marker associated with the identified area.

16. The system of claim 1, wherein the fiducial marker is a three-dimensional fiducial marker.

17. The system of claim 1, wherein the fiducial markers correspond to a set of sampling points projected onto the surface representation.

18. The system of claim 1, wherein moving the surface representation comprises tracking displacement and deformation of the tissue surface in the field of view using surface rendering techniques.

19. The system of claim 1, wherein moving the fiducial marker comprises tracking displacement or deformation of the fiducial marker associated with the tissue surface.

20. The system of claim 1, wherein moving the fiducial marker comprises tracking displacement or deformation of the fiducial marker associated with motion of the imaging system.

21. The system of claim 1, further comprising modifying the surface representation by a simultaneous localization and mapping rendering technique in response to displacement or deformation of the tissue surface.

22. The system of claim 1, further comprising modifying the surface representation by a simultaneous localization and mapping rendering technique in response to motion of the imaging system.

23. The system of claim 1, wherein a first portion of the fiducial marker moves relative to a second portion of the fiducial marker in response to the movement of the tissue surface.

24. The system of claim 1, wherein the computer readable instructions, when executed by the processor, further cause the system to display a moving fiducial marker as an overlay on the three-dimensional primary image data.

25. A system comprising: processor; and A memory having computer readable instructions stored thereon, which, when executed by the processor, cause the system to perform the following operations: receiving three-dimensional primary image data from an imaging system having a field of view; receiving enhanced image data of the field of view from a secondary imaging modality; identifying a region on a tissue surface in the enhanced image data of the field of view; and A fiducial marker is generated that is associated with the identified region.

26. The system of claim 25, wherein the enhanced image data is received from a co-registered secondary imaging modality.

27. The system of claim 25, wherein the secondary imaging modality is a fluorescence imaging modality.

28. The system of claim 25, wherein the secondary imaging modality is at least one of: a hyperspectral imaging modality, a laser speckle contrast imaging modality, an oxygenated / deoxygenated hemoglobin concentration imaging modality, or a Raman spectroscopy imaging modality.

29. The system of claim 25, wherein identifying an area comprises receiving a user indication of a structure visible in the enhanced image data.

30. The system of claim 25, wherein identifying regions comprises receiving identification of structures visible in the enhanced image data from a trained recognition system.

31. The system of claim 25, wherein the computer readable instructions when executed by the processor further cause the system to: generating a surface representation of the tissue surface in the field of view; and The surface representation and the fiducial marker are moved in response to movement of the tissue surface in the field of view.

32. The system of claim 31 , wherein moving the surface representation comprises tracking displacement and deformation of the tissue surface in the field of view using surface rendering techniques.

33. The system of claim 31 , wherein moving the fiducial marker comprises tracking displacement and deformation of the fiducial marker associated with the tissue surface.

34. The system of claim 31 , wherein a first portion of the fiducial marker moves relative to a second portion of the fiducial marker in response to movement of the tissue surface.

35. The system of claim 31, wherein: The computer readable instructions, when executed by the processor, further cause the system to display the moving fiducial marker as an overlay on the three-dimensional primary image data.

36. The system of claim 25, wherein the three-dimensional primary image data has a field of view illuminated by visible spectrum light.

37. The system of claim 25, wherein: The computer readable instructions, when executed by the processor, further cause the system to generate a surface representation of the tissue surface in the field of view.

38. The system of claim 37, wherein the surface representation comprises a mesh model.

39. The system of claim 37, wherein the surface representation comprises a point cloud model.

40. The system of claim 37, wherein the surface representation comprises a continuous surface model.

41. The system of claim 37, wherein the surface representation is generated by a simultaneous positioning and mapping rendering technique.

42. The system of claim 37, wherein the surface representation is generated by a neural radiance field rendering technique.

43. The system of claim 25, wherein the fiducial marker is visible when the enhanced image data is suppressed or dissipated.

44. A system comprising: processor; and A memory having computer readable instructions stored thereon, which, when executed by the processor, cause the system to perform the following operations: receiving primary image data from a primary imaging system having a primary field of view; receiving secondary image data from a secondary imaging system having a secondary field of view; registering the primary image data and the secondary image data; identifying an area in the secondary field of view with a fiducial marker; displaying the fiducial marker together with the primary image data; and The fiducial marker is moved in response to motion of the primary imaging system or deformation of structures in the primary field of view.

45. The system of claim 44, wherein the secondary image data comprises ultrasound image data.

46. ​​The system of claim 44, wherein the fiducial markers correspond to boundaries of structures in the secondary image data.

47. The system of claim 44, wherein the fiducial marker is a video dynamically rendered generated in the secondary field of view.

48. The system of claim 44, wherein displaying the fiducial marker comprises displaying the fiducial marker as an underlayer beneath a semi-transparent tissue image in the primary field of view.

49. The system of claim 44, wherein moving the fiducial marker is performed by a simultaneous localization and mapping rendering technique in response to motion of the primary imaging system or deformation of structures in the primary field of view.

Images