Anatomical scene visualization system and method

Abstract

An illustrative system can be configured to acquire a visual image of an anatomical scene including an imaging probe, and to enhance the visual image using a composite element that indicates a region of the anatomical scene imaged by the imaging probe within the visual image. The composite element may include spatial markers. The visualization system can also be configured to instruct a display device to display the enhanced image. Furthermore, the visualization system can be configured to instruct a display device to display a probe image captured by the imaging probe at a location outside the composite element.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 033,668, filed June 2, 2020, the contents of which are incorporated herein by reference in their entirety. Background Technology

[0003] In medical procedures, imaging modalities such as endoscopy can be used to capture images of the anatomical scene. In some scenarios, one or more additional imaging modalities can be used to capture additional images of the anatomical scene that can also be presented to medical team members. For example, ultrasound scanning, optical coherence tomography (OCT), and rapid evaporation ionization mass spectrometry (REIMS) are other imaging modalities that can be used to capture images of the anatomical scene. Images captured by different imaging modalities can be presented to medical team members (e.g., nurses, surgeons, etc.) so that they can view a visualization of the anatomical scene while performing medical procedures. Summary of the Invention

[0004] The following description provides a simplified overview of one or more aspects of the systems and methods described herein. This overview is not a comprehensive summary of all anticipated aspects, and its purpose is neither to identify key or critical elements of all aspects, nor to depict the scope of any or all aspects. Its sole purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed description presented below.

[0005] An illustrative system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to: acquire a visual image of an anatomical scene including an imaging probe; enhance the visual image with synthetic elements that indicate regions of the anatomical scene imaged by the imaging probe within the visual image, the synthetic elements including spatial markers within the regions; and instruct a display device to display the enhanced image.

[0006] Another illustrative system includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute instructions to: obtain a visual image of an anatomical scene including an imaging probe; obtain a probe image captured by the imaging probe; enhance the visual image with a composite element within the visual image indicating a region of the anatomical scene imaged by the imaging probe; and instruct a display device to display the enhanced image and the probe image, the probe image being positioned outside the composite element.

[0007] An illustrative method includes: obtaining a visual image of an anatomical scene including an imaging probe; enhancing the visual image with a synthetic element that indicates a region of the anatomical scene imaged by the imaging probe within the visual image, the synthetic element including spatial markers within the region; and instructing a display device to display the enhanced image.

[0008] Another illustrative method includes: obtaining a visual image of an anatomical scene including an imaging probe; obtaining a probe image captured by the imaging probe; enhancing the visual image with a synthetic element that indicates a region of the anatomical scene imaged by the imaging probe within the visual image; and instructing a display device to display the enhanced image and the probe image, which is positioned outside the synthetic element.

[0009] An illustrative non-transitory computer-readable medium storage instruction, which, when executed, instructs a processor of a computing device to: obtain a visual image of an anatomical scene including an imaging probe; enhance the visual image with a composite element that indicates a region of the anatomical scene imaged by the imaging probe within the visual image, the composite element including spatial markers within the region; and instruct a display device to display the enhanced image.

[0010] Another illustrative non-transitory computer-readable medium storage instruction, which, when executed, instructs a processor of a computing device to: obtain a visual image of an anatomical scene including an imaging probe; obtain a probe image captured by the imaging probe; enhance the visual image with a composite element within the visual image indicating a region of the anatomical scene imaged by the imaging probe; and instruct a display device to display the enhanced image and the probe image, the probe image being positioned outside the composite element. Attached Figure Description

[0011] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are merely examples and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals denote the same or similar elements.

[0012] Figure 1 An illustrative anatomical scene visualization system based on the principles described in this paper is presented.

[0013] Figure 2 Depicting Figure 1 An anatomical scene visualization system is configured to generate enhanced images of anatomical scenes based on the principles described herein.

[0014] Figure 3A An illustrative anatomical scene is depicted based on the principles described in this article.

[0015] Figure 3B Depicting the principles described in this article Figure 3A Illustrative visual images of anatomical scenes.

[0016] Figures 4A-7B depictions of anatomical scenes are enhanced with synthetic elements. Figure 3A depictions of anatomical scenes are enhanced with synthetic elements.

[0017] Figure 8 depictions of anatomical scenes are enhanced with synthetic elements.

[0018] Figures 9-11 depictions of anatomical scenes are enhanced with synthetic elements.

[0019] Figure 12 depictions of anatomical scenes are enhanced with synthetic elements. DETAILED DESCRIPTION

[0020] Anatomical scene visualization systems and methods are described herein. The systems and methods are configured to enhance one or more visual images of an anatomical scene and provide a display image including the enhanced image for display in any of the ways described herein. In certain examples, an anatomical scene visualization system (“visualization system”) can be configured to enhance a visual image of an anatomical scene with a synthetic element, the visual image being captured by a first imaging modality such as an endoscope, the synthetic element indicating a region of the anatomical scene that is imaged by a second imaging modality such as an imaging probe at the anatomical scene (e.g., an ultrasound probe positioned at the anatomical scene). The synthetic element can help a user (e.g., a member of a medical team) viewing the enhanced image to identify the region of the anatomical scene that is imaged by the imaging probe, while allowing at least a portion of the region to be depicted in the visual image captured by the first imaging modality to remain visible in the enhanced visual image (e.g., by not substantially obscuring the region imaged by the imaging probe as depicted in the visual image captured by the first imaging modality).

[0021] As used herein, a synthetic element refers to a graphical element that can be added to an image captured by an imaging device. The synthetic element can be added to the image to form an enhanced image including the image enhanced with the synthetic element. The synthetic element can be added to the image in any appropriate manner, such as by overlaying the synthetic element on the image, integrating the synthetic element into the image, etc. Illustrative examples of images of anatomical scenes enhanced with synthetic elements and content associated with the synthetic elements are described herein.

[0022] In some examples, the visualization system can be configured to provide an enhanced image (e.g., a visual image enhanced with a synthetic element) and a probe image captured by the imaging probe for display on a display device. The probe image can be positioned outside the synthetic element such that it does not overlap with the depiction of an area of ​​the anatomical scene imaged by the imaging probe in the synthetic element or the visual image. For example, the probe image can be presented in a corner area of ​​the visual image to avoid interfering with (e.g., obscuring) the view of the area of ​​the anatomical scene imaged by the imaging probe, as represented in the visual image. As another example, the probe image can be presented outside the visual image (e.g., presented along with the visual image but not overlapping with it in the displayed view). The probe image can help a user viewing the displayed image to simultaneously view the probe image and the area of ​​the anatomical scene imaged by the imaging probe, as depicted in the visual image captured by a first imaging modality. An illustrative example of such a display image is described herein.

[0023] In some examples, the visualization system can be configured to enhance the visual image using preview placeholders that indicate the display location of the probe image to be captured by the imaging probe. The preview placeholder can represent a location where the probe image to be captured by the imaging probe can be presented (e.g., a location on a visual image of an anatomical scene). The preview placeholder can help the user viewing the enhanced image make any desired adjustments before the probe image is captured and presented, such as adjusting the viewpoint of the first imaging modality of the endoscope, such as when capturing an image of an anatomical scene.

[0024] The systems and methods described herein offer a variety of advantages and benefits. For example, they can provide visualization of anatomical scenes that are visually realistic and / or intuitive for medical team members (e.g., surgeons), reducing the complexity of medical procedures for them and / or allowing them to simultaneously, conveniently, and / or intuitively visualize images captured by different imaging modalities (e.g., images of surface and subsurface anatomy). In some examples, one or more visualizations can facilitate accurate, efficient, and / or intuitive manipulation of imaging probes within the anatomical scene. These and other advantages and benefits of the systems and methods described herein will be readily apparent.

[0025] Figure 1An illustrative anatomical scene visualization system 100 (System 100) is depicted, configured to perform operations to provide visualizations of anatomical scenes, including any illustrative example of anatomical scene visualizations described herein. In some examples, System 100 may be configured to enhance a visual image of an anatomical scene captured by a first imaging modality, such as an endoscope, in any manner described herein. System 100 may be included in, implemented by, or connected to one or more components of a computer-assisted medical system. For example, System 100 may be implemented by one or more components of a computer-assisted medical system. As another example, System 100 may be implemented by a separate computing system communicatively coupled to a computer-assisted medical system. An illustrative computer-assisted medical system is further described below.

[0026] System 100 may include a storage facility 102 and a processing facility 104 selectively and communicatively coupled to each other. Each of facilities 102 and 104 may include or be implemented by one or more physical computing devices, which include hardware and / or software components such as processors, memory, memory drives, communication interfaces, instructions stored in memory for execution by the processor, etc. Although facilities 102 and 104 are... Figure 1 While facilities 102 and 104 are shown as separate facilities, they may be combined into a single facility or divided into more facilities that can serve a particular implementation. In some examples, each of facilities 102 and 104 may be distributed among multiple devices and / or multiple locations that can serve a particular implementation.

[0027] Storage facility 102 may maintain (e.g., store) executable data used by processing facility 104 to perform one or more of the operations described herein. For example, storage facility 102 may store instructions 106 that can be executed by processing facility 104 to perform one or more of the operations described herein. Instructions 106 may be implemented by any suitable application, software, code, and / or other instance of executable data. Storage facility 102 may also maintain any data received, generated, managed, used, and / or sent by processing facility 104.

[0028] Processing facility 104 may be configured to perform (e.g., execute instructions 106 stored in storage facility 102 to perform) various operations associated with providing visualization of anatomical scenes, including any illustrative visualizations described herein. Examples of such operations will now be described in more detail with reference to the remaining figures. In the following description, any references to operations performed by system 100 shall be understood as being performed by processing facility 104 of system 100.

[0029] Figure 2 Configuration 200 is depicted, in which system 100 is configured to generate enhanced images of an anatomical scene. As shown, configuration 200 may include multiple imaging modalities 202 (e.g., imaging modalities 202-1 and 202-2), which are configured to capture images 204 of the anatomical scene 206 (e.g., image 204-1 captured by imaging modal 202-1 and image 204-2 captured by imaging modal 202-2).

[0030] Anatomical scene 206 may include any volumetric space associated with anatomical structures, such as volumetric space associated with a medical procedure. For example, anatomical scene 206 may include any one or more parts of the body, such as the anatomical structure 208 of a patient's body (e.g., tissues, etc.) within the space associated with the medical procedure. In some examples, anatomical scene 206 may be entirely located within the body and may include space near the location within the body where a medical procedure is planned, is being performed, or has been performed. For example, for a minimally invasive medical procedure performed on tissue inside a patient, anatomical scene 206 may include the tissue, the anatomical structures beneath the tissue, and the space surrounding the tissue where, for example, a medical instrument used to perform the medical procedure is located. In other examples, anatomical scene 206 may be at least partially located outside the body. For example, for an open medical procedure performed on a patient, a portion of anatomical scene 206 (e.g., the tissue being operated on) may be inside the patient, while another portion of anatomical scene 206 (e.g., the space around the tissue where one or more medical instruments may be located) may be outside the patient. Anatomical scene 206 may include a workspace in which medical procedures are performed, such as an actual real-world workspace associated with a patient, and in which one or more medical devices are used to perform medical procedures on the patient. In some examples, the anatomical scene associated with a medical procedure, such as a surgical procedure, may be referred to as a "medical scene" or a "surgical scene".

[0031] As used herein, a medical procedure can include any procedure associated with anatomical structures, including any diagnostic or therapeutic procedure (e.g., surgical procedure, treatment procedure, etc.) in which manual, remote, computer-assisted, and / or instrumental techniques are used on a patient to investigate or treat the patient's physical condition. A medical procedure can refer to any stage of a medical procedure, such as the preoperative, surgical (i.e., intraoperative), and postoperative stages of a medical procedure.

[0032] Imaging modality 202 can be configured and / or used to capture images 204 of the anatomical scene 206. Such capture is achieved by... Figure 2The dashed line 210 in the figure represents the image. Imaging modal 202 can each capture images 204 of the anatomical scene 206 in any suitable manner and at any suitable time. Thus, one or more imaging modalities 202 can capture images 204 of the anatomical scene 206 during one or more preoperative, intraoperative, and / or postoperative phases of a medical procedure.

[0033] Imaging modality 202 may include any set of different imaging modalities that can be used to capture images of an anatomical scene. Examples of imaging modality 204 include, but are not limited to, endoscopic imaging performed by an endoscope, ultrasound imaging performed by an ultrasound machine, optical coherence tomography (OCT) imaging performed by an OCT machine, and rapid evaporative ionization mass spectrometry (REIMS) imaging performed by a REIMS machine. In other examples, any suitable additional or alternative imaging modalities may be used. In some embodiments, imaging modality 202-1 may include endoscopic imaging performed by an endoscope, and imaging modality 202-2 may include any different imaging modalities, such as ultrasound imaging performed by an ultrasound machine (e.g., an ultrasound probe), OCT imaging performed by an OCT machine, or REIMS imaging performed by a REIMS machine. In such embodiments, imaging modality 202-1 may capture image 204-1 as an endoscopic image of anatomical scene 206, and imaging modality 202-2 may capture image 204-2 as an ultrasound image, OCT image, or REIMS image of anatomical scene 206. In some implementations, imaging mode 202-2 is an imaging probe located in anatomical scene 206, such as an ultrasound probe in anatomical scene 206.

[0034] In some examples, imaging modality 202-1 may be configured to capture images of surface anatomical structures included in anatomical scene 206 (e.g., the outer surface of tissue included in the anatomical scene), and imaging modality 202-2 may be configured to capture images of subsurface anatomical structures included in anatomical scene 206 (e.g., subsurface tissue behind the outer surface of tissue included in the anatomical scene). For example, imaging modality 202-1 may include endoscopic imaging by an endoscope capturing images of surface tissue within the patient, and imaging modality 202-2 may include ultrasound, OCT, or REIMS imaging capturing images of subsurface tissue located behind the endoscope and hidden outside the endoscope's line of sight from an endoscopic perspective.

[0035] Image 204 of anatomical scene 206 may include images of anatomical scene 206 captured by imaging modality 202. For example, image 204 may include endoscopic images, ultrasound images, OCT images, REIMS images, and / or any other suitable form of image of anatomical scene 206. Image 204 may include any suitable type of image represented by data in any suitable data format. For example, image 204 may include still frame images, video, color images, infrared images, depth images, hyperspectral images, and / or any other type of image that can visually represent anatomical scene 206. Images captured by imaging modality may include a pixel grid having values ​​(e.g., color values, brightness values, etc.) representing the appearance of anatomical scene 206 captured by imaging modality. The color values ​​of pixels in the captured image may represent the actual organic colors of the anatomical scene captured by imaging modality.

[0036] Additionally or alternatively, image 204 may include one or more models of the anatomical scene 206 generated based on imaging performed by an imaging modality. For example, image 204 may include a three-dimensional (3D) model of the anatomical scene 206 generated based on imaging performed by an imaging modality, such as ultrasound, OCT, REIMS, or other suitable imaging modality. The 3D model may be a full-volume model comprising voxels (i.e., volumetric pixels) having values ​​(e.g., color values, brightness values, etc.) representing the appearance of the anatomical scene 206 at 3D points within the model. Such a volumetric model facilitates system 100 in recognizing and using any slice of the 3D model to generate images of slices of the 3D model. The color values ​​of pixels in the slice image may represent the actual organic colors of the anatomical scene captured by the imaging modality.

[0037] Although Figure 2 Two imaging modalities 202-1 and 202-2 are depicted, respectively capturing images 204-1 and 204-2 provided as input to system 100. However, other examples may include any appropriate number and / or configuration of multiple different imaging modalities that capture images provided as input to system 100 for generating an enhanced image of the anatomical scene 206. For example, three or more different imaging modalities may capture images input to system 100 for generating an enhanced image of the anatomical scene 206.

[0038] System 100 can generate an enhanced image 212 of the anatomical scene 206 based on images 204 captured by one or more imaging modalities 202. System 100 can achieve this in any of the ways described herein to generate an enhanced image comprising a visual image of the anatomical scene 206 captured by an imaging modality (e.g., imaging modality 202-1), which is enhanced with one or more synthetic elements and / or images captured by another imaging modality (e.g., imaging modality 202-2). For example, system 100 can obtain a visual image of the anatomical scene captured by imaging modality 202-1 and enhance that visual image in any of the ways described herein to form an enhanced image. To this end, system 100 can access and use synthetic element data 214 representing one or more synthetic elements to enhance the visual image with one or more synthetic elements. Additionally or alternatively, system 100 can access and use images 204-2 captured by imaging modality 202-2 to utilize images 204-2 to enhance the visual image. This article describes examples of enhanced images and how System 100 can generate enhanced images.

[0039] System 100 may instruct display device 216 to display enhanced image 212. For example, system 100 may provide display device 216 with data representing enhanced image 212, and display device 216 may be configured to display enhanced image 212 for viewing by a user of a computer-assisted medical system. Display device 216 may include any device capable of receiving and processing image data to display one or more images. For this purpose, display device 216 may include one or more displays on which images can be displayed. In some examples, display device 216 may be a component of or communicatively connected to a computer-assisted medical system.

[0040] Now refer to Figures 3A-7B Illustrative examples of enhanced images describing anatomical scenes and methods for generating enhanced images. Figure 3A An anatomical scene 302 imaged by an endoscope 304 is depicted. The anatomical scene 302 can be a real-world physical workspace located in front of the endoscope 304, which is configured to capture endoscopic images of the scene. In some examples, the anatomical scene 302 may include a patient's anatomical structure 306 and one or more medical instruments positioned relative to the anatomical structure 306 within the anatomical scene 302. In the example shown, a grasping tool 308-1 and an imaging probe 308-2 are present in the anatomical scene 302.

[0041] System 100 can acquire images of the anatomical scene 302. For example, system 100 can access visual images of the anatomical scene 302 captured by endoscope 304 and provide such visual images for display by a display device. Figure 3BAn example of a visual image 310 of an anatomical scene 302 captured by an endoscope 304 and displayed by a display device is depicted. The system 100 can enhance the visual image 310 in any of the ways described herein to generate an enhanced image of the anatomical scene 302. An illustrative enhancement of the visual image 310 will now be described. Although these examples are described with reference to the visual image 310, enhancements can be applied to any other image of the anatomical scene.

[0042] Figure 4A An illustrative enhanced image 400-1 is depicted, which can be generated by system 100 and displayed by a display device. Enhanced image 400-1 can be a visual image of an anatomical scene captured by a first imaging modality and enhanced using a synthesis element 402, which is positioned relative to the depiction of the anatomical scene in the visual image. For example, as shown, enhanced image 400-1 can be a visual image 310 of an anatomical scene 302 captured by endoscope 304 and enhanced by synthesis element 402. Synthesis element 402 indicates, within the visual image, a region of the anatomical scene imaged by an imaging probe such as imaging probe 308-2. The imaging probe can capture images using a second imaging modality, and the region of the anatomical scene imaged by the imaging probe can be referred to as the "probe imaging region".

[0043] Synthesis element 402 can enhance the visual image of the anatomical scene captured by the first imaging modality without undesirably interfering with the view of the probe imaging region described in the visual image of the anatomical scene captured by the first imaging modality (e.g., substantially not obscuring the view of the probe imaging region described in the visual image of the anatomical scene captured by the first imaging modality). To this end, synthesis element 402 can be configured to allow at least a portion of the anatomical scene region imaged by the imaging probe to remain visible in the enhanced visual image, as represented by the visual image. Therefore, a medical team member viewing the enhanced image 400-1 can view a representation of the probe imaging region within the anatomical scene within the visual image of the anatomical scene because synthesis element 402 indicates the probe imaging region without substantially interfering with (e.g., obscuring) the presentation of the probe imaging region imaged by the first imaging modality.

[0044] Composite element 402 can be any suitable shape and can include any suitable one or more graphic elements within the enhanced image 400-1 that define the area imaged by the imaging probe at the anatomical scene. For example, composite element 402 can be a circle, ellipse, quadrilateral (e.g., rectangle, sector), triangle, or any other suitable shape. Composite element 402 can include any graphic element that provides an appropriate visual representation of the probe imaging area. Composite element 402 can indicate the boundary of the area imaged by the imaging probe at the anatomical scene. For example, composite element 402 can include boundary elements that delineate the area of ​​the anatomical scene imaged by the imaging probe within the visual image. Boundary elements can be any suitable shape and can be represented by any suitable visual representation (e.g., solid lines, dashed lines, arcs, etc.) arranged to indicate the boundary of the probe imaging area.

[0045] like Figure 4A As shown, the position of the composite element 402 in the enhanced image 400-1 can define two regions of the enhanced image 400-1—a first region 404 outside the composite element 402 and a second region 406 inside the composite element 402. Both the first region 404 and the second region 406 of the enhanced image 400-1 can include a representation of the anatomical scene captured by the first imaging modality (e.g., endoscope 304). The region within the second region 406 can include a depiction of the probe imaging region captured by the first imaging modality. In some examples, the region within the second region 406 may only include a depiction of the probe imaging region captured by the first imaging modality. In other examples, additional content (e.g., content associated with the composite element 402, the probe imaging region, and / or the probe image captured by the imaging probe) can be provided within the first region 404 and / or the second region 406, as will be described herein.

[0046] To generate the enhanced image 400-1, system 100 can determine the probe imaging region of the imaging probe in any suitable manner and place the synthesis element 402 to indicate the probe imaging region within the visual image captured by the first imaging modality. For example, system 100 can determine the probe imaging region based on the pose of the imaging probe relative to the anatomical scene (e.g., the position and orientation of the imaging probe using any suitable degrees of freedom), parameters of the imaging probe, information associated with the anatomical scene (such as depth data), information associated with the first imaging modality (e.g., the viewpoint of endoscope 304), and / or other suitable information about the first imaging modality, the imaging probe, and / or the anatomical scene. System 100 can map the determined probe imaging region onto the visual image of the anatomical scene in any suitable manner, for example, by projecting the probe imaging region from the viewpoint of the first imaging modality to identify a second region 406 within the visual image. System 100 can place the synthesis element 402 within the visual image to visually indicate the second region 406, thereby visually indicating the area imaged by the imaging probe within the visual image.

[0047] In other examples, system 100 can enhance the visual image by positioning the synthetic element 402 relative to other elements in the anatomical scene. For example, system 100 can determine the position of the synthetic element 402 relative to the imaging probe by projecting the synthetic element 402 from the imaging probe in a specific manner (e.g., based on the pose of the imaging probe), such that the synthetic element 402 is positioned relative to the imaging probe, and then projected from the viewpoint of endoscope 304 to form an enhanced image 400-1.

[0048] In some implementations, system 100 may access tracking information (e.g., position information, orientation information, motion information, kinematic information, etc.) of the imaging probe from a computer-aided medical system to which the imaging probe is connected, and use the tracking information to determine the posture of the imaging probe in the anatomical scene. Additionally or alternatively, system 100 may access and use vision-based tracking information (e.g., vision-based tracking information derived from images captured by a first imaging modality and kinematic information for the first imaging modality) to determine the posture of the imaging probe in the anatomical scene. In other examples, system 100 may determine the posture of the imaging probe relative to the anatomical scene in any other suitable manner. System 100 may use the tracking information to determine the area imaged by the imaging probe, for example, by using the tracking information to determine the posture of the imaging probe, and based on the posture of the imaging probe, determine the probe imaging area.

[0049] In some examples, system 100 may be configured to select or generate composite element 402 to fit a determined probe imaging region, and to position composite element 402 relative to a visual image to visually indicate the probe imaging region depicted in the visual image within the visual image. This may include system 100 selecting, shaping, and / or adjusting the size of the composite element to fit the determined probe imaging region.

[0050] Enhanced image 400-1 illustrates an example of a composite element that includes a boundary element of the probe imaging region depicted in an image outlining an anatomical scene. As shown, the composite element can allow at least a portion of the probe imaging region (e.g., region 406) depicted in the visual image captured by the first imaging modality to remain visible in enhanced image 400-1 (e.g., by substantially not visually obscuring the probe imaging region). In other examples, additional or alternative composite elements may be used to enhance the visual image. For example, another version of composite element 402 may include additional content presented within the boundary element.

[0051] Figure 4B An illustrative enhanced image 400-2 is depicted, which is similar to enhanced image 400-1 and also includes a spatial marker 408 positioned relative to the probe imaging area. As shown, the composite element 402 may include a boundary element (e.g., as shown by dashed lines) that delineates the boundary of the area of ​​the anatomical scene imaged by the imaging probe 308-2 within the visual image, and a spatial marker 408 located within the boundary element and therefore within the area of ​​the anatomical scene imaged by the imaging probe 308-2. The spatial marker 408 may include any supplementary composite features within the composite element 402. The spatial marker 408 may be configured to visually indicate the area or space imaged by the probe and / or any information about or related to the area or space imaged by the probe. For example, in some examples, the spatial marker 408 may include any composite marker added to the visual image of the anatomical scene and visually indicating information about the probe imaging area, such as the location of the area or space imaged by the probe, the orientation of the probe imaging area, the depth of the image, and / or the location within the probe imaging area. The presentation of spatial markers 408 relative to the probe imaging area, as depicted in the visual image captured by the first imaging modality, can help the user identify the area or space imaged by the probe and / or information about that area, such as specific locations and / or specific anatomical features within the probe imaging area, as well as the spatial relationship between spatial markers 408 and anatomical features.

[0052] exist Figure 4BIn the example shown, spatial marker 408 includes a grid of horizontal and vertical lines. The positioning of the grid lines relative to the probe imaging area indicates the specific location of an anatomical feature within the probe imaging area. In other examples, additional or alternative spatial markers may be used. For example, spatial markers may include point markers (e.g., a grid of points), depth markers (e.g., depth lines of concentric arcs), patterns of markers, shading markers, and / or any other suitable composite features within composite element 402, such as composite markers indicating location and / or distance within the probe imaging area.

[0053] Figure 4C An illustrative enhanced image 400-3 is depicted, similar to enhanced image 400-1 and including a highlight element 410 and a depth line 412 located within the boundary elements of composite element 402. Highlight element 410 may include any graphic element that highlights or otherwise visually identifies selected portions of the probe imaging area. As shown, highlight element 410 may include generally elliptical spatial markers positioned relative to the probe imaging area. Highlight element 410 may include any other markers or sets of markers configured to highlight selected portions of the probe imaging area within composite element 410. Highlight element 410 may be represented by different shapes, different lines, different color areas, other suitable representations, and / or different combinations of visual representations. Depth line 412 is depicted as concentric arcs, and another example of spatial markers is shown. Depth line 412 can provide positional context for highlight element 410 within the boundary elements of composite element 402. Additionally or alternatively, other suitable spatial markers may be provided within the boundary elements of composite element 402 along with highlight element 410.

[0054] System 100 can place highlight element 410 based on any appropriate information about the anatomical scene. For example, system 100 can be configured to identify specific features of the anatomical scene depicted in a visual image and place highlight element 410 relative to those features to visually identify them. Additionally or alternatively, system 100 can place highlight element 410 relative to a visual image based on user input, such as user input configured to move highlight element 410 relative to the visual image.

[0055] Figure 4DAn illustrative enhanced image 400-4 is depicted, similar to enhanced image 400-1 and also including a selected portion 414 of the probe image located within the boundary elements of the composition element 402. The selected portion 414 of the probe image can be a specific part of the probe image captured by the imaging probe. System 100 can select the selected portion 414 for enhancing the visual image based on any suitable information about the anatomical scene. For example, system 100 can be configured to identify a specific feature of the anatomical scene depicted in the probe image and select a portion of the probe image depicting that feature to place within the boundary elements of the composition element 402. System 100 can position the selected portion 414 of the probe image to represent the correlation between the feature and its location in the probe imaging region. Additionally or alternatively, system 100 can place, select, and position the selected portion 414 of the probe image based on user input, such as user input configured to move to indicate a desired selected portion of the probe image.

[0056] In some examples, the imaging probe may be an ultrasound probe, and a selected portion 414 of the probe image may be a Doppler image portion of the probe image. The Doppler image portion may advantageously display the location of blood flow relative to a visual indication of the probe imaging area, while ensuring that at least a portion of the probe imaging area captured by the first imaging modality and depicted in the visual image is unobstructed and visible to the user.

[0057] System 100 can combine the selected portion 414 with a depiction of the probe imaging region captured by the first imaging modality in any suitable manner. For example, a representation of the selected portion 414 of the probe image can be overlaid on an image of the anatomical scene captured by the first imaging modality in a partially transparent manner. Various display features (e.g., color schemes) can be used to display the selected portion 414 within the boundary elements of the composition element 402. Although not shown, suitable spatial markers (e.g., grid lines) can also be presented within the boundary elements of the composition element 402 along with the selected portion 414 of the probe image.

[0058] Figures 4A-4D Examples of enhancements visually indicating the probe imaging area depicted in captured images of an anatomical scene are illustrated. Some enhancements also indicate additional information about and / or related to the probe imaging area. These examples are illustrative. In other examples, additional or alternative enhancements indicating the probe imaging area and / or information about the probe imaging area may be used.

[0059] System 100 can be configured to further enhance a visual image of an anatomical scene using a probe image captured by an imaging probe. The probe image may include at least a portion of a probe image, and system 100 can position the probe image so as not to overlap with a probe imaging region depicted in the visual image. In some examples, system 100 can be configured to present a visual image enhanced solely by the probe image without additional composite elements such as composite element 402. In other examples, system 100 can be configured to enhance the visual image using a composite element (e.g., composite element 402) indicating the probe imaging region and the probe image captured by the imaging probe. In an enhanced image that includes both the composite element indicating the probe imaging region and the probe image, the probe image can be positioned so as not to overlap with the depiction of the composite element or the probe imaging region. By positioning the probe image in the enhanced image, the probe image can be prevented from overlapping with the description of the composite element or the probe imaging region, such that no portion of the probe image is located within the probe imaging region or the region visually indicated by the composite element.

[0060] Figure 5A An illustrative enhanced image 500-1 is depicted, which is similar to enhanced image 400-1 and also includes a probe image 502 within a first region 404. As shown, the probe image 502 is located outside of the composite element 402. For example, the probe image 502 is placed within enhanced image 500-1 at a location that does not overlap with the depiction of the area of ​​the anatomical scene imaged by the imaging probe (i.e., the probe imaging area represented by the second region 406 indicated by composite element 402). For example, the probe image 502 may be presented in a corner region of enhanced image 500-1 to avoid obscuring the view of the probe imaging area indicated by composite element 402. In other examples, the probe image 502 may be placed in any other suitable location, including within, outside, or partially within and partially outside the visual image captured by the first imaging modality. In some examples, the position of the probe image 502 relative to the visual image captured by the first imaging modality can be selected based on user input suitable for selecting the position of the probe image 502 within the augmented image 500-1.

[0061] exist Figure 5AIn the composite element 402, the representation of the anatomical scene is shown using a dot pattern representing the anatomical scene captured by a first imaging modality (e.g., endoscopic imaging), and the representation of the probe image 502 is shown using a diagonal line pattern representing the probe image captured by a second imaging modality (e.g., ultrasound imaging) different from the first imaging modality. Medical team members viewing the enhanced image 500-1 can simultaneously view visualizations of portions of the anatomical scene captured by different imaging modalities. Because the representation of the probe image 502 does not overlap with the composite element 402 or the second region 406, medical team members can view the anatomical scene captured by different imaging modalities while having a substantially unobstructed view of the probe imaging area of ​​the anatomical scene captured by the first imaging modality within the second region 406, where clarity may be required for medical procedures.

[0062] While in some examples, probe image 502 may include an ultrasound image, any other suitable image captured by a different imaging modality can be used to enhance the visual image captured by the first imaging modality. For example, system 100 may determine slices of a 3D model (e.g., a 3D model generated from OCT or REIMS) based on the position of synthetic element 402 relative to the anatomical scene and use images of those slices to enhance the visual image captured by the first imaging modality. Thus, in enhanced image 500-1, probe image 502 can be replaced with an image captured by any suitable second imaging modality. In some examples, the image used to enhance the visual image can be selected by input from a user of the computer-assisted medical system. For example, a user of the computer-assisted medical system can provide input to switch from an imaging modality image projected onto a corner area to another imaging modality image projected onto a corner area (e.g., from an ultrasound image to an OCT or REIMS model image, and vice versa).

[0063] Enhanced image 500-1 illustrates an example of enhancement of a visual image of an anatomical scene captured by a first imaging modality and an image of an anatomical scene captured by a second imaging modality. Other examples may include additional or alternative enhancement content. For example, system 100 may further enhance the visual image using content associated with the image captured by the second imaging modality, such as by presenting synthetic elements associated with probe image 502.

[0064] Figure 5BAn illustrative enhanced image 500-2 is depicted, which is similar to enhanced image 500-1 and also includes a spatial marker 408 located within the boundary element of the composite element 402 and a spatial marker 504 located within the boundary of the probe image 502. The spatial marker 504 can be any composite marker that visually indicates information about a specific location within the probe image 502. For example, the spatial marker 504 may include a composite element positioned to indicate a specific location within the probe image 502. By presenting the spatial marker 504 relative to the probe image captured by the imaging probe, the spatial marker 504 can help the user identify specific locations within the probe image 502, such as specific anatomical features (e.g., subsurface anatomical features captured by the ultrasound probe) and the spatial relationship between the spatial marker 504 and the anatomical features.

[0065] Spatial markers 504 within probe image 502 can correspond to spatial markers 408 within composite element 402 and can aid in viewing the separately presented images of the anatomical scene in the mind of a medical team member in enhanced image 500-2 (e.g., an image of the probe imaging region captured by the first imaging modality and a probe image captured by the second imaging modality). For this purpose, specific spatial markers 408 (intersections of specific grid lines) can indicate specific locations within the probe imaging region, as depicted in the visual image captured by the first imaging modality, and corresponding spatial markers 504 (corresponding intersections of specific grid lines) can indicate specific locations within the probe image captured by the imaging probe, thus indicating the relationship between specific locations within the probe imaging region and specific locations within the probe image. Spatial markers 504 can be any suitable marker referencing locations and / or distances within the probe image.

[0066] Figure 5CAn illustrative enhanced image 500-3 is depicted, which is similar to enhanced image 500-1 and also includes a highlight element 410 and a depth line 412 located within composite element 402, and corresponding highlight elements 506 and depth lines 508 located within probe image 502. Highlight element 410 and depth line 412 have been described above. Highlight element 506 may include any graphic element that highlights or otherwise visually identifies selected portions of probe image 502. As shown, highlight element 506 may include generally elliptical spatial markers positioned relative to probe image 502. Highlight element 506 may include any other markers or sets of markers configured to highlight selected portions of probe image 502. Highlight element 506 may be represented by different shapes, different lines, different color areas, other suitable representations, and / or different combinations of visual representations. Depth line 508 is depicted as concentric arcs, and another example of spatial markers is shown. Depth line 508 can provide the positional context of highlight element 506. Additionally or alternatively, other suitable spatial markers may be provided within the probe image 502 along with the highlight element 506. By presenting the highlight element 506 and / or depth line 508 relative to the probe image captured by the imaging probe, the highlight element 506 and / or depth line 508 can help the user identify specific locations within the probe image 502, such as specific anatomical features (e.g., subsurface anatomical features captured by the ultrasound probe) and the spatial relationship between the highlight element 506 or depth line 508 and the anatomical features.

[0067] System 100 can place highlight element 506 based on any appropriate information about the probe image. For example, system 100 can be configured to identify a specific feature depicted in the probe image and place highlight element 506 relative to that feature to visually identify it. Additionally or alternatively, system 100 can place highlight element 506 relative to the visual image based on user input, such as user input configured to move highlight element 506 relative to the visual image.

[0068] Highlight element 506 within probe image 502 can correspond to highlight element 410 within composite element 402, and can aid in viewing the separate presentation of the anatomical scene in the mind of the medical team members in enhanced image 500-3 (e.g., an image of the probe imaging region captured by the first imaging modality and a probe image captured by the second imaging modality). For this purpose, highlight element 410 can indicate a specific location within the probe imaging region depicted in the visual image captured by the first imaging modality, and the corresponding highlight element 506 can indicate a specific location within probe image 502 captured by the imaging probe, thus indicating the relationship between a specific location within the probe imaging region and a specific location within the probe image. Similarly, depth line 508 within probe image 502 can correspond to depth line 412 within composite element 402, and can aid in viewing the separate presentation of the anatomical scene in the mind of the medical team members in enhanced image 500-3.

[0069] Figure 5D An illustrative enhanced image 500-4 is depicted, which is similar to enhanced image 500-1 and also includes a selected portion 510 of probe image 502 represented as a selected portion 414 within composition element 402. In some examples, the selected portion 510 of probe image 502 may be a Doppler image portion of probe image 502. The selected portion 510 may be combined with the depiction of probe image 502. Various color schemes may be suitable for displaying the selected portion 510 within probe image 502. As shown, the representation of the selected portion 510 of probe image 502 may also be presented as a selected portion 414 within composition element 402 to provide visual correlation between probe image 502 and the probe imaging region depicted in the visual image captured by the first imaging modality. For this purpose, selected portion 414 may indicate a specific location within the probe imaging region depicted in the visual image captured by the first imaging modality, and corresponding selected portion 510 may indicate a specific location within the probe image 502 captured by the imaging probe, to indicate the relationship between a specific location within the probe imaging region and a specific location within the probe image. Although not shown, suitable spatial markings may also be provided along with selected portion 510 of probe image 502. Related spatial markings may also be provided along with selected portion 414 displayed within composite element 402. In some examples, probe image 502 may be a fully regular probe image captured by the imaging probe, and selected portion 414 may be a Doppler image captured by the imaging probe.

[0070] Although Figure 5DIn this example, the Doppler image portion of the probe image is shown as a selected portion 414 of the probe image; however, in other examples, additional or alternative selected portions of the probe image may be selected and displayed within the synthesis element 402. For example, system 100 may be configured to identify anatomical features (e.g., tumors) depicted in the probe image based on the probe image, and select a portion of the probe image representing that anatomical feature to display within the synthesis element 402. System 100 may be configured to identify anatomical features depicted in the probe image in any suitable manner.

[0071] Other examples of enhanced images of anatomical scenes may include visual images of the anatomical scene enhanced with preview placeholders that indicate the display location of the probe image to be captured by the imaging probe. References will now be made to... Figures 6A-6B and Figures 7A-7B Here is an example to describe this type of image enhancement.

[0072] Figure 6A The diagram shows Figure 3A Enhanced image 600-1 of the anatomical scene. As shown, the visual image of the anatomical scene can be enhanced using composite element 402 and preview placeholder 602 indicating the display location of probe images that have not yet been captured. Figure 6A As shown in enhanced image 600-1, a preview placeholder 602 is placed above a portion of anatomical structure 306. The presentation of the preview placeholder 602 indicates to medical team members that if a probe image were displayed at the same location where the preview placeholder 602 is displayed, that portion of anatomical structure 306 would be obscured by the probe image. Advantageously, based on this indication, medical team members can make adjustments to prevent that portion of anatomical structure 306 from being obscured by the probe image. For example, the image rendering viewpoint of the visual image can be adjusted (e.g., by repositioning a first imaging modality instrument such as an endoscope) so that anatomical structure 306 is repositioned within the visual image in a manner that moves that portion of anatomical structure 306 away from the preview placeholder 602 to form an arrangement more desired by the medical team members. Figure 6B Enhanced image 600-2 is illustrated. Enhanced image 600-2 is similar to enhanced image 600-2, but in which anatomical structure 306 is shifted within the image frame to remove that part of anatomical structure 306 from preview placeholder 602, so that the probe image displayed at the same location in preview placeholder 602 does not obscure that part of anatomical structure 306.

[0073] In enhanced images 600-1 and 600-2, the preview placeholder 602 is displayed along with the composition element 402. In other examples, the preview placeholder may be displayed without the composition element 402 or any other enhancements. For example, Figure 7A and Figure 7BThe illustration shows enhanced images 700-1 and 700-2, which are similar to enhanced images 600-1 and 600-2, but do not include the composite element 402.

[0074] System 100 can be configured to enhance the visual image of the anatomical scene in any manner described herein at any suitable time and / or in response to any suitable event. For example, System 100 can enhance the visual image in response to the occurrence of a predefined event.

[0075] In some examples, a predefined event may include an imaging probe being stationary for a threshold time period. In such an example, system 100 may be configured to detect that the imaging probe is stationary for a threshold time period and, in response, enhance the visual image in any manner described herein.

[0076] A stationary state can be defined in any suitable manner. For example, a stationary state may include a state in which the imaging probe is held in an orientation that changes no more than a defined rate or tolerance. Thus, a stationary state can be defined as including movement no more than the permissible rate or tolerance, and excluding movement exceeding the permissible rate or tolerance. The permissible rate or tolerance rate can be defined based on the capabilities of the imaging probe and / or the capabilities of the system used to process the images captured by the imaging probe.

[0077] Enhancement can be controlled by allowing it to be performed only when the imaging probe is stationary, ensuring that enhancement is provided only when meaningful or enhanced data is available. To illustrate, when the imaging probe moves within the anatomical scene at a rate exceeding a threshold, the system's capability may not be fast enough to process probe images to provide meaningful or enhanced probe images for applications such as... Figures 5A-5D The enhanced version of probe image 502 is shown. Therefore, system 100 can be configured to avoid using the probe image to enhance the visual image when the imaging probe moves at a rate exceeding a threshold. System 100 may not provide enhancement or may only provide different enhancements, such as composite element 402 and / or preview placeholders 602 or 702, instead of providing the probe image as enhancement. In response to the probe image being stationary, system 100 can provide enhancements such as preview placeholders 602 or 702, probe image 502, composite element 402, and / or any other enhancements. This can allow surgical team members to bring the imaging probe to a stationary state to trigger the enhancement of the visual image, which can help surgical team members determine and / or perform the next step of the medical procedure. In some examples, the stationary state of the imaging probe can allow for enhanced probe imaging and / or more accurate detection of the imaging probe's posture.

[0078] In some examples, predefined events may include the imaging probe contacting tissue at the anatomical scene. Tissue contact of the imaging probe can be detected in any suitable manner. System 100 can be configured to detect imaging probe contact with tissue and, in response, enhance the visual image of the anatomical scene in any manner described herein.

[0079] In some examples, the predefined event may include receiving predefined user input provided by a member of the surgical team, such as user input provided to a computer-assisted medical system. System 100 may be configured to detect the reception of the predefined user input and, in response, enhance the visual image of the anatomical scene in any of the ways described herein. In some examples, a specific user input may be mapped to a corresponding enhancement, such that the reception of the specific user input triggers a specific enhancement. For example, system 100 may detect the reception of a specific user input and, in response, may enhance the visual image of the anatomical scene using selected portions 414 and / or 510 of a probe image captured by an imaging probe at the anatomical scene.

[0080] In some examples, predefined events may include determination by system 100 that a medical team member intends to interact with or use the imaging probe. System 100 may be configured to determine that a medical team member intends to interact with or use the imaging probe in any appropriate manner and based on any information regarding events concerning the imaging probe and / or computer-assisted medical systems and / or medical procedures. For example, system 100 may detect, based on such information, the intention of a surgical team member to grasp or otherwise engage an insertable imaging probe using a medical device (e.g., by detecting movement of the medical device toward the imaging probe) or the actual grasping or other engagement of the imaging probe with the medical device. In response, system 100 may provide information about the imaging probe. For example, system 100 may enhance the visual image of the anatomical scene in any of the manner described herein. For example, system 100 may enhance the visual image with a preview placeholder indicating the display position of the probe image to be captured by the imaging probe.

[0081] As described above, system 100 can be implemented in or communicatively coupled to a computer-assisted medical system. System 100 can receive input from the computer-assisted medical system and provide output thereto. For example, system 100 can access images of an anatomical scene and / or any information about the anatomical scene and / or the computer-assisted medical system from the computer-assisted medical system, use the accessed images and / or information to perform any of the processes described herein to generate an enhanced image of the anatomical scene, and provide data representing the enhanced image to the computer-assisted medical system for display (e.g., via a display device associated with the computer-assisted medical system).

[0082] Figure 8An illustrative computer-aided medical system 800 (“medical system 800”) is depicted. System 100 may be implemented by medical system 800, connected to medical system 800 and / or otherwise used in conjunction with medical system 800.

[0083] As shown in the figure, the medical system 800 may include an operating system 802, a user control system 804, and an auxiliary system 806 that are communicatively coupled. The medical system 800 can be used by a medical team to perform computer-aided medical procedures on a patient 808. As shown, the medical team may include medical team member 810-1, assistant 810-2, nurse 810-3, and anesthesiologist 810-4, all of whom can be collectively referred to as "medical team member 810". Additional or alternative medical team members may appear during a medical session that may serve a particular implementation.

[0084] Although Figure 8 A minimally invasive medical procedure in progress is depicted; however, it is understood that the medical system 800 can be similarly used to perform open medical procedures or other types of medical procedures that can similarly benefit from the accuracy and convenience of the medical system 800. Furthermore, it should be understood that the entire medical session using the medical system 800 can include not only procedures such as… Figure 8 The procedures shown may include the operational phases of a medical procedure, and may also include pre-operative, post-operative, and / or other appropriate phases of the medical procedure.

[0085] like Figure 8 As shown, the manipulation system 802 may include a plurality of manipulator arms 812 (e.g., manipulator arms 812-1 to 812-4), and a plurality of medical devices may be coupled to the plurality of manipulator arms 812. Each medical device may be implemented by any suitable medical tool (e.g., a tool with tissue interaction capabilities), surgical tool, imaging device (e.g., an endoscope, an ultrasound tool, etc.), sensing device (e.g., a force-sensing medical device), diagnostic device, etc., which may be used for computer-assisted medical procedures on patient 808 (e.g., by being at least partially inserted into patient 808 and manipulated to perform computer-assisted medical procedures on patient 808). Although the manipulation system 802 is depicted and described herein as including four manipulator arms 812, it will be appreciated that the manipulation system 802 may include only a single manipulator arm 812 or any other number of manipulator arms that may serve a particular embodiment.

[0086] The manipulator arm 812 and / or the medical device attached to the manipulator arm 812 may include one or more displacement transducers, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the medical system 800 may be configured to use kinematic information to track (e.g., determine the position of the medical device) and / or control the medical device. The system 100 may be configured to access and use kinematic information for one or more augmented operations, such as tracking the posture of an imaging probe (e.g., by tracking the posture of a medical device grasping or otherwise engaging an imaging probe at an anatomical scene).

[0087] User control system 804 can be configured to facilitate control of manipulator arm 812 and medical instruments attached to manipulator arm 812 by medical team member 810-1. For example, medical team member 810-1 can interact with user control system 804 to remotely move or manipulate manipulator arm 812 and medical instruments. To this end, user control system 804 can provide medical team member 810-1 with images (e.g., high-resolution 3D images) of the anatomical scene associated with patient 808 captured by an imaging system (e.g., any medical imaging system described herein). In some examples, user control system 804 may include a stereoscopic viewer with two displays, where medical team member 810-1 can view stereoscopic images of the anatomical scene associated with patient 808 and generated by a stereoscopic imaging system. In some examples, enhanced images generated by system 100 can be displayed by user control system 804. Medical team member 810-1 can use the images displayed by user control system 804 to perform one or more procedures using one or more medical instruments attached to manipulator arm 812.

[0088] To facilitate control of the medical devices, the user control system 804 may include a set of master controls. Medical team member 810-1 can manipulate the master controls to control the movement of the medical devices (e.g., by utilizing robotics and / or remote operation technology). The master controls can be configured to detect various hand, wrist, and finger movements of medical team member 810-1. In this way, medical team member 810-1 can intuitively execute procedures using one or more medical devices.

[0089] The auxiliary system 806 may include one or more computing devices configured to perform the main processing operations of the medical system 800. In this configuration, the one or more computing devices included in the auxiliary system 806 may control and / or coordinate operations performed by various other components of the medical system 800, such as the manipulation system 802 and the user control system 804. For example, the computing device included in the user control system 804 may send instructions to the manipulation system 802 via one or more computing devices included in the auxiliary system 806. As another example, the auxiliary system 806 may receive and process image data representing images captured by an imaging device attached to one of the manipulator arms 812 from the manipulation system 802.

[0090] In some examples, the assistive system 806 may be configured to present visual content to a medical team member 810 who may not have access to images provided to the medical team member 810-1 at the user control system 804. To this end, the assistive system 806 may include a display monitor 814 configured to display one or more user interfaces, such as images of the anatomical scene (e.g., 2D images), information associated with the patient 808 and / or medical procedures, and / or any other visual content that may serve a particular implementation. For example, the display monitor 814 may display images of the anatomical scene along with additional content displayed concurrently with the images (e.g., graphical content, contextual information, etc.). The system 100 may be configured to provide the display monitor 814 with enhanced images generated by the system 100 for display. In some embodiments, the display monitor 814 is implemented as a touchscreen display to which the medical team member 810 may interact (e.g., via touch gestures) to provide user input to the medical system 800.

[0091] The operating system 802, the user control system 804, and the auxiliary system 806 can be communicatively coupled to each other in any suitable manner. For example, such as Figure 8 As shown, the operating system 802, the user control system 804, and the auxiliary system 806 can be communicatively coupled via a control line 816, which can represent any wired or wireless communication link that can serve a particular implementation. Therefore, the operating system 802, the user control system 804, and the auxiliary system 806 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0092] Computer-aided medical system 800 illustrates one example of a computer-aided medical system. System 100 may be part of a computer-aided medical system and / or system 100 may interact with a computer-aided medical system to provide enhanced images of the anatomical scene described herein. System 100 may be implemented, communicate with, and / or operate in conjunction with any other suitable medical system (e.g., a surgical system, a robotic system, etc.).

[0093] Figure 9 Illustrative method 900 is shown. Although Figure 9 Illustrative operations according to one embodiment are described, but other embodiments may omit, add, reorder, combine and / or modify them. Figure 9 Any of the steps shown. Figure 9 One or more of the operations shown can be performed by a computing system such as system 100, any of its components and / or any implementation thereof.

[0094] In operation 902, the computing system acquires a visual image of the anatomical scene, including the imaging probe. The visual image may be captured by a first imaging modality (e.g., an endoscope), and the computing system may acquire the visual image from the first imaging modality or any other suitable source. Operation 902 may be performed in any of the manner described herein.

[0095] In operation 904, the computing system enhances the visual image with a composite element that indicates a region of the anatomical scene imaged by the imaging probe within the visual image. The composite element may include one or more illustrative composite elements described herein, such as composite element 402 and / or preview placeholders 602 or 702. For example, in some embodiments, the composite element may indicate the boundary of a region of the anatomical scene imaged by the imaging probe and may include spatial markers such as spatial markers 408 and / or 504, highlight elements 410 and / or 506, depth lines 412 and / or 508, or any suitable combination or sub-combination thereof within the boundary and thus within the region of the anatomical scene imaged by the imaging probe. In some examples, the composite element may indicate a region of the anatomical scene imaged by the imaging probe without explicitly indicating the boundary of that region, and include spatial markers within that region that may implicitly indicate the region. For example, the composite element may include speckle as a spatial element within a region imaged without hard boundary lines. Operation 904 may be performed in any of the manner described herein.

[0096] In operation 906, the computing system instructs the display device to display an enhanced image. To this end, the computing system may provide the display device with data representing the enhanced image for processing and display by the display device. Operation 906 may be performed in any of the manner described herein.

[0097] Figure 10 Illustrative method 1000 is shown. Although Figure 10 Illustrative operations according to one embodiment are described, but other embodiments may omit, add, reorder, combine and / or modify them. Figure 10 Any of the steps shown. Figure 10 One or more of the operations shown can be performed by a computing system such as system 100, any of its components and / or any implementation thereof.

[0098] In operation 1002, the computing system acquires a visual image of the anatomical scene, including an imaging probe. The visual image may be captured by a first imaging modality (e.g., an endoscope), and the computing system may acquire the visual image from the first imaging modality or any other suitable source. Operation 1002 may be performed in any of the manner described herein.

[0099] In operation 1004, the computing system obtains a probe image captured by the imaging probe. The imaging probe may be or provide a second imaging modality, and the computing system may obtain the probe image from the second imaging modality or any other suitable source. Operation 1004 may be performed in any of the manner described herein.

[0100] In operation 1006, the computing system uses synthetic elements to enhance the visual image, which indicate areas of the anatomical scene imaged by the imaging probe within the visual image. Operation 1006 can be performed in any of the manner described herein.

[0101] In operation 1008, the computing system instructs the display device to display the enhanced image and the probe image, such that the probe image is located outside the composite element, and that the probe image does not overlap with the depiction of the region of the anatomical scene imaged by the imaging probe in the visual image. For example, the computing system can provide images for display, such as... Figures 5A-5D Any image shown. To this end, the computing system can provide data representing the image to the display device so that the display device can process and display the image. Operation 1008 can be performed in any of the manner described herein.

[0102] Figure 11 Illustrative method 1100 is shown. Although Figure 11 Illustrative operations according to one embodiment are described, but other embodiments may omit, add, reorder, combine and / or modify them. Figure 11 Any of the steps shown. Figure 11 One or more of the operations shown can be performed by a computing system such as system 100, any of its components and / or any implementation thereof.

[0103] In operation 1102, the computing system acquires a visual image of the anatomical scene, including the imaging probe. The visual image may be captured by a first imaging modality (e.g., an endoscope), and the computing system may acquire the visual image from the first imaging modality or any other suitable source. Operation 1102 may be performed in any of the manner described herein.

[0104] In operation 1104, the computing system enhances the visual image using a preview placeholder that indicates the display position of the probe image to be captured by the imaging probe. For example, the computing system can enhance the visual image to form an enhanced image, such as... Figures 6A-7B Any enhanced image shown. Operation 1104 can be performed in any of the ways described herein.

[0105] In operation 1106, the computing system instructs the display device to display an enhanced image. To this end, the computing system may provide the display device with data representing the enhanced image for processing and display by the display device. Operation 1106 may be performed in any of the manner described herein.

[0106] In some examples, a non-transitory computer-readable medium for storing computer-readable instructions may be provided, based on the principles described herein. When executed by a processor of a computing device, the instructions may instruct the processor and / or computing device to perform one or more operations, including one or more of the operations described herein. Such instructions may be stored and / or transmitted using any of a variety of known computer-readable media.

[0107] The term "non-transitory computer-readable medium" as used herein can include any non-transitory storage medium that contributes to providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of the computing device). For example, non-transitory computer-readable media can include, but is not limited to, any combination of non-volatile storage media and / or volatile storage media. Illustrative non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory (RAM), and optical discs (e.g., optical discs, digital video discs, Blu-ray discs, etc.). Illustrative volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0108] Figure 12 An illustrative computing device 1200 is depicted that can be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by computing device 1200.

[0109] like Figure 12As shown, computing device 1200 may include a communication interface 1202, a processor 1204, a storage device 1206, and an input / output (I / O) module 1208 that are communicatively connected to each other via communication infrastructure 1210. Although Figure 12 The illustration shows a computing device 1200, but Figure 12 The components shown are not intended to be limiting. Additional or alternative components may be used in other embodiments. A more detailed description will now follow. Figure 12 Components of the computing device 1200 shown.

[0110] Communication interface 1202 can be configured to communicate with one or more computing devices. Examples of communication interface 1202 include, but are not limited to, wired network interfaces (such as network interface cards), wireless network interfaces (such as wireless network interface cards), modems, audio / video connections, and any other suitable interfaces.

[0111] Processor 1204 generally refers to any type or form of processing unit capable of processing data and / or interpreting, executing, and / or instructing the execution of one or more of the instructions, procedures, and / or operations described herein. Processor 1204 may perform operations by executing computer-executable instructions 1212 (e.g., applications, software, code, and / or other executable data instances) stored in storage device 1206.

[0112] Storage device 1206 may include one or more data storage media, devices, or configurations, and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1206 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data including the data described herein may be temporarily and / or permanently stored in storage device 1206. For example, data representing computer-executable instructions 1212 configured to instruct processor 1204 to perform any of the operations described herein may be stored within storage device 1206. In some examples, data may be arranged in one or more databases residing within storage device 1206.

[0113] I / O module 1208 may include one or more I / O modules configured to receive user input and provide user output. I / O module 1208 may include any hardware, firmware, software, or a combination thereof that supports input and output capabilities. For example, I / O module 1208 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touchscreen component (e.g., a touchscreen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0114] I / O module 1208 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a screen), one or more output drivers (e.g., display drivers), one or more audio speakers, and one or more audio drivers. In some embodiments, I / O module 1208 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may serve a particular implementation.

[0115] In some examples, any systems and / or facilities described herein may be implemented by or within one or more components of computing device 1200. For example, one or more applications 1212 residing in storage device 1206 may be configured to instruct the implementation of processor 1204 to perform one or more operations or functions associated with processing facility 104 of system 100.

[0116] One or more of the operations described herein can be performed or executed in real time. As used herein, operations described as occurring "in real time" will be understood as being executed immediately without inappropriate delay, even though absolute zero delay is not possible.

[0117] Any system, device, and / or component thereof may be implemented in any suitable combination or sub-combination. For example, any system, device, and / or component thereof may be implemented as an apparatus configured to perform one or more of the operations described herein.

[0118] In the foregoing description, various illustrative embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes can be made thereto, and additional embodiments can be implemented without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined or substituted with features of another embodiment described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A visualization system, comprising: Memory that stores instructions; as well as A processor, communicatively coupled to the memory and configured to execute the instructions to: Obtain endoscopic images of an anatomical scene within the body, the endoscopic images being captured by an endoscope and depicting an imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; The endoscopic image is enhanced with a composite element that indicates a region of the anatomical scene imaged by the imaging probe within the endoscopic image, the composite element including spatial markers within the region; as well as The display device shows the endoscopic image enhanced with the synthetic elements.

2. The system as claimed in claim 1, wherein: The composite elements include boundary elements that delineate the region of the anatomical scene imaged by the imaging probe within the endoscopic image; and The spatial marker is positioned within the boundary element.

3. The system of claim 1, wherein the processor executes the instructions to: Obtain the probe image captured by the imaging probe; and The endoscopic image is further enhanced by utilizing a selected portion of the probe image located within the region of the anatomical scene imaged by the imaging probe.

4. The system of claim 1, wherein the processor executes the instructions to further enhance the endoscopic image by utilizing highlight elements positioned to highlight selected portions of the region of the anatomical scene imaged by the imaging probe.

5. The system of claim 1, wherein the processor executes the instructions to: The time duration for which the imaging probe remains stationary is detected, up to a threshold value; and In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the endoscopic image is enhanced with the synthetic elements.

6. The system of claim 1, wherein the processor executes the instructions to enhance the endoscopic image using the synthetic elements by performing operations including the following steps: Determine the orientation of the imaging probe in the anatomical scene; Based on the posture of the imaging probe, the region of the anatomical scene imaged by the imaging probe is determined; as well as The synthetic elements are used to enhance the endoscopic image to visually indicate the region depicted in the endoscopic image.

7. The system of claim 1, wherein the processor executes the instructions to: Obtain the probe image captured by the imaging probe; and The display device is instructed to display the probe image outside the composite element.

8. The system of claim 7, wherein the processor executes the instructions to enhance the probe image using an additional set of spatial markers corresponding to the spatial markers of the composite element.

9. The system of claim 7, wherein the processor executes the instructions to: The time duration for which the imaging probe remains stationary is detected, up to a threshold value; and In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the display device is instructed to display the probe image.

10. The system of claim 1, wherein the processor executes the instructions to further enhance the endoscopic image using a preview placeholder, the preview placeholder indicating the display position of a probe image to be captured by the imaging probe.

11. A visualization system, comprising: Memory that stores instructions; as well as A processor, communicatively coupled to the memory and configured to execute the instructions to: Obtain endoscopic images of anatomical scenes within the body, the endoscopic images being captured by an endoscope and depicted by an imaging probe; Obtain probe images captured by the imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; The endoscopic image is enhanced with a synthetic element, which indicates the region of the anatomical scene imaged by the imaging probe within the endoscopic image; as well as The display device shows the endoscope image and the probe image, the endoscope image being enhanced by the composite element, and the probe image being positioned outside the composite element.

12. The system of claim 11, wherein the synthetic element comprises: Boundary elements that delineate the region of the anatomical scene imaged by the imaging probe within the endoscopic image; as well as Spatial markers within the boundary elements.

13. The system of claim 11, wherein the processor executes the instructions to further enhance the endoscopic image using a selected portion of the probe image located within the composite element.

14. The system of claim 11, wherein the processor executes the instructions to further enhance the endoscopic image using highlight elements positioned within the composite elements to highlight selected portions of the region of the anatomical scene.

15. The system of claim 11, wherein the processor executes the instructions to: The time duration for which the imaging probe remains stationary is detected, up to a threshold value; and In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the endoscopic image is enhanced with the synthetic elements.

16. The system of claim 11, wherein the processor executes the instructions to enhance the endoscopic image using the synthetic elements by performing operations including the following steps: Determine the orientation of the imaging probe in the anatomical scene; Based on the posture of the imaging probe, the region of the anatomical scene imaged by the imaging probe is determined; as well as The synthetic elements are used to enhance the endoscopic image to visually indicate the region depicted in the endoscopic image.

17. The system of claim 11, wherein the probe image is located in the corner region of the endoscopic image.

18. The system of claim 11, wherein: The synthetic elements include a first set of spatial markers; and The probe image is enhanced using a second set of spatial markers corresponding to the first set of spatial markers of the synthetic element.

19. A visualization method, comprising: Obtain endoscopic images of an anatomical scene within the body, the endoscopic images being captured by an endoscope and depicting an imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; The endoscopic image is enhanced with a composite element that indicates a region of the anatomical scene imaged by the imaging probe within the endoscopic image, the composite element including spatial markers within the region; as well as The display device shows the endoscopic image enhanced with the synthetic elements.

20. The method of claim 19, wherein: The composite elements include boundary elements that delineate the region of the anatomical scene imaged by the imaging probe within the endoscopic image; and The spatial marker is positioned within the boundary element.

21. The method of claim 19, further comprising: Obtain the probe image captured by the imaging probe; as well as The endoscopic image is enhanced by using a selected portion of the probe image located within the region of the anatomical scene imaged by the imaging probe.

22. The method of claim 19, further comprising: The endoscopic image is enhanced using a highlight element positioned to highlight a selected portion of the region of the anatomical scene imaged by the imaging probe.

23. The method of claim 19, further comprising: The threshold time duration for which the imaging probe remains stationary is detected; as well as In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the endoscopic image is enhanced with the synthetic elements.

24. The method of claim 19, wherein enhancing the endoscopic image with the synthetic elements comprises: Determine the orientation of the imaging probe in the anatomical scene; Based on the posture of the imaging probe, the region of the anatomical scene imaged by the imaging probe is determined; as well as The synthetic elements are used to enhance the endoscopic image to visually indicate the region depicted in the endoscopic image.

25. The method of claim 19, further comprising: Obtain the probe image captured by the imaging probe; as well as The display device is instructed to display the probe image outside the composite element.

26. The method of claim 25, further comprising: The probe image is enhanced by an additional set of spatial markers corresponding to the spatial markers of the synthetic elements.

27. The method of claim 25, further comprising: The threshold time duration for which the imaging probe remains stationary is detected; as well as In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the display device is instructed to display the probe image.

28. The method of claim 25, further comprising: The endoscopic image is enhanced with a preview placeholder indicating the display position of the probe image to be captured by the imaging probe.

29. A visualization method, comprising: Obtain endoscopic images of an anatomical scene within the body, the endoscopic images being captured by an endoscope and depicting an imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; Obtain the probe image captured by the imaging probe; The endoscopic image is enhanced with a synthetic element, which indicates the region of the anatomical scene imaged by the imaging probe within the endoscopic image; as well as The display device shows the endoscope image and the probe image, the endoscope image being enhanced by the composite element, and the probe image being positioned outside the composite element.

30. The method of claim 29, wherein the synthetic element comprises: Boundary elements that delineate the region of the anatomical scene imaged by the imaging probe within the endoscopic image; as well as Spatial markers within the boundary elements.

31. The method of claim 29, further comprising: The endoscopic image is enhanced using a selected portion of the probe image positioned within the synthetic element.

32. The method of claim 29, further comprising: The endoscopic image is enhanced using highlight elements positioned within the composite elements to emphasize selected portions of the area of ​​the anatomical scene.

33. The method of claim 29, further comprising: The threshold time duration for which the imaging probe remains stationary is detected; as well as In response to detecting that the imaging probe has been in the stationary state for the specified time duration, the endoscopic image is enhanced with the synthetic elements.

34. The method of claim 29, wherein enhancing the endoscopic image with the synthetic elements further comprises: Determine the orientation of the imaging probe in the anatomical scene; Based on the posture of the imaging probe, the region of the anatomical scene imaged by the imaging probe is determined; as well as The synthetic elements are used to enhance the endoscopic image to visually indicate the region depicted in the endoscopic image.

35. The method of claim 29, wherein the probe image is positioned in the corner region of the endoscopic image.

36. The method of claim 29, wherein: The synthetic elements include a first set of spatial markers; as well as The probe image is enhanced using a second set of spatial markers corresponding to the first set of spatial markers of the synthetic element.

37. A non-transitory computer-readable medium storing instructions, which, when executed, instruct a processor of a computing device: Obtain endoscopic images of an anatomical scene within the body, the endoscopic images being captured by an endoscope and depicting an imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; The endoscopic image is enhanced with a composite element, the composite element indicating a region of the anatomical scene imaged by the imaging probe within the endoscopic image, the composite element including spatial markers within the region; and The display device shows the endoscopic image enhanced with the synthetic elements.

38. A non-transitory computer-readable medium storing instructions, which, when executed, instruct a processor of a computing device: Obtain endoscopic images of an anatomical scene within the body, the endoscopic images being captured by an endoscope and depicting an imaging probe located at the anatomical scene, the imaging probe being different from the endoscope; Obtain the probe image captured by the imaging probe; The endoscopic image is enhanced with a composite element, the composite element indicating a region of the anatomical scene imaged by the imaging probe within the endoscopic image; and The display device shows the endoscope image and the probe image, the endoscope image being enhanced by the composite element, and the probe image being positioned outside the composite element.

Images