Composite medical imaging system and method

Abstract

The invention relates to a composite medical imaging system and method. The composite medical imaging system may instruct a display device to display an image that displays a view of a surgical field captured by the imaging device that displays surface anatomy at the surgical field and an object at the surgical field, and an enhanced region that displays supplemental content, the enhanced region creates an occlusion over at least a portion of a view of the surgical field. The system may detect an overlap in the image between at least a portion of the object and at least a portion of the enhancement region. The system may adjust the image in response to detection of the overlap to reduce the degree of occlusion of the enhancement region within the overlap.

Description

[0001] This application is a divisional application of Chinese patent application 202080051681.8 (PCT / US2020 / 035113) filed on May 29, 2020 and entitled “Composite Medical Imaging System and Method”.

[0002] Related Applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 893,043, filed on August 28, 2019, entitled “COMPOSITE MEDICAL IMAGING SYSTEMS AND METHODS,” and U.S. Provisional Patent Application No. 62 / 855,755, filed on May 31, 2019, entitled “SYSTEMS AND METHODS FOR INTEGRATING IMAGERY CAPTURED BY DIFFERENT IMAGING MODALITIES INTO COMPOSITE IMAGERY OF ASURGICAL SPACE,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0004] An imaging device (e.g., an endoscope) may be used during a surgical procedure to capture images of a surgical field associated with a patient. The images may be presented to a surgeon (e.g., in the form of a video stream) during the surgical procedure to assist the surgeon in performing the surgical procedure. In some examples, supplemental content, such as ultrasound images, may also be presented during the surgical procedure. However, there is room for improvement in the presentation of supplemental content so as not to interfere with the surgical procedure. Summary of the invention

[0005] In order to provide a basic understanding of these aspects, the following description presents a simplified summary of one or more aspects of the methods and systems described herein. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify the key or important elements of all aspects, nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects of the methods and systems described herein in a simplified form as a prelude to the more detailed description presented below.

[0006] An exemplary system may include a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to perform the following operations: instruct a display device to display an image that displays a view of a surgical field associated with a patient captured by an imaging device and an enhanced region that displays supplemental content, the view of the surgical field displaying surface anatomical structures located at the surgical field and objects located at the surgical field, the enhanced region creating an occlusion over at least a portion of the view of the surgical field; detect an overlap between at least a portion of the object in the image and at least a portion of the enhanced region; and in response to detecting the overlap, adjust the image to reduce a degree of occlusion by the enhanced region within the overlap.

[0007] An exemplary method may include instructing a display device, by a composite medical imaging system, to display an image showing a view of a surgical field associated with a patient captured by an imaging device and an enhanced region displaying supplemental content, the view of the surgical field showing surface anatomical structures located at the surgical field and an object located at the surgical field, the enhanced region creating an occlusion over at least a portion of the view of the surgical field; detecting, by the composite medical imaging system, an overlap in the image between at least a portion of the object and at least a portion of the enhanced region; and adjusting, by the composite medical imaging system and in response to detection of the overlap, the image to reduce a degree of occlusion of the enhanced region within the overlap.

[0008] An exemplary non-transitory computer-readable medium may store instructions that, when executed, instruct at least one processor of a computing device to instruct a display device to display an image showing a view of a surgical field associated with a patient captured by an imaging device and an enhanced region displaying supplemental content, the view of the surgical field showing surface anatomical structures located at the surgical field and an object located at the surgical field, the enhanced region creating an occlusion over at least a portion of the view of the surgical field; detecting an overlap in the image between at least a portion of the object and at least a portion of the enhanced region; and, in response to detection of the overlap, adjusting the image to reduce a degree of occlusion by the enhanced region within the overlap. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are examples only and do not limit the scope of the present disclosure. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0010] Figure 1 An exemplary compound medical imaging system according to the principles described herein is illustrated.

[0011] Figure 2 An exemplary composite medical image according to the principles described herein is illustrated.

[0012] Figure 3A An exemplary surgical field associated with a patient is illustrated according to the principles described herein.

[0013] Figure 3B An illustration of a surgical procedure captured by an imaging device located at a surgical field according to the principles described herein. Figure 3A Example endoscopic image of a surgical area.

[0014] Figure 3C The diagram shows a schematic diagram of a system based on the principles described herein. Figure 3B Example slope images generated from endoscopic images of .

[0015] Figure 4A An exemplary virtual surgical field is illustrated according to the principles described herein.

[0016] Figure 4B FIG. 1 illustrates a video captured by a virtual imaging device according to the principles described herein. Figure 4A An exemplary virtual image of a virtual surgical field.

[0017] Figure 4C The diagram shows a schematic diagram of a system based on the principles described herein. Figure 4B An exemplary mask image generated from a virtual image of FIG.

[0018] Figure 5 An exemplary composite medical image displayed by a display device according to the principles described herein is illustrated.

[0019] Fig. 6A An exemplary surgical field associated with a patient is illustrated according to the principles described herein.

[0020] Figure 6B An exemplary composite medical image displayed by a display device according to the principles described herein is illustrated.

[0021] Figure 6C FIG. 1 illustrates an image displayed by a display device according to the principles described herein. Fig. 6A Example images of surgical areas.

[0022] Fig. 7A and Figure 7B An exemplary detection image for detecting overlaps according to the principles described herein is illustrated.

[0023] Figure 8 A virtual surgical field is illustrated according to the principles described herein.

[0024] Fig. 9 An exemplary computer-assisted surgical system according to the principles described herein is illustrated.

[0025] Fig.10 An exemplary method according to the principles described herein is illustrated.

[0026] Fig.11 An exemplary computing device according to the principles described herein is illustrated. DETAILED DESCRIPTION

[0027] Composite medical imaging systems and methods are described herein. As will be explained in more detail below, an exemplary composite medical imaging system can instruct a display device to display an image that displays a view and an augmentation region of a surgical area associated with a patient captured by an imaging device included in a computer-assisted surgical system. The view of the surgical area displays the surface anatomical structure located at the surgical area and an object (e.g., a surgical instrument or a blood pool) located at the surgical area, and the augmentation region displays supplementary content. The augmentation region creates an occlusion above at least a portion of the view of the surgical area. The system can also detect an overlap in the image between at least a portion of the object and at least a portion of the augmentation region. The system can adjust the image in response to the detection of the overlap to reduce the degree of occlusion of the augmentation region within the overlap.

[0028] For illustration, during a minimally invasive surgical procedure performed using a computer-assisted surgical system, a surgeon at a user control system can remotely operate an ultrasound probe and a cautery instrument to perform a surgical procedure on a patient. An endoscope can capture an image of a surgical field associated with a patient, and a display device of the user control system can present the captured image to the surgeon to provide visualization of the surgical field. When the ultrasound probe contacts the surface anatomical structure in the surgical field, an ultrasound image can be generated that represents the subsurface anatomical structure located below the surface anatomical structure contacted by the ultrasound probe. The ultrasound image is then displayed in an enhanced region of the image presented by the display device. The enhanced region creates an occlusion above the captured endoscopic image. In this example, the enhanced region is a region of the display image that appears to "cut into" or "open" the surface anatomical structure to display a representation (e.g., ultrasound image) of the subsurface anatomical structure located below the portion of the surface anatomical structure within the occlusion created by the enhanced region. The enhanced region can protrude from the ultrasound probe and can move with the ultrasound probe, thereby allowing the surgeon to see the subsurface anatomical structure below the surface anatomical structure at any desired position in the surgical field by controlling the position of the ultrasound probe.

[0029] When the ultrasound image is displayed in the enhanced area, the surgeon can move the cautery instrument so that the view of the cautery instrument overlaps with the enhanced area in the image displayed by the display device. When the view of the cautery instrument overlaps with the enhanced area in the displayed image, the enhanced area (or the portion of the enhanced area within the overlapping area) and the corresponding ultrasound image can be removed from the displayed image, and only the image captured by the endoscope is presented. Therefore, the surgeon can easily see the surface anatomical structure at a location near the cautery instrument.

[0030] The systems and methods described herein can provide various benefits. For example, the systems and methods described herein can intelligently present an enhanced region (e.g., a representation of an anatomical structure below the surface, such as an image or ultrasound image of a three-dimensional anatomical structure model) displaying supplementary content in a visual image presented to a user (e.g., a surgeon) during a surgical procedure. In this way, the system and method present useful information (e.g., information about the patient's anatomical structure below the surface) and provide an improved visual experience for the user during a surgical procedure. In addition, when an object in an image (e.g., a surgical instrument, a blood pool, etc.) overlaps with an enhanced region, the systems and methods described herein can intelligently prevent the enhanced region and supplementary content from being included in the visual image. In this way, the system and method provide a view of the surface anatomical structure to facilitate the execution of a surgical procedure. In the following description, these and other benefits of the systems and methods described herein will become apparent.

[0031] Figure 1 An exemplary composite medical imaging system 100 ("system 100") that may be configured to present composite medical images is illustrated. System 100 may be included in, implemented by, or connected to any surgical system or other computing system described herein. For example, system 100 may be implemented by a computer-assisted surgical system. As another example, system 100 may be implemented by a separate computing system communicatively coupled to a computer-assisted surgical system.

[0032] As shown, the system 100 may include, but is not limited to, a storage facility 102 and a processing facility 104 that are selectively and communicatively coupled to each other. Facilities 102 and 104 may each include or be implemented by hardware and / or software components (e.g., a processor, a memory, a communication interface, instructions stored in the memory for execution by the processor, etc.). For example, facilities 102 and 104 may be implemented by any component in a computer-assisted surgical system. In some examples, facilities 102 and 104 may be distributed among multiple devices and / or multiple locations, as may serve a particular implementation.

[0033] The storage facility 102 may maintain (e.g., store) executable data used by the processing facility 104 to perform any of the operations described herein. For example, the storage facility 102 may store instructions 106 that may be executed by the processing facility 104 to perform any of the operations described herein, and the instructions 106 may be implemented by any suitable application, software, code, and / or other executable data instances. The storage facility 102 may also maintain any data received, generated, managed, used, and / or transmitted by the processing facility 104.

[0034] The processing facility 104 may be configured to perform (e.g., execute instructions 106 stored in the storage facility 102 to perform) various operations associated with presenting a composite medical image. For example, the processing facility 104 may be configured to instruct a display device to display an image showing a view of a surgical field associated with a patient captured by an imaging device included in a computer-assisted surgical system, the view of the surgical field showing surface anatomical structures located at the surgical field and objects located at the surgical field. The image also shows an enhanced region displaying supplemental content, the enhanced region creating an occlusion above at least a portion of the view of the surgical field. The processing facility 104 may be further configured to detect an overlap in the image between at least a portion of the object and at least a portion of the enhanced region. The processing facility 104 may also be configured to adjust the image in response to the detection of an overlap to reduce the degree of occlusion of the enhanced region within the overlap. These and other operations that may be performed by the processing facility 104 are described herein. In the following description, any reference to operations performed by the system 100 may be understood to be performed by the processing facility 104 of the system 100.

[0035] The system 100 is configured to instruct a display device to display a composite medical image. As used herein, the composite medical image includes a surgical field image and an enhanced region. The enhanced region creates an occlusion (e.g., opaque or at least partially transparent) over at least a portion of the surgical field image and may display (e.g., include or fill) any supplemental content that may be appropriate for a particular implementation. Figure 2 An exemplary composite medical image 200 (“image 200 ”) is illustrated. As shown, the composite medical image 200 displays a surgical field image 202 and supplemental content 204 presented in an enhanced region 206 of the surgical field image 202 .

[0036] The surgical field image 202 shows a view of the surgical field associated with the patient as captured by an imaging device (e.g., an endoscope). In some examples, the "surgical field" can be completely set in the patient's body and can include a region in the patient's body at or near a location where a surgical procedure is planned, being performed, or has been performed. For example, for a minimally invasive surgical procedure performed on tissue in the patient's body, the surgical field can include surface anatomical structures (e.g., surface tissue), subsurface anatomical structures below the surface anatomical structures (e.g., organs and vascular systems below or behind the surface tissue), and the space around the tissue, such as the space where the surgical instruments used to perform the surgical procedure are located. In other examples, the surgical field can be at least partially set outside the patient's body at or near a location where a surgical procedure is planned, being performed, or has been performed on the patient. For example, for an open surgical procedure, a portion of the surgical field (e.g., the tissue being operated) may be in the patient's body, while another portion of the surgical field (e.g., the space around the tissue where one or more surgical instruments may be set) may be outside the patient's body.

[0037] As used herein, "surface anatomical structures" (sometimes referred to herein as "surface tissues") may be disposed within a patient's body (e.g., an organ, tissue, vasculature, etc.) or may be disposed external to a patient's body (e.g., skin, etc.). In some examples, a view of a surgical field displays surface anatomical structures located at the surgical field by capturing light reflected from the surface anatomical structures. In these examples, surface anatomical structures refer to anatomical structures that are configured to reflect light to an imaging device. Additionally, as used herein, "subsurface anatomical structures" are disposed internally of a patient and are located below or behind the surface anatomical structures relative to a view of the surface anatomical structures captured by an imaging device, and are therefore not configured to reflect light to an imaging device.

[0038] Supplemental content 204 may include any suitable content configured to enhance surgical field image 202, such as, but not limited to, other medical images (e.g., images obtained by fluorescence imaging, ultrasound imaging, computed tomography (CT), optical coherence tomography (OCT), magnetic resonance imaging (MRI), x-ray imaging, or the like), textual content (e.g., labels, surgical procedure information, surgical system information, patient information, messages, etc.), non-medical images (e.g., instructional images or videos, etc.), and the like. In some examples, supplemental content 204 is a representation of a patient's subsurface anatomical structure, such as an ultrasound image, an x-ray image, a fluorescence image, an image of a three-dimensional model generated from a CT scan or MRI scan of the patient, and the like. In other aspects, supplemental content 204 may include a combination (e.g., a hybrid) of multiple different types of supplemental content.

[0039] when Figure 2 While only one example of supplemental content 204 and one enhanced region 206 is shown, image 200 may include any number of different examples of supplemental content and / or enhanced regions as may be appropriate for a particular implementation. Figure 2 The enhanced region 206 is shown as being rectangular, but the enhanced region 206 may be any shape or size suitable for a particular implementation. Figure 2 Enhanced region 206 is shown as being opaque, but may be semi-transparent and thus display a blend of the curvature of supplemental content 204 and the portion of surgical field image 202 that is occluded by enhanced region 206 .

[0040] In some examples, augmented region 206 may be movable within image 200 based on user input. For example, a user (e.g., a surgeon) may manipulate a user input device to change the location of augmented region 206 within image 200. In this manner, the user may view supplemental content 204 at any desired location within image 200.

[0041] In some examples, system 100 may be configured to generate compound medical image 200. Alternatively, compound medical image 200 may be generated by a computing system (e.g., a computing system included in a computer-assisted surgery system) that is communicatively coupled to system 100. Compound medical image 200 may be generated in any suitable manner.

[0042] Figure 3A-Figure 5 An exemplary manner of generating a composite medical image during a minimally invasive surgical procedure performed on a patient is illustrated. However, it will be appreciated that the principles followed may be applied to any other type of surgical procedure, such as an open surgical procedure, a training procedure, a testing procedure, and the like. Additionally, it will be appreciated that the composite medical image may be generated by any other suitable method that may be suitable for a particular embodiment.

[0043] Figure 3A An exemplary surgical field 300 associated with a patient is illustrated. As shown, the surgical field 300 includes a patient anatomy 302, surgical instruments 304 (e.g., an ultrasound probe 304-1 and scissors 304-2), and an imaging device 306 (e.g., an endoscope). The patient anatomy 302 can represent any organ or anatomical feature of the patient and can include surface anatomy 308 (e.g., surface tissue) and subsurface anatomy (e.g., subsurface tissue, organs, vasculature, etc.) ( Figure 3A not shown).

[0044] The ultrasound probe 304-1 is configured to capture ultrasound images by emitting sound waves and detecting sound waves reflected from the subsurface anatomical structure below the surface anatomical structure 308. The scissors 304-2 are configured to cut patient tissue. The surgical instrument 304 can have any suitable shape and / or size for a particular embodiment. In some examples, the surgical instrument 304 can have a shape and size that allows the surgical instrument 304 to be inserted into the patient's body through a port in the patient's body wall. In these examples, the movement and operation of the surgical instrument 304 in the patient's body can be manually controlled (e.g., by manually manipulating an axis connected to the ultrasound probe 304-1 or the scissors 304-2). Additionally or alternatively, the surgical instrument 304 can be controlled in a computer-assisted manner (e.g., by a computer-assisted surgical system utilizing a robot and / or remote operation technology).

[0045] In some examples, the posture (e.g., position and / or orientation) of surgical instrument 304 in surgical field 300 can be tracked by a computer-assisted surgical system. For example, surgical instrument 304 can include one or more sensors (e.g., displacement transducers, orientation sensors, position sensors, etc.) for generating kinematic information. Kinematic information can include information such as posture (e.g., position and / or orientation), motion (e.g., velocity, direction, acceleration, etc.), state (e.g., open, closed, stowed, etc.), and / or other properties of surgical instrument 304, all of which can be tracked by a computer-assisted surgical system.

[0046] The imaging device 306 is implemented by a stereo endoscope configured to capture stereo images of the surgical field 300. However, it should be understood that the imaging device 306 can be implemented by any other suitable imaging device. In some examples, the imaging device 306 can be coupled to a computer-assisted surgical system and controlled in a computer-assisted manner. The image data representing one or more images captured by the imaging device 306 can constitute one or more still images and / or videos captured by the imaging device 306.

[0047] Figure 3B An exemplary endoscopic image E captured by the imaging device 306 is illustrated. The endoscopic image E has I×J pixels and represents only one of a pair of stereo images (e.g., a left-eye image or a right-eye image). It will be understood that the following description is equally applicable to the other of the pair of stereo images captured by the imaging device 306. Similarly, the following description may also be applicable to a monoscopic image captured by the imaging device 306. Figure 3BAs shown, endoscopic image E shows a view of surgical field 300 captured by imaging device 306. Endoscopic image E may be generated by illuminating surgical field 300 and capturing light reflected by surface anatomical structure 308 and surgical instrument 304 to imaging device 306. Thus, view of surgical field 300 shows view 300 of surgical instrument 304 and surface anatomical structure 308 located at surgical field 300.

[0048] In some examples, a slope image may be generated from the endoscopic image E. As will be explained in more detail below, the slope image may be used in the generation of the composite medical image to enhance the user's depth perception. However, in other examples, the slope image may be omitted from the composite medical image. Figure 3C FIG. 4 illustrates an exemplary slope image S generated from an endoscopic image E. The slope image S is Figure 3C , illustrated in black and white. However, the slope image S may include black, white, and / or various shades of gray representing the slope. The slope image S is generated based on the gradient of the endoscopic image E and therefore will have the same size as the endoscopic image E (I×J pixels). The slope image S may be generated in any suitable manner using any suitable algorithm. In some examples, the slope image S is generated based on the horizontal and / or vertical components of the gradient of the endoscopic image E. For example, the pixel value S in the slope image S (i，j) can be set to the next horizontally adjacent pixel E in the mirror image E (i+1,j) minus the pixel E in the endoscopic image E (i，j) If S (i，j) is less than zero, then S (i，j) For pixels that have no horizontally adjacent pixels on the rightmost edge, the next horizontally adjacent pixel E (i+1,j) The value of is set to a predetermined value (eg, black, the average value of the pixel in the surrounding area, etc.). This process can be performed for each pixel until the ramp image S is completed.

[0049] Reference now Figure 4A , generating a three-dimensional ("3D") virtual surgical field 400. Virtual surgical field 400 includes one or more representations of subsurface anatomical structures below surface anatomical structures 308 located at surgical field 300. For example, Figure 4A As shown, the virtual surgical field 400 includes a 3D model 402 of the subsurface anatomical structure (e.g., a 3D model of the subsurface anatomical structure generated based on preoperative MRI and / or CT scan imaging) and an ultrasound image 404 generated by the ultrasound probe 304-1. The ultrasound image 404 represents the subsurface anatomical structure located below the surface anatomical structure 308. Figure 4AAs shown, the virtual surgical field 400 also includes a frame 406 onto which the ultrasound image 404 is projected, such as by texture mapping. The frame 406 will be described in more detail below. It will be appreciated that any one or more of the 3D model 402, the ultrasound image 404, and the frame 406 may be omitted from the virtual surgical field 400 without departing from the scope of the present disclosure.

[0050] The virtual surgical field 400 also includes a virtual imaging device 408 that is configured to “capture” images of the virtual surgical field 400 . Figure 4B An exemplary virtual image V ("Virtual Image V") captured by the virtual imaging device 408 is illustrated. Figure 4A As shown, the posture (e.g., position and orientation) of the virtual imaging device 408 within the virtual surgical field 400 determines the view of the virtual surgical field 400, including the view of the 3D model 402, the ultrasound image 404, and the frame 406, as depicted by virtual figure V.

[0051] In some examples, the view of the virtual surgical field 400 captured by the virtual imaging device 408 can be registered with the view of the surgical field 300 captured by the imaging device 306. In other words, the 3D model 402 and the ultrasound image 404 can be positioned and oriented within the virtual surgical field 400 so that the combined endoscopic image E and the virtual image V (see Figure 5 ), the 3D model 402 and / or ultrasound image 404 appear to be located in the patient's body at their actual physical locations relative to the surface anatomical structure 308 in a composite medical image C ("composite image C"). Any suitable registration technique can be used to register the view of the virtual surgical field 400 captured by the virtual imaging device 408 with the view of the surgical field 300 captured by the imaging device 306. In some examples, the registration can be based on a depth map of the surgical field, such as generated based on the image captured by the imaging device 306. The pose of the virtual imaging device 408 in the virtual surgical field 400 can be configured to change based on changes in the pose of the imaging device 306 in the surgical field 300, thereby maintaining the registration of the virtual image V with the endoscopic image E. In some examples, the virtual imaging device 302 is also configured with the same camera characteristics (e.g., resolution, zoom level, focal length, etc.) as the imaging device 306.

[0052] like Figure 4A As shown, box 406 is a box positioned and oriented within virtual surgical field 400. As will be described in more detail below, box 406 can be used to define an enhanced region in composite image C that provides a view of a representation of the underlying anatomical structure (e.g., 3D model 402 and / or ultrasound image 404).

[0053] The frame 406 can be selectively moved within the virtual surgical field 400 based on user input. In some examples, the posture of the frame 406 within the virtual surgical field 400 can be based on the posture of the surgical instrument 304 located at the surgical field 300. As an example, the position of the frame 406 in the virtual surgical field 400 can be based on the position of the ultrasound probe 304-1 in the surgical field 300 and the orientation of the frame 406 in the virtual surgical field 400 (e.g., the direction of the plane of the frame 406) is based on the orientation (e.g., rotation) of the ultrasound probe 304-1 in the surgical field 300. For example, the frame 406 can be oriented in the direction in which the ultrasound probe 304-1 transmits and receives ultrasound signals, thereby ensuring that the ultrasound image 404 generated by the ultrasound probe 304-1 accurately represents the subsurface anatomical structure located below the surface anatomical structure 308 when projected onto the frame 406.

[0054] like Figure 4B As shown, the virtual image V shows a view of a virtual surgical field 400, including a 3D model 402, an ultrasound image 404, and a frame 406 captured by a virtual imaging device 408. Thus, the view of the virtual surgical field 400 in the virtual image V shows a view of a representation of the subsurface anatomical structure located at the surgical field 300. In some examples, the virtual image V has the same size as the endoscopic image E (I×J pixels).

[0055] Then a mask image M is generated from the virtual image V. Figure 4C An exemplary mask image M is illustrated. The mask image M is set to the same size (I×J pixels) as the virtual image V and includes a mask region 410 and a window region 412. The size, shape, and position of the window region 412 determine the size, shape, and position of the enhancement region included in the composite image C. Figure 4B and Figure 4C As shown, the view of the ultrasound image 404 projected onto the frame 406 in the virtual image V sets the size, shape and position of the window area 412 in the mask image M. In an alternative embodiment, the view of the frame 406 in the virtual image V can set the size, shape and position of the window area 412 in the mask image M.

[0056] In some embodiments, all pixels in the mask area 410 are set to display black, while all pixels in the window area 412 are set to display white. For example, in the case where the pixel value ranges from 0 (black) to 1 (white), if the corresponding pixel V (i，j) is not included in the view of the ultrasound image 404, then the pixel M (i，j) The value of is set to 0 if the corresponding pixel V in the virtual image V (i，j) is included in the view of ultrasound image 404, then pixel M(i，j) The value of is set to 1 (white). With this arrangement, the mask image M is configured to mask all pixels in the virtual image V that do not overlap with the view of the ultrasound image 404 in the virtual image V.

[0057] A composite image C is generated by combining images E, S, V, and M. Figure 5 An exemplary composite image C having the same size (I×J pixels) as the endoscopic image E captured by the imaging device 306 is illustrated. The composite image C may be generated in any suitable manner. In some examples, the composite image C is generated by setting each pixel C according to the following equation [1] (i，j) The values ​​of are used to mix images E, S, V and M to generate:

[0058] C (i，j) =M (i，j) *(S (i，j )+V (i，j) )+(1-M (i，j) )*E (i，j) [1]

[0059] Therefore, when M (i，j) =0 (i.e., pixel M in mask image M (i，j) is 0, i.e. black), pixel C in the composite image C (i，j) The value of pixel E (i，j) That is, the mask region 410 of the mask image M masks the slope image S and the virtual image V so that only the endoscopic image E captured by the imaging device 306 is displayed. (i，j) =1 (i.e., pixel M(i, j) in the mask image M is 1, i.e., white), pixel C in the composite image C (i，j) The value is S (i，j) +V (i，j) That is, the window area 412 of the mask image M provides an opaque enhanced area in the endoscopic image E, in which only the combination of the slope image S and the virtual image V is displayed, so that the enhanced area creates an occlusion of the corresponding area of ​​the endoscopic image E.

[0060] In an alternative embodiment, the enhanced region may be transparent. The enhanced region may be made transparent by any suitable method. For example, if the corresponding pixel V (i，j) is included in the view of ultrasound image 404, then pixel M (i，j) The value of can be set to be less than 1 (e.g., 0.85). According to equation [1], when M (i，j) <1, pixel C in composite image C (i，j) The value is S (i，j) 、V (i，j) and E (i，j)That is, the window area 412 of the mask image M provides a transparent enhancement area that only creates a partial occlusion of the corresponding area of ​​the endoscopic image E. In some examples, the degree of occlusion of the corresponding area of ​​the endoscopic image E can be adjusted by automatically or manually adjusting (e.g., increasing or decreasing) the transparency of the enhancement area.

[0061] like Figure 5 As shown, composite image C displays a view of a surgical field 300 associated with a patient and an enhanced region 502. The view of the surgical field 300 displays surgical instruments 304 (e.g., ultrasound probe 304-1 and scissors 304-2) and surface anatomical structures 308 located at the surgical field 300, while the enhanced region 502 displays a representation of the subsurface anatomical structures (e.g., ultrasound image 404). In embodiments where the enhanced region 502 is created based on block 406, the enhanced region 502 additionally or alternatively displays a view of the 3D model 402. Furthermore, because the views of the 3D model 402 and ultrasound image 404 are registered with the view of the surgical field 300, as described above, the 3D model 402 and ultrasound image 404 provide an accurate representation of the subsurface anatomical structures at the in vivo location of the subsurface anatomical structures represented by the 3D model 402 and ultrasound image 404.

[0062] As described above, the slope image S is generated based on the gradient of the endoscopic image E. As a result, combining the slope image S with the virtual image V in the enhanced region 502 enhances the viewer's perception of the composite image C that the 3D model 402 and / or the ultrasound image 404 is located below the surface anatomical structure 308. However, in other examples, the slope image S may be omitted from the composite image C such that the enhanced region 502 displays only the virtual image V.

[0063] In composite image C, enhanced region 502 is depicted as protruding / projecting from ultrasound probe 304-1, similar in appearance to a flag protruding from a flagpole. In alternative embodiments, enhanced region 502 protrudes from a different type of surgical instrument located in surgical field 300 or from a virtual surgical instrument rather than an actual surgical instrument located in surgical field 300. For example, virtual surgical field 400 may include virtual surgical instruments ( Figure 4A-4C The ultrasound image 404 and / or the frame 406 are positioned in the virtual surgical field 400 based on the posture of the virtual surgical instrument in the virtual surgical field 400 .

[0064] In further embodiments, the enhanced region 502 does not protrude from any real or virtual surgical instrument, but is movable based on user input. For example, the user can selectively operate a controller (e.g., a joystick, a master controller, etc.) to move the position of the frame 306 in the virtual surgical field 400, thereby also moving the enhanced region 502 in the composite image C. In this manner, the user can move the enhanced region 502 to view subsurface anatomical structures without having to move surgical instruments located in the surgical field 300.

[0065] As mentioned above, Figure 4C and Figure 5 The mask image M is shown to set the shape of the enhanced area 502 to be the same shape as the view of the ultrasound image 404 in the virtual image V. In an alternative embodiment, the mask image M sets the shape of the enhanced area 502 to be the same shape as the view of the frame 306 in the virtual image V. This configuration may be used when the ultrasound image 404 is not projected onto the frame 306, thereby displaying the 3D model 402 through the enhanced area 502 in the composite image C. Alternatively, when the ultrasound image 404 is projected onto the frame 306, the 3D model 402 may be displayed in an area of ​​the frame 406 that is not covered by the ultrasound image 404.

[0066] As described above, system 100 can instruct a display device to display composite image C. For example, system 100 can instruct a display device associated with a computer-assisted surgical system to display composite image C when a user activates composite image mode. The display device can be any suitable display device, such as a display device of a user-controlled system used by a surgeon during a computer-assisted surgical procedure.

[0067] When displaying the composite image C, the system 100 can detect an overlap in the composite image C between at least a portion of an object located at the surgical field 300 and at least a portion of the enhanced region 502. The object located at the surgical field can include any foreign matter introduced into the surgical field 300 (e.g., a surgical instrument, a surgical grid, a surgeon's hand or finger, etc.) and / or any object naturally present in the surgical field (e.g., blood, tissue, organs, etc.). For example, the system 100 can detect that the surgical instrument has moved such that the view of the surgical instrument overlaps with a portion of the enhanced region 502 in the composite image C. As another example, the system 100 can detect that a view of a pool of blood overlaps with a portion of the enhanced region 502 in the composite image C.

[0068] In response to detecting an overlap between a portion of the object and a portion of the enhanced region 502, the system 100 can adjust the displayed image to reduce the degree of occlusion within the overlap of the enhanced region 502. In some examples, the system 100 can adjust the displayed image by removing the enhanced region from the image. For example, the system 100 can switch from displaying the composite image C to displaying another image that does not display the enhanced region 502 (e.g., an endoscopic image E', see Figure 6C ). Thus, the enhanced region 502 does not obstruct the view of the surgical field 300 captured by the imaging device. As another example, the system 100 can reduce the size and / or shape of the enhanced region 502, such as by removing a portion of the enhanced region 502 within a predefined vicinity of the object (e.g., within an overlapping region).

[0069] As another example of adjusting the displayed image to reduce the degree of occlusion, the system 100 can reduce the opacity of the enhanced region 502 in the composite image C. For example, the system 100 can adjust the blending of the images E, S, V, and M so that the enhanced region 502 is more transparent. In some examples, the transparency can be modulated, for example, based on the distance of the object to the surface anatomical structure and / or based on the type of object that overlaps with the enhanced region 502. Additionally or alternatively, the system 100 can adjust the blending of the images E, S, V, and M only within a predetermined vicinity of the object (e.g., within the overlapping region).

[0070] As another example of reducing the opacity of the enhanced region 502, the system 100 can adjust the slope image S to be more visible in the enhanced region 502. For example, the system 100 can adjust one or more parameters of an image filter of the slope image S. Because the slope image S is derived from the endoscopic image E, increasing the visibility of the slope image S will increase the visibility of the view of the surgical field 300 captured by the imaging device. In this way, the system 100 can adjust the composite image C to reduce the degree to which the enhanced region 502 obstructs the view of the surgical field 300.

[0071] Fig. 6A and Figure 6B An example of an overlap between at least a portion of enhancement region 502 and at least a portion of a view of an object located at surgical field 300 is illustrated. Fig. 6A and Figure 3A Same as above, except that the scissors 304-2 have been moved from their initial position (at Fig. 6A ) moves to a new position closer to the ultrasound probe 304-1 (indicated by dashed lines in FIG. Fig. 6A (shown by solid lines in FIG. 2 ). Figure 6BAn exemplary composite image C' is illustrated after the scissors 304-2 have been moved to a new position. The composite image C' shows a view of the surgical field 300 and an enhanced region 502 that displays the ultrasound image 404 generated by the ultrasound probe 304-1. Figure 6B As shown, the distal portion of the scissors 304-2 overlaps with a portion of the enhanced region 502 in the composite image C'. The system 100 is configured to detect the overlap between the enhanced region 502 and the scissors 304-2 and, in response, instruct the display device to display only the endoscopic image E', which shows the view of the surgical field 300 captured by the imaging device 306. Figure 6C An exemplary endoscopic image E' captured after the scissors 304-2 have been moved to a new position is illustrated. It can be seen that the enhancement region 502 has been removed so that the surgeon can easily see the surface anatomy 308 at the surgical field 300 and therefore determine the position of the scissors 304-2 relative to the surface anatomy 308.

[0072] The manner in which the system 100 can detect an overlap in a composite medical image between at least a portion of an enhanced region (e.g., the enhanced region 206 or the enhanced region 502 including the 3D model 402 and / or the ultrasound image 404) and a view of an object located at the surgical field will now be described. The system 100 can detect an overlap in a composite medical image between at least a portion of an enhanced region and a view of an object in other manners.

[0073] Now refer to Fig. 7A and Figure 7B An exemplary approach to position tracking based overlap detection is described. Fig. 7A An exemplary detection image D for detecting overlap is illustrated. Detection image D includes an enhanced region representation diagram 702 and an object representation diagram 704. Enhanced region representation diagram 702 represents an enhanced region in a composite medical image (e.g., enhanced region 206 in image 200 or enhanced region 502 in composite image C). Object representation diagram 704 represents an object (e.g., scissors 304-2) located at a surgical field associated with a patient and is positioned within detection image D based on the position of the object in the surgical field. The generation of detection image D will be described in more detail below. In some examples, when it is detected that object representation diagram 704 contacts enhanced region representation diagram 702 in detection image D, an overlap in the composite medical image between the enhanced region and the object can be detected, as will be explained in more detail below.

[0074] Fig. 7AThe detection image D generated when the object (e.g., the scissors 304-2) is located at the initial position is shown. As shown in the figure, the object representation image 704 does not contact the enhanced region representation image 702 because the pixels of the object representation image 704 are not adjacent to or overlapped with the pixels of the enhanced region representation image 702. Therefore, the system 100 does not detect the overlap between the object and the enhanced region in the composite medical image (e.g., the enhanced region 502 in the composite image C).

[0075] Figure 7B An exemplary detection image D' generated after the object is moved to a new position in the surgical field is shown. As shown, the object representation 704 is in contact with the enhanced region representation 702 because one or more pixels of the object representation 704 are adjacent to or overlap one or more pixels of the enhanced region representation 702. In response to detecting that the object representation 704 is in contact with the enhanced region representation 702, the system 100 determines that at least a portion of the enhanced region (e.g., the enhanced region 502 in the composite image C) overlaps with the object.

[0076] In some embodiments, the detection image D does not include the enhanced region representation map 702. Since the detection image D is set to the same size (I×J pixels) as the mask image M, the window region (e.g., the window region 412) in the mask image M represents the enhanced region (e.g., the enhanced region 502) in the composite medical image. Therefore, overlap can be detected by comparing the detection image D with the mask image M to determine whether the object representation map 704 in the detection image D and the window region (e.g., the window region 412) in the mask image M have one or more common pixel locations.

[0077] The generation of the detection image D will now be described. In some examples, the detection image D is set to the same size (I×J pixels) as the mask image M. As described above, the enhanced region representation 702 represents the enhanced region in the composite medical image. Therefore, an enhanced region representation 702 is formed in the detection image D, which has the same size, shape, and position as the enhanced region in the composite medical image (e.g., the enhanced region 206 in the image 200 or the enhanced region 502 in the composite image C). The enhanced region representation 702 can be formed in the detection image D in any suitable manner. For example, the mask image M can be inverted by setting the pixels in the mask area 410 to white and the pixels in the window area 412 to black. Alternatively, the ultrasound image 404 and / or the frame 406 can be projected onto the detection image D, for example, by capturing the detection image D through the virtual imaging device 408. That is, the detection image D shows a view of the ultrasound image 404 and / or the frame 406 in the surgical area 400 captured by the virtual imaging device 408. As yet another example, the enhanced region representation map 702 may be formed in the detected image D by using object recognition to detect the size, shape, and location of the enhanced region in the composite medical image.

[0078] As described above, object representation 704 represents objects located in a surgical field associated with a patient. Object representation 704 may be any size or shape suitable for a particular implementation. Fig. 7A As shown, object representation 704 is circular and represents, for example, a distal portion of a surgical instrument (e.g., scissors 304-2). In addition, object representation 704 is larger than the distal portion of the surgical instrument, thereby including the entire area in which the end effector of the surgical instrument can move. In other examples, object representation 704 has the same shape and size as the distal portion of the surgical instrument. Adjusting the size and / or shape of object representation 704 will adjust the sensitivity of system 100 to detect overlap between the enhanced region in the composite medical image and an object located in the surgical field. Therefore, in some examples, system 100 can be configured to receive user input to adjust the sensitivity of overlap detection, for example by adjusting the size and / or shape of object representation 704.

[0079] The object representation 704 may be formed in the detection image D in any suitable manner. In some examples, the object representation 704 is formed by tracking the pose (e.g., position and / or orientation) of the corresponding object in the surgical field associated with the patient and projecting the view of the object (or the representation of the object) onto the detection image D. For example, as described above, the view of the virtual surgical field 400 in the virtual image V is aligned with the view of the surgical field 300 in the endoscopic image E. Therefore, the pose of the scissors 304-2 may also be tracked in the virtual surgical field 400. A 3D bounding volume representing the distal portion of the scissors 304-2 may then be generated in the virtual surgical field 400 at the tracked position of the distal portion of the scissors 304-2. The 3D bounding volume representing the scissors 304-2 may then be projected onto the detection image D as the object representation 704.

[0080] Figure 8 An exemplary virtual surgical field 800 is illustrated that includes a 3D bounding volume representing the scissors 304 - 2 located in the surgical field 300 . Figure 8 and Figure 4A The same, except that the virtual surgical field 800 includes a spherical bounding volume 802 generated around the distal portion of the scissors 304-2. Figure 8 The virtual surgical field 800 is shown to include the ultrasound image 404 and the frame 406, but these may be omitted from the virtual surgical field 800 in some embodiments. The bounding volume 802 may be any shape and size suitable for a particular implementation. Because the bounding volume 802 is projected onto the detection image D as the object representation 704, the shape and size of the bounding volume 802 determines the shape and size of the object representation 704. Figure 8 As shown, the bounding volume 802 is spherical and is sized to include at least the entire area in which the distal portion of the scissors 304-2 (e.g., the end effector of the scissors 304-2) can move. In an alternative embodiment, the bounding volume 802 may have the same shape and / or size as the scissors 304-2. Once the bounding volume 802 is generated in the virtual surgical field 800, the bounding volume 802 is projected onto the detection image D as the object representation map 704. For example, the virtual imaging device 408 can "capture" an image of the virtual surgical field 800 as the detection image D, and the image of the virtual surgical field 800 shows the bounding volume 802 as a view of the object representation map 704.

[0081] Object representation 704 may alternatively be formed in detection image D based on object recognition. For example, system 100 may be configured to analyze an image showing a view of a surgical field (e.g., endoscopic image E) to recognize and identify an object located in the surgical field in the image. System 100 may utilize any suitable image recognition technology to recognize and identify an object. When an object is detected, system 100 may generate object representation 704 in detection image D at a location corresponding to the location in the image of the surgical field where the object was detected.

[0082] In some examples, the size and shape of the object representation 704 is predefined based on the type of object detected. For illustration, the system 100 may analyze the endoscopic image E (see Figure 3B ) and detects scissors 304-2. System 100 may then access a lookup table to determine the shape (e.g., circle), size (e.g., radius of 100 pixels), and location (e.g., at the joint of the end effector) of object representation map 704 of scissors 304-2.

[0083] In an alternative example, the shape and size of the object representation 704 is based on the size and shape of the object detected in the image. To illustrate, the system 100 can analyze an image of a surgical field and detect a blood pool. The system 100 can then draw the object representation 704 in the detected image D to have the same shape and size as the detected blood pool. In this way, the system 100 can be configured to detect when a view of excessive bleeding overlaps the enhanced area 502 in the composite image C.

[0084] Forming an object representation map 704 based on object recognition may be used to detect overlap with objects for which kinematic or positional information may not exist, such as surgical meshes, patches, a surgeon's fingers or hands, and the like.

[0085] As described above, in response to detecting an overlap between at least a portion of an object and at least a portion of an enhanced region included in a composite medical image, the system 100 is configured to adjust the image to reduce the degree of occlusion above at least a portion of the view of the surgical field of the enhanced region. In some examples, the adjustment of the image is further conditioned on detecting that the object is in motion. In this example, the system 100 infers from the movement of the object (e.g., a surgical instrument) that the surgeon intends to interact with the patient tissue. However, if the object is stationary and the enhanced region is moving (e.g., the user is moving the enhanced region), the system 100 infers from the lack of movement of the object that the surgeon does not intend to interact with the patient tissue. Therefore, by displaying the enhanced region, the risk of contact with the patient tissue is small. The system 100 can determine that the object is in motion in any suitable manner, such as based on kinematic information associated with the object, image recognition, sensors included in the object (e.g., surgical instrument sensors), and the like.

[0086] In some examples, the system 100 is configured to adjust the image only when it is detected that the object is moving toward the enhanced region. Object motion toward the enhanced region can be detected in any suitable manner. In some embodiments, the object can infer motion toward the enhanced region by detecting object motion toward the patient tissue because the enhanced region displays a view of the subsurface anatomical structure below the patient tissue. To this end, a depth map can be generated from a stereoscopic image of the surgical area, and the motion of the object (e.g., the distal end of the surgical procedure) relative to the surface tissue can be determined based on the depth map and / or kinematic information associated with the object.

[0087] Additionally or alternatively using the depth map, if the object is detected to be moving away from an imaging device (e.g., imaging device 306), the system 100 can determine that the object is moving toward the surface tissue. On the other hand, if the object is moving toward the imaging device, the object is moving away from the surface tissue (and thus determined to be moving away from the enhancement region). The movement of the object away from or toward the imaging device can be detected in any suitable manner, such as based on kinematic information associated with the object and the imaging device.

[0088] In some embodiments, the system 100 can be configured to detect that the overlap between the enhanced region and the object in the composite medical image has ended. In response, the system 100 can adjust the image to open and / or increase the degree of occlusion over a portion of the view of the surgical field by the enhanced region. For example, when the display device is displaying the composite image E' (see Figure 6C), the system 100 can detect that the scissors 304-2 have moved to another position so that the object representation 704 in the detection image D does not contact the enhanced area representation 702. Therefore, the system 100 can instruct the display device to display another composite image C, which displays a view of the surgical area 300 and the enhanced area 502, and the composite image C includes the ultrasound image 404 and / or the 3D model 402. As another example, if the opacity of the enhanced area (e.g., the enhanced area 502) is reduced due to the detection of an overlap between the enhanced area and the object, the system 100 can adjust the blending of images E, S, V, and M to increase the opacity of the enhanced area. In these ways, the system 100 can intelligently hide the supplementary content only when the view of the object located in the surgical area overlaps with at least a portion of the enhanced area including the supplementary content.

[0089] In the foregoing description, system 100 is configured to detect an overlap between an object located at a surgical field associated with a patient and at least a portion of an enhanced region included in a composite medical image, the enhanced region displaying a representation of a subsurface anatomical structure. However, system 100 is not limited to detecting an overlap with a portion of an enhanced region displaying a subsurface anatomical structure, but may be configured to detect an overlap with an enhanced region displaying any other type of supplemental content (one or more).

[0090] In some embodiments, the system 100 can be operated as part of or in conjunction with a computer-assisted surgical system. Thus, an exemplary computer-assisted surgical system will now be described. The exemplary computer-assisted surgical system described is illustrative and not limiting. The system 100 can be operated as part of or in conjunction with the computer-assisted surgical system described herein and / or other suitable computer-assisted surgical systems.

[0091] Fig. 9 An exemplary computer-assisted surgical system 900 ("surgical system 900") is illustrated. As described herein, system 100 may be implemented by, connected to, and / or otherwise used in conjunction with surgical system 900.

[0092] As shown, surgical system 900 may include a manipulation system 902, a user control system 904, and an auxiliary system 908 that are communicatively coupled to one another. Surgical system 900 may be used by a surgical team to perform a computer-assisted surgical procedure on a patient 908. As shown, the surgical team may include a surgeon 910-1, an assistant 910-2, a nurse 910-3, and an anesthesiologist 910-4, who may be collectively referred to as "surgical team members 910." Additional or alternative surgical team members may be present during a surgical conference as may serve a particular embodiment.

[0093] Although Fig. 9 A minimally invasive surgical procedure is illustrated as being performed, but it will be appreciated that the surgical system 900 may similarly be used to perform open surgical procedures or other types of surgical procedures that may similarly benefit from the accuracy and convenience of the surgical system 900. Furthermore, it will be appreciated that a surgical session that may be performed throughout employing the surgical system 900 may include not only the operational phase of a surgical procedure, such as Fig. 9 As shown, but may also include pre-surgical, post-surgical and / or other suitable stages of a surgical procedure. Any surgical procedure using manual and / or instrumental techniques on a patient to investigate or treat a physical condition of the patient may be included.

[0094] like Fig. 9 As shown, the manipulation system 902 may include a plurality of manipulator arms 912 (e.g., manipulator arms 912-1 to 912-4), and a plurality of surgical instruments may be coupled to the plurality of manipulator arms 912. Each surgical instrument may be implemented by any suitable surgical tool (e.g., a tool having tissue interaction functionality), a medical tool, an imaging device (e.g., an endoscope), a sensing instrument (e.g., a force sensing surgical instrument), a diagnostic instrument, or a device that can be used for a computer-assisted surgical procedure on a patient 908 (e.g., by being at least partially inserted into the patient 908 and being manipulated to perform a computer-assisted surgical procedure on the patient 908). Although the manipulation system 902 is depicted and described herein as including four manipulator arms 912, it should be appreciated that the manipulation system 902 may include only a single manipulator arm 912 or any other number of manipulator arms that may be used for a particular embodiment.

[0095] Manipulator arm 912 and / or a surgical instrument attached to manipulator arm 912 may include one or more displacement sensors, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of surgical system 900 may be configured to use kinematic information to track (e.g., determine its position and orientation) and / or control a surgical instrument.

[0096] The user control system 904 can be configured to facilitate the surgeon 910-1 to control the manipulator arm 912 and the surgical instruments attached to the manipulator arm 912. For example, the surgeon 910-1 can interact with the user control system 904 to remotely move or manipulate the manipulator arm 912 and the surgical instruments. To this end, the user control system 904 can provide the surgeon 910-1 with an image (a high-definition 3D image, a composite medical image, etc.) of the surgical field associated with the patient 908 captured by an imaging system (e.g., including any imaging device described herein). In some examples, the user control system 904 may include a stereoscopic viewer with two displays, wherein a stereoscopic image of the surgical field associated with the patient 908 and generated by the stereoscopic imaging system can be viewed by the surgeon 910-1. The surgeon 910-1 can utilize the image to perform one or more procedures in which one or more surgical instruments are attached to the manipulator arm 912.

[0097] To facilitate control of surgical instruments, user control system 904 may include a set of master controllers. These master controllers may be manipulated by surgeon 910-1 to control the movement of surgical instruments (e.g., using robotics and / or teleoperation techniques). The master controllers may be configured to detect various hand, wrist, and finger movements of surgeon 910-1. In this way, surgeon 910-1 may intuitively perform procedures using one or more surgical instruments.

[0098] The auxiliary system 906 may include one or more computing devices configured to perform the primary processing operations of the surgical system 900. In such a configuration, the one or more computing devices included in the auxiliary system 906 may control and / or coordinate operations performed by various other components of the surgical system 900 (e.g., the manipulator system 902 and the user control system 904). For example, a computing device included in the user control system 904 may transmit instructions to the manipulator system 902 via one or more computing devices included in the auxiliary system 906. As another example, the auxiliary system 906 may receive and process image data from the manipulator system 902 representing an image captured by an imaging device attached to one of the manipulator arms 912.

[0099] In some examples, the assistance system 906 can be configured to present visual content to a surgical team member 910 who may not have access to the images provided to the surgeon 910-1 at the user control system 904. To this end, the assistance system 906 can include a display monitor 914 that is configured to display one or more user interfaces, such as an image of the surgical field (e.g., a 2D image, a composite medical image, etc.), information associated with the patient 908 and / or the surgical procedure, and / or any other visual content as may be used for a particular embodiment. For example, the display monitor 914 can display an image of the surgical field and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. In some embodiments, the display monitor 914 is implemented by a touch screen display with which the surgical team member 910 can interact (e.g., via touch gestures) to provide user input to the surgical system 900.

[0100] The operating system 902, the user control system 904, and the auxiliary system 906 may be communicatively coupled to each other in any suitable manner. Fig. 9 As shown, the operating system 902, the user control system 904, and the auxiliary system 906 can be communicatively coupled via a control line 916, which can represent any wired or wireless communication link as may be used in a particular embodiment. To this end, the operating system 902, the user control system 904, and the auxiliary system 906 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0101] Fig.10 An exemplary method 1000 is shown. Fig.10 The exemplary operations according to one embodiment are illustrated, and other embodiments may omit, add, reorder, combine and / or modify Fig.10 Follow any of the steps shown in . Fig.10 One or more of the operations shown in may be performed by the system 100, any components included therein, and / or any implementation thereof.

[0102] In operation 1002, a composite medical imaging system instructs a display device to display an image (e.g., composite medical image C) showing a view of a surgical field associated with a patient as captured by an imaging device included in a computer-assisted surgery system and an enhanced region. The view of the surgical field shows surface anatomical structures located in the surgical field and objects located in the surgical field, and the enhanced region shows supplemental content, the enhanced region creating an occlusion over at least a portion of the view of the surgical field. Operation 1002 can be performed in any manner described herein.

[0103] In operation 1004, the composite medical imaging system detects an overlap in the image between at least a portion of the object and at least a portion of the enhancement region. Operation 1004 may be performed in any manner described herein.

[0104] In operation 1006, the composite medical imaging system adjusts the image in response to the detection of the overlap to reduce the degree of occlusion of the enhanced region within the overlap. Operation 1006 may be performed in any manner described herein.

[0105] In some examples, a non-transitory computer-readable medium storing computer-readable instructions can be provided according to the principles described herein. The instructions, when executed by a processor of a computing device, can instruct the processor and / or computing device to perform one or more operations, including one or more operations described herein. Such instructions can be stored and / or transmitted using any of a variety of known computer-readable media.

[0106] Non-transitory computer-readable media referred to herein may include any non-transitory storage medium that participates in providing data (e.g., instructions) that can be read and / or executed by a computing device (e.g., by a processor of a computing device). For example, non-transitory computer-readable media may include, but are not limited to, any combination of non-volatile storage media and / or volatile storage media. Exemplary non-volatile storage media include, but are not limited to, read-only memory, flash memory, solid-state drives, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), ferroelectric random access memory ("RAM"), and optical disks (e.g., compact disks, digital video disks, Blu-ray disks, etc.). Exemplary volatile storage media include, but are not limited to, RAM (e.g., dynamic RAM).

[0107] Fig.11 An exemplary computing device 1100 is illustrated, which may be specifically configured to perform one or more processes described herein. Any system, unit, computing device, and / or other component described herein may be implemented by the computing device 1100 .

[0108] like Fig.11 As shown, computing device 1100 may include communication interface 1102, processor 1104, storage device 1106, and input / output ("I / O") module 1108, which are communicatively coupled to each other via communication infrastructure 1110. Fig.11 It is shown in Fig.11 The components illustrated in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.11 Components of computing device 1100 are shown in .

[0109] The communication interface 1102 may be configured to communicate with one or more computing devices. Examples of the communication interface 1102 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0110] The processor 1104 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the execution of one or more instructions, processes, and / or operations described herein. The processor 1104 may perform operations by executing computer-executable instructions 1112 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1106.

[0111] Storage device 1106 may include one or more data storage media, devices, or configurations and may take any type, form, and combination of data storage media and / or devices. For example, storage device 1106 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including data described herein) may be temporarily and / or permanently stored in storage device 1106. For example, data representing computer executable instructions 1112 configured to instruct processor 1104 to perform any of the operations described herein may be stored in storage device 1106. In some examples, the data may be arranged in one or more databases residing within storage device 1106.

[0112] The I / O module 1108 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1108 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, the I / O module 1108 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad (numeric keypad), a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0113] The I / O module 1108 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1108 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used for a particular implementation.

[0114] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings, however, it is apparent that various modifications and changes may be made thereto, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the claims that follow. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A system comprising: a memory storing instructions; and a processor communicatively coupled to the memory and configured to execute the instructions to: Instructing a display device to display an image, the image showing: a view of a surgical field associated with the patient captured by an imaging device, the view of the surgical field displaying surface anatomical structures located at the surgical field and objects located at the surgical field, and an augmented region displaying supplemental content, the augmented region creating an occlusion over at least a portion of the view of the surgical field; detecting an overlap in the image between at least a portion of the object and at least a portion of the enhanced region; as well as In response to detecting the overlap, the image is adjusted to reduce the degree of occlusion of the enhanced region within the overlap. 2 . The system of claim 1 , wherein adjusting the image to reduce the degree of occlusion of the enhanced region within the overlap comprises removing the enhanced region from the image. 3 . The system of claim 1 , wherein adjusting the image to reduce the degree of occlusion of the overlap enhancement region within the overlap comprises reducing an opacity of the enhancement region in the image.

4. The system of claim 1, wherein: The processor is further configured to determine that the object is in motion, and In response to determining that the object is in motion, adjusting the image to reduce the degree of occlusion of the enhanced region within the overlap is further performed. 5 . The system of claim 1 , wherein the supplemental content includes a representation of subsurface anatomical structure underlying a portion of the surface anatomical structure within the occlusion created by the augmented region.

6. The system of claim 5, wherein the representation of the subsurface anatomical structure comprises one or more of an ultrasound image of the subsurface anatomical structure, an image of a three-dimensional model of the subsurface anatomical structure, and a fluorescent image of the subsurface anatomical structure.

7. The system of claim 1, wherein the detection of the overlap comprises: generating a detection image, including an object representation map representing the object and an enhanced region representation map representing the enhanced region, and It is determined that the object representation contacts the enhanced region representation.

8. A method comprising: Instructing a display device by the composite medical imaging system to display an image, the image showing: a view of a surgical field associated with the patient captured by an imaging device, the view of the surgical field displaying surface anatomical structures located at the surgical field and objects located at the surgical field, and an augmented region displaying supplemental content, the augmented region creating an occlusion over at least a portion of the view of the surgical field; detecting, by the composite medical imaging system, an overlap between at least a portion of the object and at least a portion of the enhanced region in the image; as well as The image is adjusted, by the composite medical imaging system and in response to detecting the overlap, to reduce an extent of occlusion of the enhancement region within the overlap. 9 . The method of claim 8 , wherein adjusting the image to reduce the degree of occlusion of the enhanced region within the overlap comprises removing the enhanced region from the image.

10. The method of claim 8, wherein adjusting the image to reduce the degree of occlusion of the enhanced region within the overlap comprises reducing an opacity of the enhanced region in the image.

Images