Depth-based mixed reality image generation

Abstract

An illustrative system may be configured to: determine a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image. The combined image may be presented by instructing a display device to display a first portion of the virtual model positioned within the region of interest of the image and to avoid displaying a second portion of the virtual model positioned outside the region of interest of the image.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 450,240, filed on March 6, 2023, the contents of which are hereby incorporated by reference in their entirety. Background Art

[0003] During a medical procedure, such as a minimally invasive medical procedure, an imaging device (e.g., a stereoscopic imaging device) can be used to capture images of internal anatomical structures within a subject. In some cases, it may be desirable to visualize one or more portions of the internal anatomical structure that may be obscured from the field of view of the imaging device. For example, it may be desirable to visualize features of the internal anatomical structure depicted in the image that may be located below the surface of the internal anatomical structure (e.g., vascular system, etc.). In some scenarios, a virtual model that includes (i.e., depicts) one or more portions of the internal anatomical structure may be shown in combination with the captured image to provide visualization of one or more portions of the internal anatomical structure that may be obscured.

[0004] However, in some cases, the virtual model may include other portions of the internal anatomical structure in addition to the one or more portions of the internal anatomical structure that may be occluded. Consequently, the virtual model may obscure the view of the internal anatomical structure in the captured image. Furthermore, in some cases, the combination of the virtual model and the captured image may provide a poor depth perception of one or more portions of the internal anatomical structure between the virtual model and the captured image. Summary of the Invention

[0005] The following description presents a simplified summary of one or more aspects of the systems and methods described herein. This summary is not an extensive overview of all contemplated aspects and is neither intended to identify key or core elements of all aspects nor to define the scope of any or all aspects. Its purpose is to present one or more aspects of the systems and methods described herein as a prelude to the detailed description presented below.

[0006] An illustrative computer-assisted medical system includes a repositionable manipulator arm configured to be coupled to an imaging device, the imaging device configured to capture images depicting a scene including one or more anatomical objects positioned within a field of view of the imaging device, and a computing device communicatively coupled to the imaging device. The computing device may be configured to: determine a region of interest within the image; access a virtual model associated with the one or more anatomical objects; and present a combined image including the virtual model superimposed on the image by instructing a display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image and to avoid displaying a second portion of the virtual model outside the region of interest of the image within the combined image.

[0007] An illustrative system includes a memory storing instructions and one or more processors communicatively coupled to the memory. The one or more processors may be configured to execute the instructions to perform a process comprising: determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image. Presenting the combined image may include instructing a display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image, and to avoid displaying a second portion of the virtual model within the combined image that is positioned outside the region of interest of the image.

[0008] An illustrative method includes determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image. Presenting the combined image may include instructing a display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image and to avoid displaying a second portion of the virtual model within the combined image that is positioned outside the region of interest of the image.

[0009] An illustrative non-transitory computer-readable medium may store instructions that, when executed, instruct a processor of a computing device to perform a process comprising: determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image. Presenting the combined image may include instructing a display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image, and to avoid displaying a second portion of the virtual model within the combined image that is positioned outside the region of interest of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings illustrate various embodiments and are part of this specification. The embodiments shown are examples only and do not limit the scope of this disclosure. Throughout the drawings, the same or similar reference numerals represent the same or similar elements.

[0011] Figure 1 An illustrative computer-assisted medical system is shown.

[0012] Figure 2 An illustrative implementation including a mixed reality image generation system is shown.

[0013] Figure 3 An illustrative method of operating a mixed reality image generation system is shown.

[0014] Figures 4A to 4C Illustrative implementations of displays that may be generated using a mixed reality image generation system are shown.

[0015] Figure 5A and Figure 5B An illustrative implementation of determining a region of interest using a mixed reality image generation system is shown.

[0016] 6A to 6D Additional illustrative implementations of determining a region of interest using a mixed reality image generation system are shown.

[0017] Figure 7 An illustrative method of operating a mixed reality image generation system is shown.

[0018] Figure 8A and Figure 8B An illustrative implementation of adjusting visual characteristics using a mixed reality image generation system is shown.

[0019] Figure 9 An illustrative implementation of adjusting visual characteristics using a mixed reality image generation system is shown.

[0020] Figure 10A and Figure 10B An illustrative implementation of adjusting visual characteristics using a mixed reality image generation system is shown.

[0021] Figure 11 An illustrative computing system according to the principles described herein is shown. DETAILED DESCRIPTION

[0022] An illustrative system for generating a mixed reality image can be configured to restrict the overlay of a virtual model to a region of interest within the image. For example, the mixed reality image generation system can be configured to: determine a region of interest within an image depicting a scene including one or more anatomical objects; access a virtual model of the one or more anatomical objects; and present a combined image including the virtual model superimposed on the image by instructing a display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image and to avoid displaying a second portion of the virtual model within the combined image that is positioned outside the region of interest of the image.

[0023] In some implementations, one or more regions of interest can be determined in the image by the mixed reality image generation system, for example, based on the depth of the one or more anatomical objects. Additionally or alternatively, one or more visual characteristics (e.g., transparency, brightness, etc.) of the virtual model can be scaled by the mixed reality image generation system, for example, based on the depth of the one or more anatomical objects.

[0024] The principles described herein can produce improved mixed reality images, as well as provide other benefits as described herein, compared to conventional techniques that do not restrict the superposition of virtual models to regions of interest. For example, restricting the superposition of virtual models to regions of interest can enable a clearer depiction of a scene, such as by depicting one or more features of an anatomical object that can be included in the virtual model within the region of interest (e.g., below the surface of the anatomical object) without obscuring other portions of the scene depicted in the image that are outside the region of interest. This can enable faster and / or easier performance of medical procedures associated with the anatomical object. Additionally, restricting the superposition of virtual models to regions of interest can improve depth perception of the anatomical object between the virtual model and the scene depicted in the image.

[0025] In some examples, one or more components of the system for generating mixed reality images may be implemented by a computer-assisted medical system. Figure 1 An illustrative computer-assisted medical system 100 is shown that may be used to perform various types of medical procedures, including surgical and / or non-surgical procedures.

[0026] As shown, the computer-assisted medical system 100 may include a manipulator assembly 102 ( Figure 110 ), a manipulator cart 104, and an auxiliary device 106, all of which are communicatively coupled to one another. The computer-assisted medical system 100 can be used by a medical team to perform a computer-assisted medical procedure or other similar operation on the body of a patient 108, or on any other body as may serve a particular implementation. As shown, the medical team may include a first user 110-1 (e.g., a surgeon for a surgical procedure), a second user 110-2 (e.g., a patient-side assistant), a third user 110-3 (e.g., another assistant, a nurse, a trainee, etc.), and a fourth user 110-4 (e.g., an anesthesiologist for a surgical procedure), all of which are collectively referred to as users 110, and each user may control, interact with, or otherwise be a user of the computer-assisted medical system 100. There may be more, fewer, or alternative users during a medical procedure, as may serve a particular implementation. For example, the team composition for different medical or non-medical procedures may be different and include users with different roles.

[0027] Although Figure 1 A minimally invasive medical procedure, such as a minimally invasive surgical procedure, is shown being performed, but it will be understood that the computer-assisted medical system 100 can similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations can also be performed.

[0028] like Figure 1 As shown, the manipulator assembly 102 may include one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments may be used for computer-assisted medical procedures performed on a patient 108 (e.g., in the surgical example, by being at least partially inserted into and manipulated within the patient 108). Although the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, it will be appreciated that the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although Figure 1 The examples of FIGURE 1 illustrate the manipulator arm 112 as a robotic manipulator arm, but it will be understood that in some examples, one or more instruments may be partially or fully manually controlled, such as by being held and manually controlled by a person. For example, these partially or fully manually controlled instruments may be coupled to a Figure 1 The computer-assisted instrumentation of the manipulator arm 112 is shown for use in conjunction with or as an alternative thereto.

[0029] During a medical procedure, the user control device 104 can be configured to facilitate teleoperation control of the manipulator arm 112 and instruments attached to the manipulator arm 112 by the user 110-1. To this end, the user control device 104 can provide the user 110-1 with imagery of the operating area associated with the patient 108, as captured by an imaging device. The manipulator arm 112 or any instruments coupled to the manipulator arm 112 can mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this way, the user 110-1 can intuitively perform procedures using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arm 112 in order to perform one or more surgical procedures (e.g., an incision procedure, a suturing procedure, etc.).

[0030] The auxiliary device 106 may include one or more computing devices configured to perform auxiliary functions to support the medical procedure, such as providing insufflation, electrocautery energy, lighting or other energy to an imaging device, image processing, or coordinating components of the computer-assisted medical system 100. In some examples, the auxiliary device 106 may be configured with a display monitor 114 configured to display one or more user interfaces, or graphical or textual information to support the medical procedure. In some cases, the display monitor 114 may be implemented by a touch screen display and provide user input functionality. The augmented content provided by the region-based augmentation system may be similar to or different from the content associated with the display monitor 114 or one or more display devices in the operating area (not shown).

[0031] The manipulator assembly 102, the user control device 104, and the auxiliary device 106 may be communicatively coupled to one another in any suitable manner. Figure 1 As shown, the manipulator assembly 102, the user control device 104, and the auxiliary device 106 can be communicatively coupled via a control line 116, which can represent any wired or wireless communication link that can serve a particular implementation. To this end, the manipulator assembly 102, the user control device 104, and the auxiliary device 106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0032] Figure 2An illustrative implementation 200 configured to generate a mixed reality image is shown. As shown, the implementation 200 includes a mixed reality image generation system 202 in communication with an imaging device 204 and a user interface system 206. The implementation 200 may include additional or alternative components as may be useful for a particular implementation. In some examples, the implementation 200 or certain components of the implementation 200 may be implemented by a computer-assisted medical system, such as the computer-assisted medical system 100 discussed above.

[0033] The mixed reality image generation system 202 can be implemented by one or more computing devices and / or computer resources (e.g., processors, memory devices, storage devices, etc.) that can serve a specific implementation. As shown, the mixed reality image generation system 202 can include, but is not limited to, a memory 208 and a processor 210 selectively and communicatively coupled to each other. The memory 208 and the processor 210 can each include or be implemented by computer hardware configured to store and / or process computer software. Figure 2 Various other components of computer hardware and / or software not explicitly shown may also be included within mixed reality image generation system 202. In some examples, memory 208 and / or processor 210 may be distributed among multiple devices and / or multiple locations as may serve a particular implementation.

[0034] The memory 208 may store and / or otherwise hold executable data used by the processor 210 to perform any of the functions described herein. For example, the memory 208 may store instructions 212 that may be executed by the processor 210. The memory 208 may be implemented by one or more memories or storage devices, including any memory or storage device described herein, that are configured to store data in a transient or non-transient manner. The instructions 212 may be executed by the processor 210 to cause the mixed reality image generation system 202 to perform any of the functions described herein. The instructions 212 may be implemented by any suitable application, software, code, and / or other executable data instance. Additionally, the memory 208 may also hold any other data that is accessed, managed, used, and / or transmitted by the processor 210 in a particular implementation.

[0035] The processor 210 may be implemented by one or more computer processing devices, including a general-purpose processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, etc.), a special-purpose processor (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), an image signal processor, etc. Using the processor 210 (e.g., when the processor 210 is instructed to execute operations represented by instructions 212 stored in the memory 208), the mixed reality image generation system 202 may perform various operations as described herein.

[0036] The imaging device 204 can be implemented by a stereoscopic imaging device or other suitable device configured to capture and output one or more images (e.g., images, videos, sequences of image frames, etc.) depicting the scene 214. In some implementations, the imaging device 204 can include, but is not limited to, one or more of the following: a video imaging device, an infrared imaging device, a visible light imaging device, a non-visible light imaging device, an intensity imaging device (e.g., a color imaging device, a grayscale imaging device, a black and white imaging device), a depth imaging device (e.g., a stereoscopic imaging device, a time-of-flight imaging device, an infrared imaging device, a red-green-blue (RGB) imaging device, a red-green-blue and depth (RGB-D) imaging device, a light detection and ranging (LIDAR) imaging device, etc.).

[0037] In some implementations, images captured by imaging device 204 can include image data (e.g., color, grayscale, saturation, intensity, brightness, depth, etc.). In some cases, the image data can be associated with data points expressed in a common coordinate system, such as 3D voxels or two-dimensional (2D) pixels of the image captured by imaging device 204. In some implementations, imaging device 204 can move relative to scene 214 to capture one or more images of scene 214 at different viewpoints.

[0038] Scene 214 may include an environment (e.g., an area within a subject undergoing a medical procedure) and / or one or more objects within the environment. In some examples, scene 214 may include a surgical area associated with a body on or within which a medical procedure is performed (e.g., the body of a living animal, a human or animal cadaver, a portion of a human or animal anatomical structure, tissue removed from a human or animal anatomical structure, a non-tissue artifact, a training model, etc.). For example, scene 214 may include one or more anatomical objects 216 positioned within the field of view of imaging device 204. Anatomical objects 216 may include objects associated with a subject (e.g., the body of a living animal, a human or animal cadaver, a portion of a human or animal anatomical structure, tissue removed from a human or animal anatomical structure, a non-tissue artifact, a training model, etc.). In some implementations, anatomical objects 216 may include tissue of the subject (e.g., an organ, soft tissue, connective tissue, etc.). Scene 214 may also include other non-anatomical objects, such as physical tools used in the medical procedure (e.g., a scalpel, scissors, forceps, clamps, etc.) and / or other objects (e.g., staples, mesh, sponges, etc.). In some embodiments, scene 2014 may include a portion of an anatomical object rather than its entirety.

[0039] In some implementations, a repositionable manipulator arm (e.g., the manipulator arm 112 of the computer-assisted medical system 100) can be coupled to the imaging device 204. For example, the repositionable manipulator arm can be movable to position the field of view of the imaging device 204 to include the anatomical object 216. In some implementations, an additional repositionable manipulator arm (e.g., the manipulator arm 112 of the computer-assisted medical system 100) can be coupled to the instrument such that the additional repositionable manipulator arm can be movable to position the instrument within the field of view of the imaging device 204 (e.g., to manipulate the anatomical object 216).

[0040] The user interface system 206 of the illustrated implementation includes a display device 218 and a user input device 220. The display device 218 can be implemented by a monitor or other suitable device configured to display information to a user. For example, the display device 218 can be configured to display one or more images captured by the imaging device 204 and / or a mixed reality image generated by the mixed reality image generation system 202. The user input device 220 can be implemented by any suitable one or more devices (e.g., buttons, joysticks, touch screens, keyboards, handles, microphones, etc.) that are configured to receive user input, such as to interact with a display presented by the display device 218 and / or to cause movement of one or more repositionable manipulator arms.

[0041] Figure 3An illustrative method 300 is shown that may be performed by the mixed reality image generation system 202. Figure 3 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Figure 3 In addition, Figure 3 Each of the operations depicted in can be performed in any of the ways described herein.

[0042] As shown, the mixed reality image generation system 202 can determine a region of interest within the image at operation 302. In some implementations, the mixed reality image generation system 202 can determine the region of interest within the image by identifying a region (e.g., the region of interest) within at least a portion of the image. For example, the region of interest can include a region in the image that depicts one or more anatomical objects (e.g., anatomical object 216) of interest (e.g., associated with a medical procedure). To illustrate, the region of interest can include at least a portion of one or more anatomical objects depicted in the image (e.g., the region of interest can include the entirety of a single anatomical object, the entirety of multiple anatomical objects, an anatomical object positioned within another anatomical object, a feature of an anatomical object, a portion of an anatomical object, a portion of multiple anatomical objects, etc.). In some implementations, the region of interest can include one or more anatomical objects associated with the medical procedure such that it may be desirable to visualize features of one or more anatomical objects (e.g., below a surface depicted in the image). Additionally, in some cases, the region of interest can have a specified shape (e.g., a circle, a square, a freeform shape, etc.).

[0043] In some implementations, the mixed reality image generation system 202 can determine the region of interest by detecting user input specifying the region of interest. For example, a user can interact with the display of an image (e.g., using the user input device 220) to specify a region of interest in the image.

[0044] Additionally or alternatively, the mixed reality image generation system 202 may determine the region of interest by identifying one or more anatomical objects in the image and designating a region in the image that depicts at least a portion of the one or more anatomical objects as the region of interest. For example, the mixed reality image generation system 202 may implement and apply an artificial intelligence algorithm, such as a machine learning algorithm, to identify the one or more anatomical objects in the image and / or designate the region of interest as the region in the image that depicts at least a portion of the one or more anatomical objects. To illustrate, a machine learning algorithm may be used to identify one or more anatomical objects and / or features of one or more anatomical objects in the image, such that the mixed reality image generation system 202 may determine a region of interest associated with the identified anatomical objects and / or features. Any suitable form of artificial intelligence and / or machine learning may be used, including, for example, deep learning, neural networks, etc. For example, a machine learning algorithm may be generated by a machine learning process and applied to a recognition operation. In some implementations, the machine learning algorithm may be instructed to identify one or more anatomical objects and / or features of one or more anatomical objects within the image. The machine learning algorithm may operate as a recognition function that is applied to individual and / or fused images to classify the one or more anatomical objects in the image.

[0045] In addition to or in lieu of a machine learning algorithm, other suitable methods can be used to identify one or more anatomical objects to determine a region of interest. For example, the mixed reality image generation system 202 can be configured to determine a region of interest by implementing and applying an object recognition algorithm. For example, the object recognition algorithm can be used to identify a predetermined type of object (e.g., anatomical object 216) within an image, such as by comparing image data of the image with model object data of the predetermined type of object. Such model object data can be stored in a model database that can be communicatively coupled to the mixed reality image generation system 202.

[0046] In some implementations, determining the region of interest may further include accessing an image. For example, the mixed reality image generation system 202 may access an image depicting one or more anatomical objects (e.g., captured by the imaging device 204) in any suitable manner. For example, the mixed reality image generation system 202 may access data representing the image directly from a computing device storing the image, directly from an imaging device configured to capture the image (e.g., the imaging device 204), or the like, via one or more networks (e.g., a local area network, the Internet, etc.).

[0047] In some implementations, determining the region of interest can also include processing the image. For example, the mixed reality image generation system 202 can be configured to fuse or otherwise combine the image with another image depicting one or more anatomical objects, such as by stitching together non-overlapping voxels or pixels (e.g., stitching the images together along non-overlapping boundaries of the images), merging aligned and / or overlapping voxels or pixels (e.g., blending the intensity and / or depth values ​​of the aligned voxels or pixels), etc.

[0048] The mixed reality image generation system 202 may also access a virtual model of one or more anatomical objects at operation 304. For example, the virtual model may include one or more anatomical objects depicted within a region of interest in the image. In some implementations, the virtual model may represent one or more anatomical objects and / or features of one or more anatomical objects (e.g., beneath a surface depicted in the image) in one or more forms (e.g., solid, wireframe, surface, etc.). The mixed reality image generation system 202 may access the virtual model in any suitable manner. For example, the mixed reality image generation system 202 may access data representing the virtual model directly from a computing device storing the virtual model via one or more networks (e.g., a local area network, the Internet, etc.). Additionally or alternatively, the data representing the virtual model may be stored by the mixed reality image generation system 202 (e.g., by memory 208).

[0049] The mixed reality image generation system 202 may also present a combined image (e.g., a mixed reality image) including the virtual model superimposed on the image at operation 306. For example, presenting the combined image may include instructing a display device (e.g., display device 218) to display the combined image. In some implementations, the virtual model may be restricted to a region of interest, such that the mixed reality image generation system 202 may instruct the display device to display a first portion of the virtual model within the combined image that is positioned within the region of interest of the image at operation 308, and instruct the display device to avoid displaying a second portion of the virtual model within the combined image that is positioned outside the region of interest of the image at operation 310. To illustrate, the mixed reality image generation system 202 may superimpose the virtual model on pixels or voxels of the image identified as being within the region of interest, and avoid superimposing the virtual model on pixels or voxels of the image identified as being outside the region of interest.

[0050] In some implementations, the virtual model can be repositioned in the combined image. For example, a user can interact with the display of the combined image (e.g., using the user input device 220) to reposition the virtual model relative to the region of interest. The mixed reality image generation system 202 can detect that the virtual model has been repositioned, so that presenting the combined image can include detecting that a first portion of the virtual model has been repositioned outside the region of interest and, in response, avoiding displaying the first portion. Additionally or alternatively, presenting the combined image can include detecting that a second portion of the virtual model has been repositioned within the region of interest and, in response, displaying the second portion.

[0051] As an illustrative example, Figures 4A to 4C 2 shows an implementation of a display that can be generated using the mixed reality image generation system 202. As shown, Figure 4A An illustrative implementation 400 of a display depicting an image 402 (e.g., captured by the imaging device 204) of the scene 214 including the anatomical object 216 (e.g., a kidney) is shown. The image 402 also includes a region of interest 404 (e.g., determined by the mixed reality image generation system 202 in any of the manners described herein). As shown, the region of interest 404 encompasses a portion of the anatomical object 216. For example, it may be desirable to visualize features of the anatomical object 216 and / or other anatomical objects beneath the outer surface of the anatomical object 216 within the portion of the anatomical object 216 indicated by the region of interest 404 (e.g., to perform a medical procedure).

[0052] Figure 4B An illustrative implementation 406 of displaying a virtual model 408 (e.g., accessed by the mixed reality image generation system 202) in conjunction with the image 402 is shown. For example, the virtual model 408 can include a 2D or 3D model of the anatomical object 216 depicted in the image 402. In some implementations, the virtual model 408 can depict features of the anatomical object 216 depicted in the image 402 and / or other anatomical objects beneath the outer surface of the anatomical object 216 (e.g., vasculature, etc.). Additionally, in some cases, the virtual model 408 can include other features and / or anatomical objects in addition to the anatomical object 216 depicted in the image 402.

[0053] In some implementations, the virtual model 408 can be generated, for example, based on preoperative imaging of the anatomical object 216. For example, the virtual model 408 can be generated based on a 3D image representation of the anatomical object 216 from a scanning system (e.g., computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, etc.). The 3D image representation of the anatomical object 216 can include image data (e.g., pixels or voxels) arranged in a 3D grid configuration. The virtual model 408 can be derived from the 3D image representation of the anatomical object 216, for example, by generating vertices of the virtual model 408 arranged in the 3D grid configuration and associating the vertices with 3D positions corresponding to positions of the image data within the 3D image representation of the anatomical object 216.

[0054] In some implementations, the virtual model 408 can be sized and / or oriented to correspond to the anatomical object 216 depicted in the image 402. To illustrate, a user can interact with the display of the image 402 (e.g., using the user input device 220) to position the virtual model 408 within the image 402. As shown, the virtual model 408 includes a first portion 410-1 positioned within the region of interest 404 and a second portion 410-2 extending beyond the region of interest 404. In the example shown, the second portion 410-2 of the virtual model 408 can occlude portions of the image 402 positioned outside the region of interest 404.

[0055] Figure 4C An illustrative implementation 412 is shown in which the display of the virtual model 408 is restricted to the region of interest 404 in the image 402. For example, the mixed reality image generation system 202 can instruct the display device to display a first portion 410-1 of the virtual model 408 that is within the region of interest 404 and to avoid displaying a second portion 410-2 of the virtual model 408 that is outside the region of interest 404. This can enable the virtual model 408 to depict features of the anatomical object 216 within the region of interest 404 and / or other anatomical objects beneath the outer surface of the anatomical object 216 while not obscuring the image 402 outside the region of interest 404. In some implementations, displaying the first portion 410-1 of the virtual model 408 can include displaying a boundary (e.g., an outline) of the region of interest 404 to, for example, indicate the area of ​​the first portion 410-1 of the virtual model 408 that is being displayed. In the event that the virtual model 408 is repositioned in the image 402, the mixed reality image generation system 202 can avoid displaying an area of ​​the first portion 410-1 of the virtual model 408 that is repositioned outside the area of ​​interest 404 and / or displaying an area of ​​the second portion 410-2 of the virtual model 408 that is repositioned within the area of ​​interest 404.

[0056] The mixed reality image generation system 202 may use various techniques to determine the region of interest 404. For example, Figure 5A An illustrative implementation 500 of determining a region of interest based on the depth of an anatomical object 216, for example, relative to an imaging device 204 (e.g., based on the physical depth of the physical anatomical object 216 and / or based on the virtual depth of a virtual anatomical object 216 depicted in an image) is shown. As shown, the anatomical object 216 includes an outer surface 502 that can be positioned at various depths relative to a distal end 504 (e.g., an end positioned toward the anatomical object 216) of the imaging device 204. The depth d of the anatomical object 216 in an image (e.g., image 402) can be determined by accessing and / or generating a depth map of a scene (e.g., scene 214) captured by the imaging device 204 that includes the anatomical object 216. In some implementations, the depth map can be generated, for example, by processing stereo images (e.g., using a machine learning algorithm), using a simultaneous localization and mapping (SLAM) algorithm, and / or by a depth sensor (e.g., a time-of-flight sensor) associated with the imaging device 204. The mixed reality image generation system 202 may also identify anatomical objects 216 in the scene (eg, using machine learning algorithms, object recognition, etc.) in order to generate a depth map.

[0057] To illustrate, the mixed reality image generation system 202 can generate a point cloud in an image having a plurality of nodes representing surface points 506 (e.g., surface points 506-1 to 506-2) on the outer surface 502 of the anatomical object 216. In some implementations, each node can be associated with a region in the image that includes one or more pixels or voxels. For illustrative purposes, Figure 5A A first surface point 506-1 that can be associated with a first node in the image and a second surface point 506-2 that can be associated with a second node in the image are shown. Additionally or alternatively, the nodes can be associated with one or more other points (e.g., a center of mass, a point closest to the imaging device 204, a point farthest from the imaging device 204, etc.) of one or more anatomical objects 216 depicted in the image.

[0058] The mixed reality image generation system 202 may determine a depth value associated with each node representing a depth d (e.g., depths d1 through d3) of a corresponding surface point 506 relative to the distal end 504 of the imaging device 204. For example, a first node may be associated with a first depth value representing a first depth d1 of a first surface point 506-1 relative to the distal end 504 of the imaging device 204, and a second node may be associated with a second depth value representing a second depth d2 of a second surface point 506-2 relative to the distal end 504 of the imaging device 204. The depth values ​​may be represented by any suitable metric, such as a discrete value (e.g., a distance, a ratio, a percentage, etc.).

[0059] The mixed reality image generation system 202 can determine a region of interest in an image based on the depth values. For example, the mixed reality image generation system 202 can determine whether the depth value associated with each node in the image is below a depth threshold, such that the region of interest can include areas where at least some of the depth values ​​are below the depth threshold. For example, a depth value below the depth threshold can indicate that the node associated with the depth value corresponds to a portion of the anatomical object 216 that can be positioned proximate to the imaging device 204, such that it may be desirable for a user to visualize the portion of the anatomical object 216 (e.g., to perform a medical procedure). To illustrate, the first depth d1 of the first surface point 506-1 is less than the second depth d2 of the second surface point 506-2, such that the first depth value associated with the first depth d1 can be below the depth threshold, while the second depth value associated with the second depth d2 can meet or exceed the depth threshold. Therefore, the mixed reality image generation system 202 can determine the region of interest to include at least the first node associated with the first depth value below the depth threshold.

[0060] In some implementations, a region of interest can be determined based on a depth value associated with each node in the image. For example, the region of interest can be sized to include at least some of the nodes in the image that are associated with depth values ​​below a depth threshold and / or an area (e.g., a shape) surrounding at least some of the nodes. Additionally or alternatively, the region of interest can be determined based on a combination of depth values ​​associated with multiple nodes (e.g., an average, mean, median, etc.), such that a single depth value can be associated with multiple nodes.

[0061] The mixed reality image generation system 202 can also be configured to determine a single region of interest and / or multiple regions of interest in the image based on the depth value. For example, a single region of interest can be determined to include at least some of the nodes associated with a depth value below a depth threshold, such as when the nodes are within a predetermined distance of each other in the image. Additionally or alternatively, multiple regions of interest can be determined to include at least some of the nodes associated with a depth value below a depth value, such as when the nodes are outside a predetermined distance of each other in the image. One or more regions of interest can include at least a portion of one or more anatomical objects 216 depicted in the image.

[0062] In some implementations, the mixed reality image generation system 202 can update the depth value and / or region of interest based on the movement of the anatomical object 216 (e.g., during a medical procedure). For example, the mixed reality image generation system 202 can track (e.g., using a SLAM algorithm) the surface point 506 of the anatomical object 216 so that the mixed reality image generation system 202 can update the position of a node associated with the surface point 506 in the image based on the movement of the surface point 506. The mixed reality image generation system 202 can also update the depth value associated with the updated position of the node and determine whether the updated depth value is below a depth threshold. If some of the updated depth values ​​cross (e.g., fall below and / or meet or exceed) the depth threshold, the mixed reality image generation system 202 can update the region of interest based on the crossing of the updated depth values. Additionally or alternatively, the depth value and / or region of interest can be fixed for at least a period of time.

[0063] In some cases, one or more portions of the anatomical object 216 may be occluded in the field of view of the imaging device 204, such that the mixed reality image generation system 202 may be configured to interpolate depth values ​​for one or more nodes that may be associated with one or more surface points 506 on the potentially occluded portion of the anatomical object 216. In such cases, the mixed reality image generation system 202 may be configured to associate a confidence value with each depth value that represents a confidence level for the depth value. The confidence value may be represented by any suitable metric, such as a discrete value (e.g., a level, a ratio, a percentage, etc.).

[0064] Based on the confidence value, the mixed reality image generation system 202 can be configured to determine a measurable area in the image, for example, by determining an area containing at least some nodes associated with depth values ​​having confidence values ​​that meet or exceed a confidence threshold. For example, a confidence value that meets or exceeds the confidence threshold can indicate that the depth value of the corresponding node can be sufficiently accurate within the measurable area. Therefore, the mixed reality image generation system 202 can determine the region of interest based on the depth values ​​of the nodes located within the measurable area.

[0065] Figure 5BAnother illustrative implementation 508 of determining a region of interest based on the depth of an anatomical object 216, for example, relative to a tissue surface 510 (e.g., the surface of a skin layer, the surface of another anatomical object, etc.) is shown. To illustrate, the mixed reality image generation system 202 can determine a depth value associated with one or more nodes in the image that represents a depth D (e.g., depth D1 to D2) of a surface point 506 on the anatomical object 216 relative to the tissue surface 510. For example, a first node can have a first depth value representing a first depth D1 of a first surface point 506-1 relative to the tissue surface 510, and a second node can have a second depth value representing a second depth D2 of a second surface point 506-2 relative to the tissue surface 510. The depth values ​​can be represented by any suitable metric, such as a discrete value (e.g., a distance, a ratio, a percentage, etc.).

[0066] In some implementations, the depth value can be based on a depth D having any suitable orientation between the surface point 506 and the tissue surface 510 (e.g., an orientation perpendicular to the outer surface 502 of the anatomical object 216 at the surface point 506 and / or the tissue surface 510, an orientation that forms the shortest distance between the surface point 506 and the tissue surface 510, an orientation that forms the greatest distance between the surface point 506 and the tissue surface 510, etc.). In addition to or in lieu of the surface point 506, the depth value can be associated with one or more other points (e.g., a center of mass, etc.) of one or more anatomical objects 216.

[0067] The mixed reality image generation system 202 can determine a region of interest based on the depth value. For example, the mixed reality image generation system 202 can determine whether the depth value associated with the node is below a depth threshold, such that the region of interest can include an area in which at least some of the depth values ​​are below the depth threshold. For example, a depth value below the depth threshold can indicate that the node associated with the depth value corresponds to a portion of the anatomical object 216 located near the tissue surface 510, such that it may be desirable for the user to visualize the portion of the anatomical object 216 (e.g., to perform a medical procedure).

[0068] To illustrate, the first depth D1 of the first surface point 506-1 is less than the second depth D2 of the second surface point 506-2, such that a first depth value associated with the first depth D1 may be below the depth threshold, while a second depth value associated with the second depth D2 may meet or exceed the depth threshold. Thus, the mixed reality image generation system 202 may determine the region of interest to include at least a first node associated with a first depth value below the depth threshold. In some implementations, the mixed reality image generation system 202 may determine one or more regions of interest to include at least some of the nodes associated with depth values ​​below the depth threshold, such that the one or more regions of interest include at least a portion of one or more anatomical objects 216 depicted in the image.

[0069] Figure 6A An illustrative implementation 600 of a region of interest 404 is shown that determines the size of one or more anatomical objects 216 depicted in an image (e.g., image 402). For example, the mixed reality image generation system 202 can determine the size of the one or more anatomical objects 216, such as by identifying (e.g., using a machine learning algorithm, object recognition, etc.) one or more anatomical objects 216 and / or one or more features of one or more anatomical objects 216 in the scene of the image and associating the identified anatomical objects 216 and / or features with pixels or voxels of the image. The mixed reality image generation system 202 can also determine the region of interest 404 based on the size of the identified anatomical objects 216 and / or features, such as by sizing the region of interest 404 to include all and / or a portion of the pixels or voxels associated with the identified anatomical objects 216 and / or features. To illustrate, Figure 6A The region of interest 404 is shown corresponding to the dimensions of the anatomical object 216. Although the implementation 600 shows the region of interest 404 as including the boundaries of the anatomical object 216, the region of interest 404 may additionally or alternatively include a shape (e.g., a circle, a square, etc.) that encompasses the boundaries of one or more anatomical objects 216.

[0070] Figure 6BAn illustrative implementation 602 of a region of interest 404 is shown that is determined based on points 604 associated with one or more anatomical objects 216 in an image. For example, the mixed reality image generation system 202 can identify a point 604 associated with an anatomical object 216 (e.g., a center of mass, a point closest to the imaging device 204 and / or the tissue surface 510, a point farthest from the imaging device 204 and / or the tissue surface 510, etc.). Although the implementation 602 shows a point 604 associated with a single anatomical object 216, the point 604 can additionally or alternatively be associated with multiple anatomical objects 216. For illustration, the point 604 can include the center of mass of multiple anatomical objects 216. In some implementations, the point 604 can be identified by detecting user input specifying the point 604 within the image (e.g., using the user input device 220) and / or using a machine learning algorithm.

[0071] The region of interest 404 can be determined to include a boundary that is spaced a selected distance R from a point 604 in the image. As shown, the region of interest 404 includes a circle having a radius of the selected distance R around the point 604, such that the selected distance R is continuous around the point 604. Additionally or alternatively, the selected distance R can vary around the point 604, such that other suitable shapes (e.g., a square, a triangle, a freeform shape, etc.) can be used to form the boundary of the region of interest 404 relative to the point 604. In some implementations, the selected distance R of the boundary of the region of interest 404 can be adjustable (e.g., based on user input).

[0072] Figure 6C An illustrative implementation 606 of multiple regions of interest 404 (e.g., regions of interest 404-1 to 404-2) determined based on multiple anatomical objects 216 (e.g., anatomical objects 216-1 to 216-3) depicted in an image is shown. As shown, a first anatomical object 216-1 and a second anatomical object 216-2 are positioned within a third anatomical object 216-3. Additionally, a first point 604-1 is associated with the first anatomical object 216-1, and a second point 604-2 is associated with the second anatomical object 216-2. The multiple regions of interest 404 can be determined to include a first region of interest 404-1 having a first boundary that includes the first point 604-1 associated with the first anatomical object 216-1 and a second region of interest 404-2 having a second boundary that includes the second point 604-2 associated with the second anatomical object 216-2. In some implementations, the boundaries of each region of interest 404 can be based on the size of the corresponding anatomical object 216, the size of a feature of the corresponding anatomical object 216, a selected distance from the corresponding point 604, and the like.

[0073] Figure 6DAn illustrative implementation 608 of a region of interest 404 is shown that is determined based on a plurality of points 604 associated with one or more anatomical objects 216. For example, the region of interest 404 can be determined to include a boundary that encompasses both a first point 604-1 associated with a first anatomical object 216-1 and a second point 604-2 associated with a second anatomical object 216-2 in the image. Although the implementation 608 shows each point 604 associated with a different anatomical object 216, the points 604 can additionally or alternatively be associated with the same anatomical object 216 (e.g., the points 604 can be associated with different features of the same anatomical object 216). In some implementations, the boundary of the region of interest 404 can be based on the size of the one or more anatomical objects 216, the size of one or more features of the one or more anatomical objects 216, a selected distance away from the one or more points 604, and the like.

[0074] Other suitable techniques for determining the region of interest 404 may also be used. For example, the region of interest 404 may be determined based on information associated with the medical procedure. To illustrate, the information may indicate one or more target anatomical objects 216 associated with the medical procedure, such that the mixed reality image generation system 202 may identify the target anatomical objects 216 in the image and determine the region of interest 404 to include the target anatomical objects 216.

[0075] Figure 7 Another illustrative method 700 that may be performed by the mixed reality image generation system 202 is shown. Figure 7 Exemplary operations according to one embodiment are shown, but other embodiments may omit, add to, reorder, and / or modify Figure 7 In addition, Figure 7 Each of the operations depicted in can be performed in any of the ways described herein.

[0076] As shown, the mixed reality image generation system 202 can determine a region of interest (e.g., region of interest 404) within the image at operation 702. For example, the image can depict a scene including one or more anatomical objects (e.g., anatomical object 216), such that the region of interest can be determined to include at least a portion of the one or more anatomical objects in the scene. In some implementations, determining the region of interest can be based on a depth of the one or more anatomical objects (e.g., relative to the imaging device 204 and / or tissue surface 510), a size of the one or more anatomical objects, one or more points associated with the one or more anatomical objects (e.g., point 604), detecting user input specifying the region of interest, etc.

[0077] The mixed reality image generation system 202 may also access a virtual model of one or more anatomical objects (e.g., the virtual model 408) at operation 704 and present a combined image including the virtual model superimposed on the image at operation 706. The mixed reality image generation system 202 may also determine whether the virtual model is positioned within the region of interest at operation 708. For example, if a portion of the virtual model is not positioned within the region of interest (no at operation 708), the mixed reality image generation system 202 may instruct the display device to avoid displaying the portion of the virtual model outside the region of interest at operation 710. Alternatively, if a portion of the virtual model is positioned within the region of interest (yes at operation 708), the mixed reality image generation system 202 may instruct the display device to display the portion of the virtual model positioned within the region of interest at operation 712.

[0078] In some implementations, the mixed reality image generation system 202 may also adjust visual characteristics (e.g., transparency, brightness, etc.) of the portion of the virtual model positioned within the region of interest at operation 714. For example, one or more visual characteristics of the virtual model may be adjusted based on detecting user input specifying the visual characteristics (e.g., using the user input device 220) and / or using a machine learning algorithm. To illustrate, the one or more visual characteristics may be adjusted based on the depth of one or more anatomical objects (e.g., relative to the imaging device 204 and / or the tissue surface 510), the size of one or more anatomical objects, one or more points associated with the one or more anatomical objects (e.g., point 604), etc. In some implementations, adjusting the one or more visual characteristics of the virtual model may improve the depth perception between the virtual model and the rest of the image.

[0079] As an example, Figure 8A An illustrative implementation 800 of a virtual model 802 is shown that can be displayed within a region of interest in an image and adjusted based on a depth relative to the imaging device 204. The virtual model 802 can implement or be similar to the virtual model 408. As shown, the virtual model 802 includes a first anatomical object 216-1 and a second anatomical object 216-2. In some implementations, the anatomical objects 216 of the virtual model 802 can be sized and / or oriented in the image (e.g., based on user input) to align the virtual model 802 with corresponding anatomical objects 216 depicted in the image (e.g., as captured by the imaging device 204). For example, a first outer surface 804-1 of the first anatomical object 216-1 of the virtual model 802 can be aligned with an outer surface (e.g., outer surface 502) of the corresponding anatomical object 216 in the image, and / or a second outer surface 804-2 of the second anatomical object 216-2 of the virtual model 802 can be aligned with an outer surface of the corresponding anatomical object 216 in the image.

[0080] In some implementations, the anatomical objects 216 of the virtual model 802 can be registered with corresponding anatomical objects 216 in the image, for example, by associating vertices 806 (e.g., vertices 806-1 through 806-1) of the virtual model 802 with corresponding nodes representing surface points (e.g., surface points 506) on the anatomical objects 216 depicted in the image. This can enable depth values ​​associated with the nodes of the anatomical objects 216 depicted in the image to be further associated with the corresponding vertices 806 of the virtual model 802. For example, a first vertex 806-1 of the virtual model 802 located on a first outer surface 804-1 of the first anatomical object 216-1 can be associated with a first depth value of the first node representing a first depth d1 relative to the distal end 504 of the imaging device 204. Similarly, a second vertex 806-2 of the virtual model 802 located on a second outer surface 804-2 of the second anatomical object 216-2 can be associated with a second depth value of the second node representing a second depth d1 relative to the distal end 504 of the imaging device 204.

[0081] Based on the depth value, the mixed reality image generation system 202 can be configured to adjust visual characteristics within the region of interest of the virtual model 802. For example, the mixed reality image generation system 202 can adjust the visual characteristics so that a display device (e.g., the display device 218) displays portions of the virtual model 802 associated with a lower depth value than other portions of the virtual model 802 as less transparent than other portions of the virtual model 802 (e.g., the transparency of the virtual model 802 can decrease as the depth value decreases). For example, a lower depth value can indicate that the portion of the virtual model 802 can correspond to an anatomical object 216 located closer to the imaging device 204, such that it may be desirable for the user to more clearly visualize the anatomical object 216 located closer to the imaging device 204 than the anatomical object 216 located farther away from the imaging device 204. To illustrate, a first depth d1 associated with a first vertex 806-1 of the virtual model 802 is less than a second depth d2 associated with a second vertex 806-2 of the virtual model 802, such that the first anatomical object 216-1 associated with the first vertex 806-1 is shown as more opaque than the second anatomical object 216-2 associated with the second vertex 806-2.

[0082] In some implementations, the mixed reality image generation system 202 can be configured to adjust the visual characteristics of the virtual model 802 based on relative depth values, such that portions of the virtual model 802 associated with depth values ​​lower than depth values ​​of other portions of the virtual model 802 can be shown as less transparent. Additionally or alternatively, the visual characteristics can be adjusted based on absolute depth values, such that portions of the virtual model 802 associated with depth values ​​below a depth threshold can be shown as less transparent, while portions of the virtual model 802 associated with depth values ​​that meet or exceed the depth threshold can be shown as more transparent.

[0083] In the illustrated implementation, the adjustment of the visual characteristics of the virtual model 802 is applied consistently across each anatomical object 216 of the virtual model 802, such that each anatomical object 216 is shown with a constant opacity (e.g., the entire first anatomical object 216-1 is shown as more opaque than the entire second anatomical object 216-2). Additionally or alternatively, the visual characteristics of the virtual model 802 can be adjusted to vary across one or more anatomical objects 216 of the virtual model 802. For example, as the depth value of a portion of the virtual model 802 increases (e.g., away from the imaging device 204), the transparency of the portion of the virtual model 802 can increase along the anatomical objects 216 of the virtual model 802.

[0084] In some implementations, the mixed reality image generation system 202 can update the visual characteristic adjustment of the virtual model 802 (e.g., based on movement of the corresponding anatomical object 216 in the image, user input, etc.). For example, the mixed reality image generation system 202 can update the visual characteristic adjustment based on a change in a depth value associated with the virtual model 802, such that if the depth value decreases, then a portion of the virtual model 802 can be shown as more opaque, and / or if the depth value increases, then a portion of the virtual model 802 can be shown as more transparent. Additionally or alternatively, the adjustment of the visual characteristic can be fixed for at least a period of time. In some implementations, the visual characteristic adjustment can be further adjusted, for example, based on user input that specifies further adjustment of the visual characteristic after the visual characteristic has been adjusted based on the depth of one or more anatomical objects 216.

[0085] Figure 8BAn illustrative implementation 808 of adjusting the visual characteristics of a virtual model 802 based on a depth relative to a tissue surface 510 is shown. As shown, a first vertex 806-1 located on a first outer surface 804-1 of a first anatomical object 216-1 included in the virtual model 802 can be associated with a depth value representing a first depth D1 relative to the tissue surface 510, and a second vertex 806-2 located on a second outer surface 804-2 of a second anatomical object 216-2 included in the virtual model 802 can be associated with a depth value representing a second depth D2 relative to the tissue surface 510. In some implementations, the mixed reality system 202 can determine which tissue surface 510 is used to determine the depth value (e.g., the tissue surface 510 closest to the corresponding vertex 806, the tissue surface 510 perpendicular to the corresponding vertex 806, etc.). Additionally or alternatively, a user can specify which tissue surface 510 is used to determine the depth value (e.g., using the user input device 220).

[0086] Based on the depth value, the mixed reality image generation system 202 can be configured to adjust visual characteristics within the region of interest of the virtual model 802. For example, the mixed reality image generation system 202 can adjust the visual characteristics so that a display device (e.g., the display device 218) displays portions of the virtual model 802 associated with a lower depth value than the depth values ​​of other portions of the virtual model 802 as less transparent than other portions of the virtual model 802 (e.g., the transparency of the virtual model 802 can decrease as the depth value decreases). For example, a lower depth value can indicate that the portion of the virtual model 802 can correspond to the anatomical object 216 located near the tissue surface 510, such that it may be desirable for the user to more clearly visualize the anatomical object 216 located near the tissue surface 510 than the anatomical object 216 located farther away from the tissue surface 510. To illustrate, a first depth D1 associated with a first vertex 806-1 of the virtual model 802 is less than a second depth D2 associated with a second vertex 806-2 of the virtual model 802, such that the first anatomical object 216-1 associated with the first vertex 806-1 is shown as more opaque than the second anatomical object 216-2 associated with the second vertex 806-2.

[0087] In some implementations, the visual characteristics of the virtual model 802 can be adjusted based on a single depth value (e.g., associated with one or more vertices 806 of the virtual model 802, where the one or more vertices 806 can correspond to the center of mass of one or more anatomical objects 216, the point closest to the tissue surface 510, the point farthest from the tissue surface 510, etc.). This can enable the visual characteristics to be adjusted consistently across each anatomical object 216 of the virtual model 802, such that each anatomical object 216 can be shown with a constant opacity (e.g., the entire first anatomical object 216-1 is shown as more opaque than the entire second anatomical object 216-2). Additionally or alternatively, the visual characteristics of the virtual model 802 can be adjusted based on multiple depth values ​​associated with various portions of the virtual model 802, such that as the depth values ​​of the various portions of the virtual model 802 increase, the transparency of the various portions of the virtual model 802 can increase along the anatomical objects 216 of the virtual model 802. In some implementations, the visual characteristics of the virtual model 802 can be scaled (e.g., scaled based on depth relative to the imaging device and / or tissue surface 510) so that the region of interest in the image can be unambiguously determined. Other depth-based techniques for adjusting the visual characteristics of the virtual model 802 can also be used.

[0088] For example, Figure 9 An illustrative implementation 900 is shown for adjusting visual characteristics of a virtual model 802 based on a plurality of segmented depth values ​​associated with one or more vertices 902 (e.g., vertices 902-1 through 902-4) of the virtual model 802. For example, a vertex 902 of the virtual model 802 can be identified at an outer surface 804 of an anatomical object 216 included in the virtual model 802, which can be aligned with a reference location 904 (e.g., associated with the imaging device 204 and / or the tissue surface 510). A segmented depth value can be associated with the identified vertex 902 based on one or more segmented depths F (e.g., segmented depths F1 through F4), which can represent a depth from the identified vertex 902 to the reference location 904 and / or another identified vertex 902. Thus, the visual characteristics of the virtual model 802 can be adjusted based on the segmented depth values ​​associated with the vertex 902 of the virtual model 802.

[0089] For illustration, implementation 900 includes a first vertex 902-1 located on the first outer surface 804-1 of the first anatomical object 216-1, a second vertex 902-2 located on the second outer surface 804-2 of the second anatomical object 216-2, a third vertex 902-3 located on the second outer surface 804-2 of the second anatomical object 216-2 opposite the second vertex 902-2, and a fourth vertex 902-4 located on the first outer surface 804-1 of the first anatomical object 216-1 opposite the first vertex 902-1, such that each vertex 902 is aligned with a reference position 904.

[0090] Based on the identified vertex 902, the mixed reality image generation system 202 can determine a segment depth value associated with the identified vertex 902. For example, a first segment depth value can be associated with the first vertex 902-1 (e.g., the vertex 902 closest to the reference position 904) based on a first segment depth F1 from the first vertex 902-1 to the reference position 904. In some implementations, the depth values ​​associated with the remaining vertices 902 can be based on a single segment depth F and / or a combination of segment depths F (e.g., such that the segment depths F can be built on top of each other). For illustration, a second segment depth value can be associated with the second vertex 902-2 based on a second segment depth F2 from the second vertex 902-2 to the first vertex 902-1 and / or a combination of the first segment depth F1 and the second segment depth F2. A third segment depth value may be associated with the third vertex 902-3 based on the third segment depth F3 from the third vertex 902-3 to the second vertex 902-2 and / or a combination of the first segment depth F1, the second segment depth F2, and the third segment depth F3. A fourth segment depth value may be associated with the fourth vertex 902-4 based on the fourth segment depth F4 from the fourth vertex 902-4 to the third vertex 902-3 and / or a combination of the first segment depth F1, the second segment depth F2, the third segment depth F3, and the fourth segment depth F4.

[0091] The mixed reality image generation system 202 can adjust the visual characteristics of the virtual model 802 based on the segment depth values ​​associated with the vertices 902 of the virtual model 802. For example, as the segment depth value associated with a portion of the virtual model 802 increases and / or as the number of segment depths F used to determine the segment depth value increases, the portion of the virtual model 802 can be displayed as more transparent. To illustrate, the display of the virtual model 802 can increase in transparency from the first vertex 902-1 to the fourth vertex 902-4.

[0092] Figure 10AAn illustrative implementation 1000 is shown of a portion of a virtual model 1002 displayed within the region of interest 404 that can be adjusted within the display area 1004 positioned within the region of interest 404. The virtual model 1002 can implement or be similar to the virtual model 408 and / or the virtual model 802. As shown, the virtual model 1002 includes the first anatomical object 216-1 and the second anatomical object 216-2 positioned within the display area 1004 and the third anatomical object 216-3 and the fourth anatomical object 216-4 positioned outside of the display area 1004.

[0093] In some implementations, the display area 1004 can be formed by a display boundary 1006 associated with a point 1008. For example, the point 1008 can be associated with the virtual model 1002 and / or the region of interest 404. To illustrate, the point 1008 can be identified as the center of mass of the region of interest 404, the center of mass of one or more anatomical objects 216 depicted in the region of interest 404, the center of mass of one or more anatomical objects 216 of the virtual model 1002, etc. In some implementations, the point 1008 can be identified by detecting user input (e.g., using the user input device 220) specifying the point 1008 within the image and / or using a machine learning algorithm.

[0094] The display border 1006 can be spaced apart from the point 1008 within the region of interest 404 by a selected width W. As shown, the display border 1006 includes a circle having a radius of the selected width W around the point 1008, such that the selected width W is continuous around the point 1008. Additionally or alternatively, the selected width W can vary around the point 1008, such that other suitable shapes (e.g., a square, a triangle, a freeform shape, etc.) can be used to form the display border 1006 relative to the point 1008. In some implementations, the selected width W of the display border 1006 can include a predetermined distance, such as a predetermined distance specified by user input. Additionally or alternatively, the selected width W of the display border 1006 can be adjustable (e.g., based on user input).

[0095] In some implementations, one or more visual characteristics of the virtual model 1002 can be adjusted within the display area 1004. For example, the transparency of the anatomical objects 216 of the virtual model 1002 can be adjusted within the display area 1004. For illustration, the first anatomical object 216-1 of the virtual model 1002 is shown as being less transparent within the display area 1004 than the second anatomical object 216-2 of the virtual model 1002. In some implementations, the visual characteristics of the virtual model 1002 can be adjusted based on the distance of portions of the virtual model 1002 relative to the point 1008 (e.g., portions of the virtual model 1002 positioned closer to the point 1008 can be less transparent than other portions of the virtual model 1002 positioned further away from the point 1008). Additionally or alternatively, the visual characteristics of the virtual model 1002 can be adjusted based on the depth of one or more anatomical objects (e.g., relative to the imaging device 204 and / or the tissue surface 510), the size of one or more anatomical objects, etc.

[0096] In some implementations, portions of the virtual model 1002 positioned outside of the display area 1004, such as the third anatomical object 216-3 and the fourth anatomical object 216-4, can be displayed as transparent (e.g., relative to the first anatomical object 216-1 and the second anatomical object 216-2). Alternatively, portions of the virtual model 1002 positioned within the display area 1004, such as the first anatomical object 216-1 and the second anatomical object 216-2, can be displayed as opaque, while the visual characteristics of other portions of the virtual model 1002 positioned outside of the display area 1004, such as the third anatomical object 216-3 and the fourth anatomical object 216-4, can be adjusted (e.g., to increase the transparency of the portion of the virtual model 1002 when the portion of the virtual model 1002 is positioned away from the display area 1004 and / or the point 1008).

[0097] Figure 10BAn illustrative implementation 1010 of a portion of a virtual model 1002 displayed within a region of interest 404 is shown, which can be adjusted within a plurality of display regions 1004 (e.g., display regions 1004-1 through 1004-2) positioned within the region of interest 404. As shown, the plurality of display regions 1004 include a first display region 1004-1 formed by a first display boundary 1006-1 spaced from a point 1008 by a first width W1, and a second display region 1004-2 formed by a second display boundary 1006-2 spaced from the point 1008 by a second width W2 that is greater than the first width W1. Each width W of the display boundaries 1006 can be continuous around the point 1008 and / or can vary around the point 1008. In some implementations, each width W of the display boundaries 1006 can include a predetermined distance, such as a predetermined distance specified by user input. Additionally or alternatively, each width W of the display boundaries 1006 can be adjustable (e.g., based on user input).

[0098] In some implementations, one or more visual characteristics of the virtual model 1002 can be adjusted between the multiple display areas 1004. For example, the transparency of the anatomical objects 216 of the virtual model 1002 can be adjusted between the first display area 1004-1 and the second display area 1004-2. To illustrate, the first anatomical object 216-1 and the second anatomical object 216-2 of the virtual model 1002 are positioned within the first display boundary 1006-1 of the first display area 1004-1 such that the first anatomical object 216-1 and the second anatomical object 216-2 can be shown as opaque. The third anatomical object 216-3 of the virtual model 1002 is positioned between the first display boundary 1006-1 and the second display boundary 1006-2 within the second display area 1004-2 such that the visual characteristics of the third anatomical object 216-3 can be adjusted. For example, the third anatomical object 216-3 can be displayed as more transparent than the first anatomical object 216-1 and the second anatomical object 216-2 within the first display area 1004-1 (e.g., based on the distance of the third anatomical object 216-2 relative to the point 1008, the depth of the third anatomical object 216-3, the size of the third anatomical object 216-3, etc.). The fourth anatomical object 216-4 of the virtual model 1002 is positioned outside the second display boundary 1006-2, such that the fourth anatomical object 216-4 can be displayed as transparent (e.g., relative to the third anatomical object 216-3).

[0099] In some implementations, each display boundary 1006 of each display area 1004 can include a shape (e.g., a circle, a square, etc.) when the display area 1004 is completely positioned within the area of ​​interest 404. Additionally, the width W of the display boundary 1006 can increase toward the border of the area of ​​interest 404 so that when the display boundary 1006 meets the border of the area of ​​interest 404, the shape of the display boundary 1006 can be modified to confine the display area 1004 to the area of ​​interest 404. For example, the second display boundary 1006-2 is shown as a circle surrounding the point 1008. As the width W2 of the second display boundary 1006-2 increases, the edges of the second display boundary 1006-2 can flatten into the square shape of the area of ​​interest 404 to confine the second display area 1004-2 to the area of ​​interest 404.

[0100] Other suitable techniques for adjusting one or more visual characteristics of the virtual model may also be used. For example, the one or more visual characteristics may be adjusted based on information associated with the medical procedure. To illustrate, the information may indicate one or more target anatomical objects associated with the medical procedure, such that the mixed reality image generation system 202 may adjust one or more visual characteristics of the virtual model based on the one or more target anatomical objects (e.g., one or more target anatomical objects included in the virtual model may be displayed as less transparent than other anatomical objects included in the virtual model).

[0101] In certain embodiments, one or more of the processes described herein may be implemented at least in part as instructions embodied in a non-transitory computer-readable medium and executable by one or more computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes these instructions to perform one or more processes, including one or more of the processes described herein. Any of various known computer-readable media may be used to store and / or transmit such instructions.

[0102] Computer-readable media (also referred to as processor-readable media) include any non-transitory media that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a computer's processor). Such media can take many forms, including but not limited to non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memories. Volatile media can include, for example, dynamic random access memory ("DRAM"), which typically constitutes main memory. Common forms of computer-readable media include, for example, magnetic disks, hard disks, tapes, any other magnetic media, compact disc read-only memory ("CD-ROM"), digital video discs ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM, any other memory chip or cassette tape, or any other tangible medium that a computer can read from it.

[0103] Figure 11 An illustrative computing device 1100 is shown that can be specifically configured to perform one or more of the processes described herein. Any of the systems, computing devices, and / or other components described herein can be implemented by the computing device 1100.

[0104] like Figure 11 As shown, computing device 1100 may include a communication interface 1102, a processor 1104, a storage device 1106, and an input / output ("I / O") module 1108 communicatively connected to one another via a communication infrastructure 1110. Figure 11 An illustrative computing device 1100 is shown in FIG. Figure 11 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Figure 11 Components of computing device 1100 shown in .

[0105] The communication interface 1102 can be configured to communicate with one or more computing devices. Examples of the communication interface 1102 include, but are not limited to, a wired network interface (e.g., a network interface card), a wireless network interface (e.g., a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0106] The processor 1104 generally represents any type or form of processing unit capable of processing data and / or interpreting, executing, and / or directing the performance of one or more of the instructions, processes, and / or operations described herein. The processor 1104 may perform operations by executing computer-executable instructions 1112 (e.g., applications, software, code, and / or other executable data instances) stored in the storage device 1106.

[0107] The storage device 1106 may include one or more data storage media, devices, or configurations and may take any type, form, and combination of data storage media and / or devices. For example, the storage device 1106 may include, but is not limited to, any combination of non-volatile media and / or volatile media described herein. Electronic data (including the data described herein) may be temporarily and / or permanently stored in the storage device 1106. For example, data representing computer-executable instructions 1112 configured to instruct the processor 1104 to perform any of the operations described herein may be stored within the storage device 1106. In some examples, the data may be arranged in one or more databases residing within the storage device 1106.

[0108] The I / O module 1108 may include one or more I / O modules configured to receive user input and provide user output. The I / O module 1108 may include any hardware, firmware, software, or combination thereof that supports input and output functions. For example, the I / O module 1108 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0109] The I / O module 1108 may include one or more devices for presenting output to a user, including but not limited to a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1108 is configured to provide graphics data to the display for presentation to the user. The graphics data may represent one or more graphical user interfaces and / or any other graphical content as may be useful in a particular implementation.

[0110] In the previous description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and alterations may be made to the present invention, and additional embodiments may be implemented, without departing from the scope of the present invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with or substituted for features of another embodiment described herein. Accordingly, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A computer-assisted medical system comprising: a repositionable manipulator arm configured to couple to an imaging device configured to capture images depicting a scene including one or more anatomical objects positioned within a field of view of the imaging device; as well as a computing device communicatively coupled to the imaging device and configured to: determining a region of interest within the image; accessing a virtual model associated with the one or more anatomical objects; and Presenting a combined image including the virtual model superimposed on the image by instructing a display device to perform the following operations: displaying within the combined image a first portion of the virtual model positioned within the region of interest of the image, and Display of a second portion of the virtual model located outside the region of interest of the image within the combined image is avoided.

2. The computer-assisted medical system according to claim 1, wherein: The computing device is further configured to cause the repositionable manipulator arm to position the field of view of the imaging device to include the one or more anatomical objects.

3. The computer-assisted medical system of claim 1 , further comprising a second repositionable manipulator arm configured to couple to the instrument, wherein The computing device is further configured to cause the second repositionable manipulator arm to move the instrument within the field of view of the imaging device to manipulate the one or more anatomical objects.

4. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest is based on a depth of the one or more anatomical objects relative to the imaging device.

5. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest is based on a depth of the one or more anatomical objects relative to a tissue surface.

6. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest is based on a size of the one or more anatomical objects.

7. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest includes determining a boundary of the region of interest, the boundary being spaced a selected distance from points associated with the one or more anatomical objects.

8. The computer-assisted medical system according to claim 7, wherein: The point is based on an identified centroid of the one or more anatomical objects.

9. The computer-assisted medical system according to claim 7, wherein: The point is specified on the one or more anatomical objects by user input.

10. The computer-assisted medical system according to claim 7, wherein: The selected distance of the border is adjustable.

11. The computer-assisted medical system according to claim 1, wherein: The region of interest includes a first sub-region of interest and a second sub-region of interest, wherein determining the region of interest includes: determining the first subregion of interest by determining a first boundary containing a first point associated with a first anatomical object; and The second sub-region of interest is determined by determining a second boundary containing a second point associated with a second anatomical object.

12. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest includes determining a boundary including a first point associated with a first anatomical object and a second point associated with a second anatomical object.

13. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest is based on detecting user input specifying the region of interest.

14. The computer-assisted medical system according to claim 1, wherein: Determining the region of interest includes identifying the one or more anatomical objects.

15. The computer-assisted medical system according to claim 1, wherein: Displaying the first portion of the virtual model includes displaying an outline of the region of interest.

16. The computer-assisted medical system according to claim 1, wherein: Displaying the first portion of the virtual model includes adjusting visual characteristics of the first portion of the virtual model within the region of interest.

17. The computer-assisted medical system according to claim 16, wherein: Adjusting the visual characteristic is based on a depth of the one or more anatomical objects relative to the imaging device.

18. The computer-assisted medical system according to claim 16, wherein: Adjusting the visual characteristic is based on a depth of the one or more anatomical objects relative to a tissue surface.

19. The computer-assisted medical system according to claim 18, wherein: The depth of the one or more anatomical objects relative to the tissue surface is based on a single depth value.

20. The computer-assisted medical system according to claim 18, wherein The depth of the one or more anatomical objects relative to the tissue surface is based on a plurality of segmented depth values.

21. The computer-assisted medical system according to claim 16, wherein: Adjusting the visual characteristic is based on a size of the one or more anatomical objects.

22. The computer-assisted medical system according to claim 16, wherein: The visual characteristic is adjustable within a display region positioned within the region of interest, wherein the display region includes a display boundary spaced a selected distance from points associated with the one or more anatomical objects.

23. The computer-assisted medical system according to claim 22, wherein: The point is based on an identified centroid of the one or more anatomical objects.

24. The computer-assisted medical system according to claim 22, wherein: The point is specified on the one or more anatomical objects by user input.

25. The computer-assisted medical system according to claim 22, wherein: The selection distance of the display border is adjustable.

26. The computer-assisted medical system according to claim 22, wherein: The selected distance of the display boundary is a predetermined distance.

27. The computer-assisted medical system according to claim 22, wherein: The selected distance of the display boundary is specified by user input.

28. The computer-assisted medical system according to claim 22, wherein: The display boundaries include a first display boundary separated from a point associated with the one or more anatomical objects by a first distance and a second display boundary separated from the point by a second distance greater than the first distance, wherein the visual characteristic is adjustable between the first display boundary and the second display boundary.

29. The computer-assisted medical system according to claim 28, wherein: Select one or both of the first distance and the second distance as being adjustable.

30. The computer-assisted medical system according to claim 22, wherein: When the display boundary is positioned within the region of interest, the display boundary comprises a shape.

31. The computer-assisted medical system according to claim 30, wherein: The display boundary is constrained to the region of interest such that a shape of the display boundary is modified when the display boundary meets a boundary of the region of interest.

32. The computer-assisted medical system according to claim 16, wherein: Adjusting the visual characteristic is based on detecting user input specifying the visual characteristic.

33. The computer-assisted medical system according to claim 1, wherein: Presenting the combined image includes detecting that the first portion of the virtual model is repositioned outside of the region of interest, and in response, avoiding display of the first portion.

34. The computer-assisted medical system according to claim 1, wherein: Presenting the combined image includes detecting that the second portion of the virtual model is repositioned within the region of interest, and in response, displaying the second portion.

35. The computer-assisted medical system according to claim 1, wherein: The virtual model is based on pre-operative images of the one or more anatomical objects.

36. A system comprising: a memory for storing instructions; as well as one or more processors communicatively coupled to the memory and configured to execute the instructions to perform a process comprising: determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image, the presenting including instructing a display device to: displaying within the combined image a first portion of the virtual model positioned within the region of interest of the image, and Display of a second portion of the virtual model located outside the region of interest of the image within the combined image is avoided.

37. The system of claim 36, wherein: Determining the region of interest is based on a depth of the one or more anatomical objects relative to an instrument.

38. The system of claim 36, wherein: Determining the region of interest is based on a depth of the one or more anatomical objects relative to a tissue surface.

39. The system of claim 36, wherein: Determining the region of interest includes determining the depth of the one or more anatomical objects by generating a depth map of the scene.

40. The system of claim 36, wherein: Determining the region of interest is based on a size of the one or more anatomical objects.

41. The system of claim 36, wherein: Determining the region of interest includes determining a boundary of the region of interest, the boundary being spaced a selected distance from points associated with the one or more anatomical objects.

42. The system of claim 36, wherein: Determining the region of interest is based on detecting user input specifying the region of interest.

43. The system of claim 36, wherein: Displaying the first portion of the virtual model includes adjusting visual characteristics of the first portion of the virtual model within the region of interest.

44. The system of claim 43, wherein: Adjusting the visual characteristic is based on a depth of the one or more anatomical objects relative to the instrument.

45. The system of claim 43, wherein: Adjusting the visual characteristic is based on a depth of the one or more anatomical objects relative to a tissue surface.

46. ​​The system of claim 43, wherein: Adjusting the visual characteristic is based on a size of the one or more anatomical objects.

47. The system of claim 43, wherein: The visual characteristic is adjustable within a display region positioned within the region of interest, wherein the display region includes a display boundary spaced a selected distance from points associated with the one or more anatomical objects.

48. The system of claim 47, wherein: The display boundaries include a first display boundary separated from a point associated with the one or more anatomical objects by a first distance and a second display boundary separated from the point by a second distance greater than the first distance, wherein the visual characteristic is adjustable between the first display boundary and the second display boundary.

49. The system of claim 43, wherein: Adjusting the visual characteristic is based on detecting user input specifying the visual characteristic.

50. The system of claim 36, wherein: Presenting the combined image includes detecting that the first portion of the virtual model is repositioned outside of the region of interest, and in response, avoiding display of the first portion.

51. The system of claim 36, wherein: Presenting the combined image includes detecting that the second portion of the virtual model is repositioned within the region of interest, and in response, displaying the second portion.

52. A method comprising: determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image, the presenting including instructing a display device to: displaying within the combined image a first portion of the virtual model positioned within the region of interest of the image, and Display of a second portion of the virtual model located outside the region of interest of the image within the combined image is avoided.

53. The method of claim 52, wherein: Determining the region of interest is based on a depth of the one or more anatomical objects.

54. The method of claim 52, wherein: Determining the region of interest is based on a size of the one or more anatomical objects.

55. The method of claim 52, wherein: Determining the region of interest includes determining a boundary of the region of interest, the boundary being spaced a selected distance from points associated with the one or more anatomical objects.

56. The method of claim 52, wherein: Determining the region of interest is based on detecting user input specifying the region of interest.

57. The method of claim 52, wherein: Displaying the first portion of the virtual model includes adjusting visual characteristics of the first portion of the virtual model within the region of interest.

58. The method of claim 52, wherein: Presenting the combined image includes detecting that the first portion of the virtual model is repositioned outside of the region of interest, and in response, avoiding display of the first portion.

59. The method of claim 52, wherein: Presenting the combined image includes detecting that the second portion of the virtual model is repositioned within the region of interest, and in response, displaying the second portion.

60. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform a process comprising: determining a region of interest within an image depicting a scene including one or more anatomical objects; accessing a virtual model associated with the one or more anatomical objects; and presenting a combined image including the virtual model superimposed on the image, the presenting including instructing a display device to: displaying within the combined image a first portion of the virtual model positioned within the region of interest of the image, and Display of a second portion of the virtual model located outside the region of interest of the image within the combined image is avoided.

Images