System and method for generating a 3D navigation interface for medical procedure

Abstract

Systems and methods for generating a 3D navigation interface for a robot-assisted medical procedure are described. The method may include: (i) receiving a 3D model representing a volume of a patient; (ii) receiving 2D data representing one or more 2D images corresponding to at least a portion of the volume of the patient; (iii) generating co-registered operational data relating the 3D model to the 2D data; (iv) generating a 3D navigation interface, including: generating a display of at least a portion of the 3D model based on the co-registered operational data; and (v) causing the display device to display the 3D navigation interface to the user. The method may be implemented by a system including a head-mounted device that displays a 3D navigation interface in an augmented reality view.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of the filing date of provisional U.S. patent application No. 63 / 477,752, entitled “SYSTEMS AND METHODS FORGENERATING 3D NAVIGATION INTERFACES FOR MEDICAL PROCEDURES,” filed on December 29, 2022. The entire contents of this provisional application are hereby expressly incorporated herein by reference. Technical Field

[0003] The disclosed examples relate to three-dimensional navigation systems. Specifically, the disclosed examples relate to systems and methods for generating and modifying 3D navigation systems for performing medical procedures on patients. Background Art

[0004] Minimally invasive medical technology is intended to reduce the amount of tissue damaged during the medical procedure, thereby reducing patient recovery time, discomfort and harmful side effects. Such minimally invasive technology can be performed through the natural orifice in the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, the doctor can insert a minimally invasive medical device (including surgical operation, diagnosis, treatment and / or biopsy instrument) to reach the target tissue position. A kind of such minimally invasive technology is to use a flexible and / or steerable elongation device (such as a flexible catheter), which can be inserted into an anatomical passage and navigated towards the region of interest in the patient's anatomical structure.

[0005] Past medical technology has used two-dimensional displays / interfaces when navigating the patient volume and when determining the location to perform a specific procedure (e.g., a biopsy, ablation, etc.). However, because the patient's interior is three-dimensional, such displays / interfaces may lack details that are useful to the physician, especially when performing complex procedures. For example, a two-dimensional display may require the physician to mentally map the two-dimensional information to a mental / imaginary three-dimensional construct to determine where to proceed, which is mentally taxing for the physician and may lead to errors during surgery. Similarly, conventional displays or interfaces may not provide the physician with sufficient information about their surroundings when performing a medical procedure. As such, the physician may be unaware of important details during the medical procedure. Summary of the Invention

[0006] The following presents a simplified summary of various examples described herein and is not intended to identify key or critical elements or to delineate the scope of the claims.

[0007] In some examples, a computer-implemented method for generating a three-dimensional (3D) navigation interface for a robotic-assisted medical procedure is provided. The method can be implemented via one or more local or remote processors, servers, sensors, transceivers, memory units, and / or other electronic or electrical components. The method can include: (i) receiving, by one or more processors, a 3D model representing a volume of a patient; (ii) receiving, by one or more processors, 2D data representing one or more two-dimensional (2D) images corresponding to at least a portion of the volume of the patient; (iii) generating, by the one or more processors, co-registered operation data that correlates the 3D model with the 2D data; (iv) generating, by the one or more processors, a 3D navigation interface, including: generating a display of at least a portion of the 3D model based on the co-registered operation data; and (v) causing, by the one or more processors, a display device to display the 3D navigation interface to a user.

[0008] In another example, a system for generating a 3D navigation interface for a robotic-assisted medical procedure is provided. The system may include one or more processors; a communication unit; a display device; and a non-transitory computer-readable medium coupled to the one or more processors and the communication unit and having instructions stored thereon that, when executed by the one or more processors, cause the system to: (i) receive a 3D model representing a volume of a patient; (ii) receive 2D data representing one or more 2D images corresponding to at least a portion of the volume of the patient; (iii) generate co-registered operational data relating the 3D model to the 2D data; (iv) generate a 3D navigation interface, including generating a display of at least a portion of the 3D model based on the co-registered operational data; and (v) cause the display device to display the 3D navigation interface to a user.

[0009] In yet another example, a method for generating an interactive view for a robotically assisted medical procedure is provided. The method can be implemented via one or more local or remote processors, servers, sensors, transceivers, memory units, and / or other electronic or electrical components. The method can include: (i) receiving, by one or more processors, a 3D model representing a volume of a patient; (ii) generating, by one or more processors, a 2D view of the volume of the patient, the 2D view representing a 2D imaging modality when positioned at a specific device projection angle; (iii) determining a display orientation of the 2D view relative to a user; (iv) registering the 2D view to the 3D model such that both the 2D view and the 3D model share a display orientation; (v) causing, by one or more processors, a display device to simultaneously display the 2D view and the 3D model to the user according to the shared display orientation; (vi) receiving, by one or more processors, a control input from the user; and (vii) updating, by the one or more processors, at least the 2D view of the volume of the patient based on the control input.

[0010] In another example, a system for generating an interactive view for a robotic-assisted medical procedure is provided. The system may include one or more processors; a communication unit; a display device; and a non-transitory computer-readable medium coupled to the one or more processors and the communication unit and having instructions stored thereon that, when executed by the one or more processors, cause the system to: (i) receive a 3D model representing a volume of a patient; (ii) generate a 2D view of the volume of the patient, the 2D view representing a 2D imaging modality when positioned at a specific device projection angle; (iii) determine a display orientation of the 2D view relative to a user; (iv) register the 2D view to the 3D model such that the 2D view and the 3D model share a display orientation; (v) cause the display device to simultaneously display the 2D view and the 3D model to the user according to the shared display orientation; (vi) receive control input from the user; and (vii) update at least the 2D view of the volume of the patient based on the control input.

[0011] It should be understood that the foregoing general description and the following detailed description are illustrative and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram of an example environment in which a user may perform a medical procedure using a medical device, according to some examples.

[0013] Figure 2A and Figure 2Bare diagrams of different views of an example environment displaying a 3D model of a patient's internal volume along with 2D images to assist a user in performing a medical procedure, according to some examples.

[0014] Figure 3 Depicted is an example architecture for a system that assists a user in performing a medical procedure according to some examples.

[0015] Figure 4 Another example architecture of a system for assisting a user in performing a medical procedure according to some examples is depicted.

[0016] Figure 5 is an example flow chart for generating a 3D navigation interface for a robotic-assisted medical procedure, according to some examples.

[0017] Figure 6 is another example flow chart for generating a 3D navigation interface for a robotic-assisted medical procedure according to some examples.

[0018] Figure 7 is a simplified diagram of a medical system in which the techniques disclosed herein may be implemented, according to some examples.

[0019] Figure 8A is a simplified diagram of a medical device system including a flexible elongate device, according to some examples, that can be used in conjunction with the technology disclosed herein.

[0020] Figure 8B Based on some examples Figure 8A Simplified illustration of a medical tool within a flexible extension device.

[0021] Figure 9A and Figure 9B is a simplified diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some examples.

[0022] By referring to the following detailed description, examples of the present disclosure and its advantages can be better understood. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the accompanying drawings, in which the illustrations are for the purpose of illustrating examples of the present disclosure, not for the purpose of limiting the examples of the present disclosure. DETAILED DESCRIPTION

[0023] In the following description, the specific details of some examples that describe compliance with the present disclosure are set forth. Many specific details are set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples may be put into practice without some or all of these specific details. The specific examples disclosed herein are intended to be exemplary and non-restrictive. Those skilled in the art may implement other elements that are not specifically described herein but within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with an example may be incorporated into other examples, unless specifically described otherwise or if one or more features will render the example inoperative. In some cases, well-known methods, processes, components, and circuits are not described in detail to avoid unnecessarily obscuring the various aspects of the examples.

[0024] This disclosure describes various instruments and parts of instruments according to their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or a part of an object in three-dimensional space (e.g., along the three translational degrees of freedom of Cartesian x-coordinates, y-coordinates, and z-coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a part of an object (e.g., one or more rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a part of an object on at least one translational degree of freedom and the orientation of the object or a part of an object on at least one rotational degree of freedom (e.g., up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, and / or orientations measured along an object. As used herein, the term "distal" refers to a position closer to a process site, and the term "proximal" refers to a position further away from a process site. Therefore, when an instrument is designed to perform a process, the distal portion or distal end of the instrument is closer to the process site than the proximal portion or proximal end of the instrument.

[0025] The present disclosure may involve using a mixed reality head-mounted display (HMD) or other display device to provide a three-dimensional (3D) navigation volume and input interface. In particular, the disclosed systems and methods include techniques for providing a 3D interface by generating co-registered data relating a 3D model of a patient volume to a 2D image of at least a portion of the patient volume. The 3D interface can display relevant graphics / images and possibly other information (e.g., historical data, annotations, etc.) to a physician preparing for or performing a medical procedure. This allows the physician to more easily identify relevant portions of the patient's anatomy and navigate the anatomical atlas without having to account for mismatches between the image and the patient model, and without having to mentally map the graphical representation of the 3D model to different portions of a 2D image (e.g., a 2D intraoperative image). Similarly, the physician maintains awareness of the surrounding physical area in which the procedure is being performed, making it easier to track and predict how ancillary medical tools or devices move or interact with the patient. Similarly, the disclosed systems and methods include techniques for generating a 2D view of the patient volume while maintaining its orientation relative to the physician. In this way, the physician may modify or otherwise interact with the 2D view without requiring the physician to remember or incorporate changes in orientation between views, images, modifications, and the like.

[0026] The systems and methods described herein can provide numerous improvements by generating and using a 3D navigation environment. For example, a 3D navigation environment can enable a user to navigate the environment as described herein even when not physically in the environment. Furthermore, by generating and using a 3D navigation environment, additional users can observe the process in real time without interrupting the first (e.g., primary) user.

[0027] Similarly, the systems and methods described herein can provide users with improved spatial perception by visualizing and displaying 3D content in a 3D setting rather than a 2D medium. Additionally, the use of precise hand and / or joint tracking can provide users with options for more accurately navigating a 3D environment rather than relying solely on windows, icons, menus, and other such traditional 2D methods.

[0028] Furthermore, the introduction of real-time systems and methods for 3D information via a head-mounted display (HMD) can improve ergonomics by enabling the user to function within a free and natural range of motion, as if the user were in a normal environment, while still providing the benefits of a virtual environment. For example, the system can improve the user's mental mapping by reducing reliance on 2D displays and instead displaying 3D models in conjunction with 2D information where appropriate, allowing the user to rotate, modify, and otherwise adjust the display to a preferred comfort level.

[0029] It will be understood that this modification does not constitute an exhaustive list and that other modifications will be apparent from the various examples discussed herein.

[0030] First refer to Figure 1 , shows an example environment 100 in which a user can perform a medical procedure using a medical device as described in more detail herein. In particular, the environment 100 includes a robotic-assisted platform 102, a medical device 104, and a head-mounted display (HMD) 110. It will be understood that the environment 100 is an example, and alternative examples including additional, fewer, or alternative components are contemplated. For example, according to the example, the robotic-assisted platform 102 may include or omit Figure 1 Monitor and manual input console depicted in .

[0031] The robotic-assisted platform 102 is a tool for assisting a physician in performing a medical procedure on a patient, such as an endocavity procedure (eg, a minimally invasive lung biopsy or ablation procedure). In some examples, the robotic-assisted platform 102 includes Figure 7 At least some components of the robot-assisted platform 102 (as described in more detail below), such as the manipulator assembly, the control system, the sensor system, the display system, and / or the main assembly. According to an example, the robot-assisted platform 102 can receive commands from the user through the HMD 110. Figure 1 In an example, the robotic-assisted platform 102 is physically connected to the medical device 104 and can manipulate the medical device 104 in response to received commands. According to an example, the robotic-assisted platform 102 can include a monitor for presenting information and / or an input console for a user to manipulate the medical device 104 as described below.

[0032] The medical instrument 104 is an instrument that is configured and prepared to be manipulated by a user (via the robotic-assisted platform 102 and / or the HMD 110) when performing a medical procedure on a patient. The medical instrument 104 may be a flexible elongated device (e.g., a catheter), as described below with respect to Figure 8A and 8B Similarly, in some examples, the medical device 104 includes the following Figure 7 A more detailed description of the medical device is as follows Figure 7 A more detailed description of a part of a medical device or as described below Figure 7 A more detailed description of the medical device.

[0033] The HMD 110 is a device designed to be mounted on the user's head and to display information to the user in an extended reality (XR) view 110R. According to an example, the HMD 110 can use XR technology, for example, by presenting a mixed reality (MR) view, an augmented reality (AR) view, a virtual reality (VR) view, etc. In some examples, the HMD 110 includes one or more processors, a display device, a memory, a sensor, a controller, etc., and can be communicatively coupled to the robotic-assisted platform 102 and / or the medical device 104. According to an example, the processor of the HMD 110 can perform various operations as described herein, and can cause the display device to generate, display, modify, or otherwise manipulate elements of the XR view 110R based on instructions or movements from the user. Similarly, the HMD 110 can receive information from the robotic-assisted platform 102 and / or the medical device 104 (and / or Figure 1 Other components not shown in the figure) receive feedback, input, and / or instructions and may generate, display, modify, or otherwise manipulate elements of the XR view 110R.

[0034] In some examples, the HMD 110 receives control inputs from the user based on one or more manual inputs 106 via a controller, trackball, keyboard, mouse, touchscreen device, touch sensor, motion sensor, accelerometer, gyroscope, positioning sensor, etc. According to an example, the HMD 110 can receive manual input 106 by detecting the user's movements, such as gestures, head movements, etc., and can also receive manual input 106 by interpreting specific actions based on the movements. For example, if the user spreads their thumb and index finger apart or brings their thumb and index finger together, the HMD 110 can determine that the user is indicating a zoom in or zoom out on an element, respectively, and can then modify the XR view 110R accordingly. In another example, the user can interchangeably use multiple methods for transmitting manual input 106 as described above. For example, the user can use a motion sensor to provide manual input 106 through gestures before switching to a mouse for more refined control.

[0035] The XR view 110R may include a 3D model 112R of an internal volume of the patient. According to an example, the internal volume may be or include a specific organ (e.g., lungs), a body system (e.g., respiratory system), a larger area of ​​the patient (e.g., the patient's chest), the patient as a whole, etc. Figure 1In the example of FIG, HMD 110 displays various airways in a patient's lungs. According to an example, 3D model 112R may include one or more landmarks 116R, such as lesions or other similar identifiable features. In some examples, HMD 110 may generate an indication of landmark 116R in XR view 110R, highlight landmark 116R, enlarge landmark 116R, and / or otherwise emphasize landmark 116R to enable a user to more easily detect landmark 116R.

[0036] In addition, the XR view 110R may include a virtual representation of the medical device 104 as a virtual device 114R. According to an example, the virtual device 114R may substantially match the medical device 104, or may be a simplified version of the medical device 104, such as a simple shape (e.g., a line) that follows the outline of the medical device. In some examples, the user may be able to provide manual input 106 to the HMD 110 by interacting with the virtual device 114R.

[0037] In another example, the HMD 110 can transmit manual input 106 received from the user to the robotic-assisted platform 102 and / or the medical device 104. In this way, the robotic-assisted platform 102 can manipulate the medical device 104 and / or the medical device can perform various functions based on the manual input 106 from the user in the XR view 110R. For example, the user can manipulate (interact with) the virtual device 114R to "drag" the virtual device 114R along the model 112R. The physical medical device 104 can then follow the path of the dragged virtual device 114R. In other examples, the HMD 110 can display a confirmation message in the XR view 110R before the physical medical device 114R follows the path in question. In another example, the user can click a landmark 116R or other location on the 3D model 112R, and the medical device 104 can follow the path to the indicated location.

[0038] In some examples, the HMD 110 may display a 3D model and / or available commands or instructions to the user when an application or device is launched. By selecting a command or performing an action according to the instructions, the HMD 110 may display other elements in the XR view 110R, as described below with respect to Figure 2A and Figure 2B More detailed explanation.

[0039] will understand, though Figure 1 The illustrative environment 100 depicts an HMD 110 as displaying information to a user, but alternative examples of the environment 100 may use different display devices, such as 3D computers, handheld XR devices, mobile computing devices, etc.

[0040] Next refer to Figure 2A and Figure 2B , example AR views 200A and 200B of an environment 200 are shown. In some examples, AR view 200B is an enlarged representation of AR view 200A. Both AR views 200A and 200B depict a 3D model positioned above a patient that includes physical medical instruments and virtual medical instruments.

[0041] In particular, the environment 200 includes a medical device 204, a display device 210 depicting a navigation interface 210R, and an imaging device 220. It will be understood that the AR views 200A and 200B are example views, and alternative examples including additional, fewer, or alternative components can be envisioned. The navigation interface 210R can include a 2D view 211, a 3D model 212, a 2D instrument projection 213, a virtual instrument 214, a 2D landmark projection 215, a directional 2D view 218, and a 2D instrument video 219. According to an example, the navigation interface 210R can include more, fewer, or alternative elements than those shown and described herein. For example, the navigation interface 210R can include an interface panel with commands, instructions, patient information, preoperative image data, etc. In some examples, the elements of the navigation interface 210R include or are the same as those described above with respect to Figure 1 For example, the display device 210 may include the HMD 110, the navigation interface 210R may include the XR view 110R, the medical device 204 may include the medical device 104, the 3D model 212 may include the 3D model 112R, the virtual device 214 may include the virtual device 114R, the landmark 216 may include the landmark 214, etc. Figure 1 The alternative examples described can similarly be applied appropriately to Figure 2A and Figure 2B components.

[0042] In some examples, display device 210 receives an image depicting a 2D view 211 of a patient volume, the image being captured by an imaging device, such as imaging device 220. According to this example, imaging device 220 can be a C-arm that performs X-ray fluoroscopy, a computed tomography (CT) imaging device, a cone-beam computed tomography (CBCT) imaging device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) imaging device, a tomosynthesis imaging device, a combination of devices, and / or any other similar imaging device or devices. Display device 210 can display 2D view 211 in response to receiving the image and / or in response to a user command. According to an example, display device 210 can display 2D view 211 in such a manner that 2D view 211 is superimposed on the patient's body, above, below, adjacent to, or the like, a 3D model 212.

[0043] It will be understood that although the examples described herein refer to specific components of the environment 200 (e.g., the display device 210) as performing specific processes, other components may perform similar or identical processes according to the examples. For example, the robotic-assisted platform 102 or other components of the environment 100 may perform various processes instead of or in addition to the various components of the environment 200 described above.

[0044] In some examples, display device 210 determines that there is a discrepancy between 2D view 211 and 3D model 212 or the patient volume. In such examples, display device 210 can update 2D view 211 by causing a component of environment 200 (e.g., imaging device 220) to capture an intraoperative image to replace 2D view 211. In other examples, display device 210 replaces 2D view 211 with another type of 2D data (e.g., replacing a CT image with CBCT volume data).

[0045] In further examples, the 2D view 211 may include 2D projections of various elements in the 3D model 212. For example, a virtual instrument 214 in the 3D model 212 may be represented by a 2D instrument projection 213 in the 2D view 211. In some examples, the display device 210 adjusts the 2D instrument projection 213 based on changes in the positioning, angle, etc. of the medical instrument 204 and / or the virtual instrument 214. Similarly, the display device 210 may project a landmark 216 into the 2D view 211 as a 2D landmark projection 215. In some examples, the landmark 216 and / or the 2D landmark projection 215 indicates a location to which the user wants to guide the medical instrument 204, and clicking on the landmark 216 or the 2D landmark projection 215 may cause the medical instrument 204 to navigate to the indicated location, as described above with respect to Figure 1 Described in more detail.

[0046] In some examples, the display device 210 can generate and / or display a path to the landmark 216. According to an example, the path can be generated using the centerline of the model path. For example, in the 3D model 212 of the lungs, the display device 210 can calculate the path using the centerline of the airway. In another example, the medical device 204 and / or a computing device associated with the medical device 204 can determine the amount of force applied to the medical device 204 and then determine to stop applying the force if the force reaches a threshold amount (and, in response, stop applying the force).

[0047] According to an example, a user can cause the medical device 204 to sample the landmark 216 and / or other locations (lesions, tissue, etc.) to which the medical device 204 is navigated. For example, the medical device 204 can capture an image of the landmark 216, collect a biopsy sample of the landmark 216, collect a histological sample via rapid histological sampling, etc. In some such examples, the display device 210 associates the sample and / or sample information with the sample location in the 3D model 212 and / or the entire 3D model 212. The display device 210 can determine whether the tissue is malignant (e.g., by scraping and recording the tissue material before sending it to a laboratory for analysis) and can mark the location with the information in question.

[0048] In another example, the display device 210 can generate a post-procedure report representing the sample associated with the precise location in response to an instruction from the user and / or a determination by the display device 210. In some examples, the procedure report can include a navigation history so that the user can replay part and / or all of the procedure after the operation. The procedure report can be based on virtual simulation data as well as real data received from the medical device 204.

[0049] In some examples, the navigation interface 210R includes a view into the actual patient volume. For example, the navigation interface 210R may include an oriented 2D view 218 and / or a 2D instrument video 219. In some examples, the oriented 2D view 218 may depict the current view from the medical tool 204, similar to the 2D instrument video 219, but oriented according to the pose of the virtual instrument 214. For example, the oriented 2D view 218 may be displayed in an orientation corresponding to the pointing direction of the virtual instrument 214. As another example, the oriented 2D view 218 may depict the current view from the medical tool 204 oriented according to the user's position. For example, the oriented 2D view 218 may include a live camera view or feed oriented to always face the user. In another example, the oriented 2D view 218 may additionally or alternatively depict a virtual 2D view of a specific point and / or portion generated from the 3D model 212. For example, in the example of AR view 200B, the oriented 2D view 218 depicts a 2D view of the 3D model generated based on the position and / or location of the virtual instrument 214 and / or medical instrument 204. In some examples, the display device 210 automatically generates and / or updates the oriented 2D view 218 as the medical instrument 204 and / or virtual instrument 214 moves and / or the 3D model 212 otherwise changes. In other examples, the display device 210 generates and / or updates the oriented 2D view 218 in response to instructions from the user. Similarly, the display device 210 can display a 2D instrument video 219 depicting a live feed from the medical instrument 204 in the navigation interface 210R. In some examples, the oriented 2D view 218 includes an image captured by the medical instrument 204 at a location indicated by the user (e.g., a landmark 216, a sample location, etc.). Similar to the 2D view 211, the display device 210 can display the 2D instrument video in the navigation interface 210R such that the 2D instrument video 219 is offset relative to the 3D model 212 and / or the patient, is displayed superimposed with the 3D model 212 and / or the patient, etc. It will be understood that although Figure 2B A directional 2D view 218 is depicted, but the display device 210 may additionally or alternatively generate a 3D view of a portion of the model, such as a CBCT volume according to the scan angle.

[0050] In some examples, the display device 210 generates co-registration operational data that relates the 2D view 211 or other 2D data to the 3D model 212. The co-registration operational data can be data that relates the 2D data to the 3D data, such as the locations of similar structures and / or matching structures (e.g., entry / exit points, landmarks, joints, etc.). In another example, the co-registration operational data can include a pixel-by-pixel mapping between 2D data (e.g., a 2D image, a 2D coordinate system, etc.) and 3D data (e.g., a 3D model, a 3D coordinate system, etc.). Similarly, the co-registration operational data can include optical markers and / or fiducials for visual tracking via a 2D or 3D imaging sensor (e.g., an RGB camera, an infrared camera, etc.). In some examples, the display device 210 can use 3D shape sensor data and / or external electromagnetic tracking data to perform the co-registration described herein. According to an example, the display device 210 may continuously or dynamically update the co-registration based on data measured throughout the process (eg, via visual identification of landmarks within the 3D model 212 from the 2D data or 2D view 211 ).

[0051] In further examples, the co-registered operational data may include a common reference frame, which may be, for example, a surgical reference frame for a user, a patient reference frame for a patient, an observer reference frame for an observer, etc. Thus, the co-registered operational data may enable the display device 210 to orient, overlay, and / or otherwise generate components of a view (such as the 2D view 211 , the 3D model 212 , etc.).

[0052] In some examples, the display device 210 can automatically generate co-registration data using machine learning, image analysis, optical recognition, and / or other similar techniques. In another example, the display device 210 can prompt the user to interact with one or more objects on the 3D model 212 and the 2D view 211, and can generate co-registration operation data based on these interactions. According to examples, the interaction can include touching the 3D model 212 and / or the 2D view 211, dragging a virtual instrument 214 to a certain location, moving a physical controller to a certain location, and / or any other similar method of interacting with the virtual environment. Similarly, the display device 210 can prompt the user to interact with the 3D model 212 to determine a reference frame for the patient volume. In another example, the co-registration process can involve matching measurement points on the patient with the 2D view 211 and / or 3D model 212 using rigid and / or non-rigid transformations. The measurement points can be generated using anatomical landmarks, electromagnetic coils scanned and tracked during the medical procedure, shape sensor systems, and / or other similar techniques. According to an example, registering the measured points to points in the 3D model 212 can be accomplished using an iterative closest point (ICP) technique. ICP and other registration techniques are described in PCT Application Publication No. WO2017 / 030913 and PCT Application Publication No. WO2017 / 030915, both filed on August 14, 2015, the entire contents of which are incorporated herein by reference.

[0053] The system can use the co-registration data and / or reference frame to determine the display orientation of the 2D view 211 and / or 3D model 212. In some examples, the 2D view 211 and 3D model 212 share an orientation, and the display device 210 simultaneously displays the 2D view 211 and 3D model 212 according to the shared orientation. When a user updates one of the 2D view 211 or 3D view 212 (e.g., by manipulating and / or modifying the virtual view), the display device 210 can automatically adjust the other view so that the 2D view 211 and 3D model 212 continue to share the shared orientation. In other examples, the display device 210 can adjust one view but not the other in response to an instruction from the user.

[0054] According to an example, the user can turn on or off elements of the navigation interface 210R. In some examples, the display device 210 is combined with a platform for controlling medical devices (e.g., as described above with respect to Figure 1The described robotic assistance platform 102 continuously monitors changes in the 3D model 212 and / or 2D view 211 and updates appropriate elements so that when a user reopens a view, both views continue to share the same orientation and / or user-specified markers. According to an example, if the display device 210 detects that the user is distracted by various elements, the display device 210 can automatically turn those elements on or off. In another example, the display device 210 can make elements opaque or translucent depending on the task at hand or user context. For example, if a user is dragging a virtual tool 214 through the 3D model 212, the display device 210 can make the virtual tool 214 opaque when the user makes contact with the virtual tool 214. Similarly, if the user turns away from certain elements, the system can make the elements at least temporarily translucent. In another example, the display device 210 switches between XR modes, AR, VR, and MR, depending on the situation. For example, if display device 210 and / or another component of environment 200 detects a sufficient amount of motion in the operating room, display device 210 may determine that the motion may be distracting to the user and switch from AR mode to VR mode to block the visual distraction.

[0055] In some examples, the navigation interface 210R may include a virtual representation of the imaging device 220. According to an example, the display device 210 may display the virtual imaging device such that the virtual imaging device is superimposed on the imaging device 220, is located near the imaging device 220, or is offset relative to the 3D model 212 by the same amount as the imaging device 220 is offset relative to the patient. In addition, the display device 210 may link the virtual imaging device and the imaging device 220 so that moving one device can send a signal to cause the other device to follow the same path. For example, if the user manipulates the virtual imaging device (e.g., rotates a virtual C-arm), the display device 210 may send a signal to cause the real-world imaging device 220 to follow a similar movement path (e.g., rotates an actual C-arm). According to an example, the display device 210 may display a prompt to the user before causing the imaging device 220 to follow the path, asking the user to confirm the movement path and / or displaying the movement path.

[0056] In another example, the user instructs the display device 210 to generate a virtual 2D view of the patient volume. For example, the 2D view 211 can be a virtual 2D view generated based on the 3D model 212, rather than a 2D x-ray image generated by the imaging device 220. According to an example, the user can move or position the virtual imaging device and cause the display device 210 to generate a predicted 2D view using the positioning and / or angle of the virtual imaging device. The user can then decide whether to (i) confirm the predicted 2D view by moving the imaging device 220 to a matching position and imaging the patient, or (ii) discard the predicted 2D view and move the virtual imaging device to a different position. In some such examples, the additional virtual determination using the virtual imaging device reduces the number of scans required using the imaging device 220, thereby reducing the exposure of the patient and / or others nearby to ionizing radiation.

[0057] Figure 3 Described in the above Figures 1 to 2B 1 or 200. In particular, the block diagram depicts an example architecture 300 of a system similar to the system implemented in the environment 100 or the environment 200 depicted in FIG. 1 . In particular, the block diagram depicts the system 300 including various input devices, such as an XR device 310 worn by a user O, a computer 320, and components 302. According to an example, the XR device 310 may include, be, or be similar to the system implemented in the environment 100 or the environment 200. Figures 1 to 2B 1 and / or display device 210 as depicted in FIG. Similarly, component 302 may include, be, or be similar to Figure 1 The robotically assisted platform 102 is depicted and may include or be communicatively coupled to a medical device (such as the medical device 104 or the medical device 204).

[0058] Component 302 can manipulate a medical instrument and / or other device including one or more sensors that can generate shape sensor data 355 representing the shape of the medical instrument within the patient's anatomy. According to an example, shape sensor data 355 can include data related to the shape of information collected by one or more shape sensors for the instrument (e.g., fiber optic shape sensors, EM sensors, etc.). Component 302 can stream the shape sensor data from component 302 to an intermediate computing device (such as computer 320) via a system data streaming application 350. In other examples, component 302 can receive and / or send shape sensor data 355 via a TCP / IP socket connection, a Wi-Fi connection, a Bluetooth connection, etc.

[0059] Computer 320 can receive shape sensor data 355 at streaming application 340 and can modify, update, and / or otherwise manipulate shape sensor data 355 to generate shape sensor data 345. Similarly, according to an example, computer 320 can capture video data from component 302 using a recording device (e.g., a capture card) in accordance with video processor 347, or can otherwise manipulate already captured video data using video processor 347. Similarly, computer 320 can retrieve 3D assets (3D assets) 349 including 3D models, 3D objects, etc. defined, generated, or otherwise provided by user O, a team associated with computer 320, a patient, etc. In some implementations, the 3D assets can include assets associated with the patient's volume (e.g., a model generated from 3D volume data, instrument video data, etc.), pre-generated internal models, and / or other similar data as described herein. Computer 320 similarly transmits the relevant data from streaming application 340 to XR device 310, and therefore to user O. According to an example, the computer 320 may similarly receive and / or send shape sensor data 355 via a TCP / IP socket connection, a Wi-Fi connection, a Bluetooth connection, or the like.

[0060] The XR device 310 receives relevant data, such as shape sensor data 345, from the computer 320 via the extended reality application 330. The XR device 310 can modify, update, and / or otherwise manipulate the shape sensor data 345 to generate XR shape sensor data 335, video data 336 (e.g., received from a medical device such as an endoscope, from a virtual medical device, etc. via a real-time transport protocol (RTP) such as WebRTC), and / or 3D assets 339 (e.g., models, targets, virtual keyboards, etc.), as described herein. In addition, the XR device 310 can generate and / or receive imaging modality data 338. According to an example, the XR device can generate and / or receive imaging modality data 338 based on the simulation in the XR application 330 and / or based on the imaging device associated with the component 302 (e.g., as described above with respect to Figure 2A and Figure 2B20). In examples where the XR device generates imaging modality data 338 based on one or more imaging modalities associated with the imaging device 220 (e.g., as described above), the imaging modality data 338 may be generated based on one or more imaging modalities associated with the imaging device 220. In examples where the XR device generates imaging modality data 338 based on a simulated imaging modality (e.g., from a virtual representation of the imaging device), the imaging modality data 338 may also be based on a desired position and / or orientation of the imaging device (e.g., as indicated by a user and / or depicted by the XR device as a virtual representation of the imaging device in virtual space). In examples where the XR device generates and / or receives imaging modality data 338 from an imaging device (e.g., imaging device 220), the imaging modality data 338 may include or be based on an image of the patient (e.g., an X-ray image, a CT scan, etc.) from the respective device.

[0061] Similarly, the XR device 310 may retrieve, render, generate, and / or otherwise display a 3D model to the user based on model data 332 based on XR shape sensor data 335, video data 336, 3D assets 339, user input 334, etc. The XR device may additionally generate a navigation interface 310R similar to the XR view 110R and / or navigation interface 210R based on data processed and / or generated by the XR application 330.

[0062] Figure 4 Depicts the Figure 3 The architecture and system are similar to the example architecture of system 400. Figure 4 In some examples, system 400 is accessed by multiple users, and wherein the XR devices and components are communicatively coupled directly, rather than through an intermediary computing device. In some examples, the system includes an XR device 410 and a component 402 used by at least user O. The XR device 410 and / or component 402 may each include, be, or be similar to the XR device 410 and component 402 described above. Figure 3 The XR device 310 and / or component 302 are depicted.

[0063] In some examples, component 402 may include system application 450, which is similar to the system application described above with respect to Figure 3The described system data streaming application 350 functions similarly. However, unlike component 302 and system data streaming application 350, component 402 and system application 450 can communicate directly and bidirectionally with XR device 410 and XR application 430 without using an intermediary computing device. Furthermore, component 402 can include a console 403 for a user to input, update, generate, and / or otherwise manipulate data received from XR device 410, generated by component 402, and the like. In some examples, console 403 receives console interactions 464 to manipulate component 402, console 403, displayed data in XR device 410, and the like. For example, console 403 can receive input from a user directly (e.g., via a controller, keyboard, mouse, etc. associated with console 403 and / or component 402) and / or indirectly (e.g., via one or more controllers, sensors, etc. of XR device 410) and can update component 402, console 403, system application 450, and / or any stored data accordingly.

[0064] Component 402 generates shape sensor data 455 similar to shape sensor data 355. In some examples, component 402 generates and / or receives additional data based on information from XR device 410, such as 3D data 431, 2D data 441, co-registration information between 3D data 431 and 2D data 441, and the like. Similarly, component 402 can generate video data 456 (e.g., from a medical device such as an endoscope, from a virtual medical device, etc., via a real-time transport protocol (RTP) such as WebRTC, UDP socket communication, etc.), system event data 458, imaging data 461, and the like. In some examples, system event data 458 is collected via an API associated with console 403 and / or component 402. According to an example, imaging data 461 can be 2D or 3D imaging data as described herein (e.g., CT data, CBCT data, fluoroscopy data, (radial) EBUS data, etc.). According to an example, the system application 450 may communicate data with the XR application 430 via a TCP / IP socket connection, a Wi-Fi connection, a Bluetooth connection, or the like.

[0065] XR application 430 can be used with Figure 3The described XR application 330 functions similarly. In particular, model data 432, user input 434, shape sensor data 435, video data 436, and / or 3D assets 439 can be similar to the corresponding data described above. Similarly, the XR application 430 can generate and / or receive 3D data 431 (e.g., patient volume data, preoperative data such as CT scan data, intraoperative data such as CBCT data, etc.) or 2D data 441 (preoperative data such as fluoroscopy data, intraoperative data such as R-EBUS data, etc.). The XR application 430 can also receive and / or generate user interface data 444 from the component 402 and / or the console 403, such as commands for medical devices associated with the component 402, or XR toolkit elements.

[0066] In some examples, the system 400 includes one or more additional users Q via the multiplayer synchronization module 445. In some such examples, the system 400 may include additional XR devices and / or computing devices similar to the XR device 410 for use by the additional users Q. In some such examples, the system 400 automatically displays a navigation interface 410R generated for the user O and the XR device 410. In other such examples, the system 400 does not register and / or accept commands or instructions from the additional users Q. Thus, the additional users Q are only able to observe. In other examples, the additional users Q may view a modified navigation interface (e.g., with fewer elements and / or information), and the additional users Q may modify the first (e.g., primary) navigation interface 410R to some extent (e.g., changing the viewing angle, changing the angle, switching various UI elements, etc.) without affecting it. In yet another example, the additional users Q may cause notifications to appear in the first navigation interface 410R (e.g., a chat window, an alert, a question, a highlight, etc.). Similarly, additional user Q and / or user O can annotate the navigation interface to leave notes, video markers, drawn objects, etc.

[0067] In some examples, the system 400 provides different navigation interfaces to additional users Q based on the roles and / or permissions granted to the users Q. For example, a student observer may only see 3D data 431 and 2D data 441 , while a member of the patient care team may see details about the patient's vital signs.

[0068] Next refer to Figure 5 , a flowchart depicting an example method 500 for generating a 3D navigation interface for a robotic-assisted medical procedure. Figure 1 The illustrated environment 100 and its components describe the method 500, but it will be understood that other similarly suitable devices and components may be used instead, including Figure 3 and / or Figure 4Those devices and components shown.

[0069] At block 502, the robotic-assisted platform 102 receives a 3D model representing a volume of a patient. According to examples, the 3D model can be a 3D model of a specific organ (e.g., a lung) or portion of an organ of the patient, a larger system (e.g., a respiratory system) of the patient, the entire patient anatomy, and the like. In some examples, a component of the environment 100 can generate the 3D model and transmit the model within the environment 100. In other examples, an element external to the environment 100 can generate the 3D model and transmit the model to the environment 100. According to examples, any of a computed tomography (CT) scanner, a cone-beam computed tomography (CBCT) scanner, a magnetic resonance imaging (MRI) scanner, a positron emission tomography (PET) scanner, a tomosynthesis device, and the like can generate the 3D model. Furthermore, the device generating the 3D model can generate the model prior to the medical procedure (e.g., immediately before, hours before, days before, etc.) or during the medical procedure.

[0070] At box 504, the robotic-assisted platform receives 2D data representing one or more 2D images corresponding to at least a portion of the patient's volume. In some examples, the 2D image may include a 2D x-ray image, such as a fluoroscopic image, that can be captured by a C-arm. Additionally or alternatively, the 2D image may include a synthetic 2D image generated based on the 3D model and representing the 2D x-ray image captured by the C-arm. According to an example, the C-arm may be associated with the environment 100, or may be a separate C-arm that sends the 2D image to the environment 100. The robotic-assisted platform 102 may additionally track the C-arm (e.g., C-arm movement, C-arm pose, C-arm position, etc.). The robotic-assisted platform 102 may track the C-arm via sensors located on the C-arm (e.g., using accelerometers, gyroscopes, position sensors, etc.) or via a head-mounted display (HMD), as described herein.

[0071] At block 506, the robotic-assisted platform 102 generates co-registered operational data that correlates the 3D model with the 2D data. The co-registered operational data may be specific data of interest that correlates the 2D data with the 3D data, such as the locations of similar and / or matching structures (e.g., entry / exit points, landmarks, joints, etc.). In another example, the co-registered operational data may include a pixel-by-pixel mapping between 2D data (e.g., a 2D image, a 2D coordinate system, etc.) and 3D data (e.g., a 3D model, a 3D coordinate system, etc.). Similarly, the co-registered operational data may include optical markers and / or fiducials for visual tracking via a 2D or 3D imaging sensor (e.g., an RGB camera, an infrared camera, etc.). In another example, the co-registered operational data may include a common reference system, which may be, for example, a surgical reference system for a user, a patient reference system for a patient, an observer reference system for an observer, etc. Thus, the co-registered operational data may enable the robotic-assisted platform 102 to orient, overlay, and / or otherwise generate components of a view, such as a 2D view, a 3D model, and the like.

[0072] In some examples, the robotic-assisted platform 102 can automatically generate co-registration data using machine learning, image analysis, optical recognition, and / or other similar technologies. In another example, the robotic-assisted platform 102 can prompt the user to interact with one or more targets on the 3D model and / or 2D data, and can generate co-registration data based on the interaction. According to an example, the interaction can be touching the 3D model and / or 2D data, dragging a virtual instrument to the location, moving a physical controller to the location, and / or any other similar method of interacting with the virtual environment. In another example, the robotic-assisted platform 102 can use 3D shape sensor data and / or external electromagnetic tracking data to perform co-registration as described herein. According to an example, the robotic-assisted platform 102 can continuously or dynamically update the co-registration based on data measured throughout the process (e.g., via visual recognition of landmarks within the 3D model based on 2D data, etc.).

[0073] At block 508, the robotic-assisted platform 102 generates a 3D navigation interface. In some examples, generating the 3D navigation interface includes generating a display of at least a portion of the 3D model based on the co-registered operational data. The robotic-assisted platform 102 may display the 3D model at a predetermined position relative to the patient or cause a display device to display the 3D model at a predetermined position relative to the patient. For example, the robotic-assisted platform 102 may be positioned above the patient (e.g., as shown in FIG. Figure 2A and Figure 2B ), adjacent to the patient, superimposed on the patient, etc., or with the display device positioned above the patient (e.g., as depicted in FIG. Figure 2A and Figure 2B , adjacent to the patient, overlaid with the patient, etc. In another example, the robotic-assisted platform 102 can cause the display device to adjust the position of the 3D model based on user input. For example, the robotic-assisted platform 102 can ensure that the display device displays the 3D model as it remains in the proper position above the patient as the user moves the view within the 3D navigation interface. Alternatively, the robotic-assisted platform 102 can move the 3D model display in response to receiving an instruction from the user to move the display (e.g., the user drags the display elsewhere, the user enters a specific command to change the positioning, etc.).

[0074] In another example, the robotic-assisted platform 102 generates a 3D navigation interface by additionally generating a display of one or more 2D images based at least on the co-registered operational data. The robotic-assisted platform 102 can generate a 3D navigation interface by orienting the 2D image so that the orientation of the 2D image is based at least on the 3D model. In this way, the robotic-assisted platform 102 can orient the 2D image and the 3D model so that the 2D image and the 3D model share an orientation from the user's perspective, thereby allowing the user to more easily identify shared locations, landmarks, etc. between the 2D image and the 3D model. Alternatively, the robotic-assisted platform 102 can orient the 2D image relative to the 3D model so that the 2D image rotates or tilts as the user moves the 3D model (e.g., a top view of the 2D image and a front view of the 3D model can still rotate as the 3D model rotates).

[0075] In another example, the 3D navigation interface includes a live video feed of an instrument from the environment 100. For example, the 3D navigation interface may include a feed from one or more imaging devices associated with a flexible extension device as described herein. According to an example, the live video feed may be a 2D video feed or a 3D video feed. In addition, the robotic-assisted platform 102 may cause the display device to align a view of the live video feed with the 3D model. For example, the robotic-assisted platform 102 may generate a 3D navigation interface such that the video feed and the 3D model share an orientation, as described above with respect to the 2D view. Additionally or alternatively, the robotic-assisted platform 102 may overlay some or all of the video feed with the 3D model, or otherwise indicate where the video feed is to be displayed with respect to the 3D model. According to an example, the robotic-assisted platform 102 may display the video feed in various orientations. For example, the robotic-assisted platform 102 may be directly behind the patient and / or the 3D model (e.g., similar to the 3D model described above with respect to the 2D view). Figure 2A and Figure 2B 2D instrument video 219 as described above), according to the current pose orientation of the virtual instrument (e.g., similar to the Figure 2A and Figure 2BThe video feed may be oriented in the described 2D view 218 ), oriented according to the user's position, oriented above and / or below the 3D model, oriented superimposed with the 3D model, etc.

[0076] At block 510, the robotic-assisted platform 102 causes a display device to display the 3D navigation interface to a user. In some examples, the display device is a head-mounted display device (HMD). In such examples, the robotic-assisted platform 102 causes the HMD to display the 3D navigation interface in the form of extended reality (XR). According to examples, the robotic-assisted platform 102 may cause the HMD to display the 3D navigation interface as (i) a mixed reality (MR) navigation interface, (ii) an augmented reality (AR) navigation interface, or (iii) a virtual reality (VR) navigation interface.

[0077] In some examples, the robotic-assisted platform 102 also receives sensor data from one or more sensors configured to generate data associated with the patient, an internal volume of the patient, a portion of the interior of the patient, etc. According to an example, the robotic-assisted platform 102 can receive 2D sensor data or 3D sensor data. The sensor data can include any of the following: computed tomography (CT) data, cone-beam computed tomography (CBCT) data, catheter data, endoscopic video data, magnetic resonance imaging (MRI) data, C-arm data, radial endobronchial ultrasound (EBUS) data, a combination of data types, and / or any other such data as described herein.

[0078] In further examples, the robotic-assisted platform 102 may additionally or alternatively receive navigation information about a patient, an internal volume of a patient, a portion of an internal volume of a patient, and / or the like. In some such examples, the navigation information may include historical navigation within the patient. Historical navigation may include navigation history during the current session, navigation history during past sessions with the same patient, generalized and / or standardized navigation history for a common volume (e.g., various navigation paths taken by physicians for the lung under similar circumstances), and / or the like. According to an example, the robotic-assisted platform 102 may cause a display device to display the navigation history as paths drawn or otherwise generated along a 3D model, each path indicating a past navigation path. In some such examples, each path includes an indication of when the navigation occurred, relative to other paths, and / or according to the actual time or date of the navigation. Additionally or alternatively, the historical navigation may include the history of past sessions as a separate map, as a list of landmarks passed, as a description of distance traveled or turns taken, and / or the like. In other such examples, the robotic-assisted platform 102 may cause a display device to display a subset of the navigation history based on user preferences, user instructions, the current user task, and / or the like.

[0079] Similarly, the navigation history may additionally or alternatively include a navigation path representing a recommended path for the instrument in the patient's volume. According to an example, a physician may generate a navigation path using the robotic-assisted platform 102 or an external computing device before or during a medical procedure. In another example, the robotic-assisted platform 102 may generate a navigation path based on instructions from the user, navigation history, etc. using a trained neural network and / or machine learning technique for predicting a preferred path for the user. In some examples, the navigation information additionally or alternatively includes visual or auditory elements representing landmarks within the patient (e.g., distinct and / or easily identifiable locations within the patient) and / or one or more sampled tissue locations (e.g., locations where an instrument of the robotic-assisted platform 102 has sampled tissue).

[0080] In another example, the robotic-assisted platform 102 and / or the display device modify at least a portion of the 3D navigation interface based on control input received from the user. The control input can modify the 3D navigation interface directly (e.g., moving interface elements, zooming in or out, shifting the user's perspective or view, etc.) or in response to movement of physical elements (e.g., modifying the model display in response to the user moving an instrument within the patient, etc.).

[0081] According to examples, a user may provide control input using a controller, trackball, keyboard, mouse, touchscreen device, touch sensor, motion sensor, accelerometer, gyroscope, position sensor, etc. Thus, a user may provide control input through input to a physical device (e.g., pressing a button, moving a joystick, rolling a trackball, etc.) and / or by interacting with a virtual object. For example, the robotic-assisted platform 102 may detect a user moving their hand to interact with an object displayed in a virtual space via a display device and may accordingly cause the object to move within the user's field of view. In some examples, a user may drag a virtual representation of an instrument within the boundaries of a 3D navigation interface to indicate where the corresponding instrument will move within the patient volume. In another example, a user may tap an indication of a virtual target in the patient volume, and the robotic-assisted platform 102 may cause an instrument, including a sensor, to follow a path to the indicated target. In some examples, the robotic-assisted platform 102 causes the instrument to follow a path based on navigation information. In other examples, the robotic-assisted platform 102 may automatically generate a path in response to a user indication using navigation history, landmarks, tissue sample locations, etc., and follow the generated path.

[0082] In some examples, the robotic-assisted platform 102 modifies the visibility of portions of the 3D navigation interface based on the current task, user context, and the like. For example, the robotic-assisted platform 102 may determine that at least one element is distracting and / or potentially distracting the user. In response, the robotic-assisted platform 102 may dim the visibility of the element (e.g., by reducing opacity, lowering lighting, lightening the element's color, etc.) to allow the user to better focus on other elements. In another example, the robotic-assisted platform 102 may instead change the XR type in response to determining that an element is distracting the user. For example, the robotic-assisted platform 102 may determine that another individual in the user's view is distracting the user and may change from an AR view to a VR view that does not display the other individual. The robotic-assisted platform 102 may determine that an object or element is distracting or potentially distracting the user based on user indications, such as through a machine learning algorithm trained to predict distractions. In some examples, dimming the visibility of an element or a wider portion of the interface may occur naturally with and / or in conjunction with changing the XR type.

[0083] In yet another example, while a first user performs a medical procedure, an additional second user (e.g., a student, a supervising physician, an observer, etc.) can observe a 3D navigation interface. In some such examples, the robotic-assisted platform 102 generates a second 3D navigation interface and displays the second interface to the second user. According to an example, the second 3D navigation interface can include a reduced set of information, such as removing past navigation history, specific patient information, etc. Similarly, the robotic-assisted platform 102 can modify the second 3D navigation interface in response to input from the first user, but can refrain from modifying the first 3D navigation interface and / or the second 3D navigation interface in response to input from the second user.

[0084] Next refer to Figure 6 , a flowchart depicting an example method 600 for generating a 3D navigation interface for a robotic-assisted medical procedure. Figure 1 The method 600 is described with reference to the environment 100 and its components shown in FIG. , but it will be understood that other similarly suitable imaging devices and components may be used instead.

[0085] At box 602, the robotic-assisted platform 102 receives a 3D model representing a volume of a patient. According to an example, the 3D model can be a 3D model of a specific organ of the patient (e.g., a lung) or a portion of an organ, a larger system of the patient (e.g., a respiratory system), the entire patient anatomy, etc. In some examples, components of the robotic-assisted platform 102 can generate the 3D model and transmit the model within the environment 100. In other examples, elements external to the environment 100 can generate the 3D model and transmit the model to the environment 100. According to an example, any of a computed tomography (CT) scanning device, a cone-beam computed tomography (CBCT) scanning device, a magnetic resonance imaging (MRI) scanning device, a positron emission tomography (PET) scanning device, a tomosynthesis device, etc. can generate the 3D model. In addition, the device that generates the 3D model can generate the model before the medical procedure (e.g., immediately before the medical procedure, a few hours before, a few days before, etc.) or during the medical procedure. Similar to Figure 5 , the system can display the 3D model above the patient and / or the view of the patient, adjacent to the patient and / or the view, superimposed with the patient and / or the view, etc.

[0086] In some examples, the view of the patient is a view of a physical patient as seen via a display device, such as in an AR view. In other examples, the view of the patient is a representation of the patient, such as a representation generated in a VR view. In other examples, the 3D model includes a virtual device that is rotated to match the positioning of the corresponding physical device. In some such examples, the virtual device is offset from the physical device by the same distance that the 3D model is offset from the patient's view. In other examples, the virtual device is superimposed on the physical device even if the rest of the 3D model is not superimposed on the patient's view.

[0087] At box 604, the robotic-assisted platform 102 generates a 2D view of the patient's volume. In some examples, the 2D view represents a 2D imaging modality when positioned at a specific device projection angle. For example, the 2D view can be an x-ray image or represent an x-ray image taken by an imaging modality (e.g., a C-arm) at a given angle. In another example, the 2D view includes at least one of a synthetic fluoroscopic image, a 2D x-ray image (e.g., from a C-arm), an EBUS image, etc. Similarly, the imaging modality can include at least one of: (i) C-arm imaging, (ii) CT imaging, (iii) MRI imaging, (iv) CBCT imaging, (v) EBUS imaging, or (vi) any other similar suitable imaging modality.

[0088] In some examples, the robotic-assisted platform 102 receives additional images (e.g., CT images, etc.) and / or generates 2D views such that the images deviate from the patient volume. In response to determining that the images deviate from the patient volume, the robotic-assisted platform 102 may send a request for an updated image using the same or a different imaging modality (e.g., a CBCT image to replace the CT image).

[0089] At block 606, the robotic-assisted platform 102 determines a display orientation of the 2D view relative to the user. In some examples, the robotic-assisted platform 102 determines the display orientation of the 2D view based on input from the user, based on a registration or co-registration process, automatically based on the 3D model, based on the patient position, etc.

[0090] At block 608, the robotic-assisted platform 102 registers the 2D view to the 3D model such that both the 2D view and the 3D model share a display orientation. In some examples, the robotic-assisted platform 102 orients the 2D view and the 3D model relative to the user such that the shared display orientation is based on the user's perspective. For example, the robotic-assisted platform 102 can orient the 2D view and the 3D model such that the shared orientation depicts a first-person view from the user's perspective, a third-person view from the user's perspective (e.g., over the shoulder), a top-down view of the patient from the user's perspective, a task view representing at least a portion of the patient's volume based on a current task being performed by the user, or another similar orientation.

[0091] In some examples, the robotic-assisted platform 102 can similarly scale or otherwise modify the 3D model and / or 2D view in a shared orientation based on the user and / or user activity. For example, the robotic-assisted platform 102 can scale the 3D model relative to the speed of movement of the elongated flexible device and / or another instrument, the navigation context of the patient volume, the user's indicated accuracy preference, the current user task, etc. In some such examples, scaling the 3D model includes modifying the navigation path displayed to the user based on the speed of movement (e.g., by changing the spacing between component dashes of a dashed line indicating the navigation path based on the speed of movement). For example, as the user increases the speed of movement of the elongated flexible device, the robotic-assisted platform 102 can decrease the spacing between component dashes so that the user continues at a safe speed. In some such examples, the robotic-assisted platform 102 indicates that the user is moving at an unsafe or maximum speed when the dashed line transitions to a solid line.

[0092] At block 610, the robotic-assisted platform 102 causes the display device to simultaneously display the 2D view and the 3D model to the user according to the shared display orientation. For example, the robotic-assisted platform 102 may display the 2D view above the 3D model, below the 3D model, behind the 3D model, superimposed with the 3D model, etc. (e.g., similar to the above description of the 2D view). Figure 2A and Figure 2B 2D image 211 and 3D model 212 described. In some examples, the display device is an HMD, and the HMD displays the 2D view and 3D model in an XR manner such as MR, AR, or VR. In other examples, the display device is a 3D computing device.

[0093] At block 612, the robotic-assisted platform 102 receives a control input from the user. According to an example, the control input may include an input for moving a device associated with the robotic-assisted platform 102 (e.g., an instrument, an external imaging device, etc.). For example, the control input may be a control input for rotating a C-arm. In some examples, by moving the device, both the physical device and the virtual representation of the device move to reflect the input. In other examples, the robotic-assisted platform 102 moves the virtual representation of the device, as indicated in more detail below with respect to block 614, but does not move the physical device until confirmation and / or instructions are received from the user.

[0094] In some examples, the control input comprises user manipulation (e.g., direct interaction) of a virtual representation of the elongated flexible device (e.g., a virtual device as described above with respect to block 604). In some such examples, the robotic-assisted platform 102 causes the elongated flexible device to move based on the user manipulation of the virtual representation in response to receiving the control input. In other examples, the control input comprises user interaction with a physical location or a virtual representation of a physical location, such as tapping a virtual representation of a location along a 3D model. In some such examples, the robotic-assisted platform 102 automatically generates a virtual path to the location in response to receiving the control input. The robotic-assisted platform 102 may prompt the user to confirm the path and / or cause the elongated flexible device to follow the path.

[0095] At block 614, the robotic-assisted platform 102 updates at least the 2D view of the patient's volume based on the control input. In examples where the control input includes input for moving the device, the robotic-assisted platform 102 updates the 2D view to correspond to the new position of the device. For example, the robotic-assisted platform 102 may update the 2D view after controlling the C-arm to rotate to a new angle. In other examples, the robotic-assisted platform 102 updates the 3D model or both the 2D view and the 3D model. In some such examples, the system updates the 2D view and the 3D model while maintaining a shared orientation between the two. In another example, the user indicates to the robotic-assisted platform 102 that the robotic-assisted platform 102 should modify one of the 2D view and the 3D model, but not the other. Depending on the example, the robotic-assisted platform 102 may modify one of the 2D view and the 3D model while maintaining the shared orientation (e.g., where the modification does not involve changing the orientation), or the robotic-assisted platform 102 may break the shared orientation (e.g., where the modification does involve changing the orientation). Similarly, the robotic-assisted platform 102 may cause a display device to display instructions on how to rotate / orient at least one of the 2D view or the 3D model. In some examples, the indication may be displayed automatically after the user elects to break a shared orientation, in response to a user indication, or the like.

[0096] In some examples, the robotic-assisted platform 102 embeds the histological sample image into the 3D model. For example, the robotic-assisted platform 102 can capture a 2D histological sample image at a fork (or any other location) in the model, and the robotic-assisted platform 102 can embed the 2D histological sample at the fork (or any other location) in the model. In this way, the user can quickly and accurately determine where a particular image indicates in the 3D model.

[0097] In another example, the robotic-assisted platform 102 generates a procedure report in response to an instruction from a user. According to an example, the instruction can be or include a request from the user, an indication of completion of the medical procedure, an indication of the start of the medical procedure, etc. The procedure report can include navigation information representing at least one of a 2D view or a 3D model of the patient's volume. According to an example, the navigation information can include: (i) navigation guidance information representing a recommended path for the sensor in the patient's volume, (ii) historical navigation representing past navigation of the sensor in the patient's volume, (iii) a shared display orientation of the 2D view and the 3D model, (iv) a slice of the 3D model (e.g., a 2D image of a single view of the 3D model), (v) histological data representing at least one of the 2D view or the 3D model, (vi) or any other similar information. Similarly, the navigation information can include any combination of the above items.

[0098] Similar to the above Figure 5, the robotic-assisted platform 102 can enable another second user to view the displayed 2D view and / or 3D model. In some examples, the robotic-assisted platform 102 generates a second user orientation for the second user that is different from the first user orientation. Subsequently, the robotic-assisted platform 102 can cause a second display device (e.g., another HMD, a computing device, etc.) to display the 2D view and / or 3D model to the second user based on the second user orientation. In other examples, the robotic-assisted platform 102 displays the first orientation to the second user but does not respond to control inputs from the second user.

[0099] It should be understood that although the above description is about specific components Figure 5 and Figure 6 , but other components may perform some of the above functions as appropriate. For example, in some examples, Figure 1 The HMD 110 may perform some of the functions described. Similarly, in other examples, the HMD 110 may perform some of the functions described. Figure 1 、 Figure 3 or Figure 4 The other components described may perform the functions described above.

[0100] Figures 7 to 9B Depicted is a diagram of a medical system that, in some examples, can be used to manipulate a medical device according to any of the methods and systems described above.

[0101] Figure 7 is a simplified diagram of a medical system 700 according to some examples. Medical system 700 can be used, for example, in surgical, diagnostic (e.g., biopsy) or therapeutic (e.g., ablation, electroporation, etc.) procedures. Although some examples of such procedures are provided herein, any reference to medical or surgical instruments and medical or surgical methods is non-limiting. The systems, instruments, and methods described herein can be used with animals, human cadavers, animal cadavers, parts of human or animal anatomy, non-surgical diagnostics, and in industrial systems, general or specialized robotic systems, general or specialized teleoperated systems, or robotic medical systems.

[0102] like Figure 7As shown, medical system 700 may include a manipulator assembly 702 that controls the operation of medical instruments 704 when performing various procedures on patient P. Medical instruments 704 may extend through openings within patient P's body and into internal locations within patient P. Manipulator assembly 702 may be teleoperated, non-teleoperated, or a hybrid of teleoperated and non-teleoperated, having one or more degrees of freedom of motion that may be motorized and / or one or more degrees of freedom that may be non-motorized (e.g., manually operated). Manipulator assembly 702 may be mounted to and / or positioned near patient table T. A main assembly 706 enables operator O (e.g., a surgeon, clinician, physician, or other user) to control manipulator assembly 702. In some examples, main assembly 706 enables operator O to view the surgical site or other graphical or informational displays. In some examples, manipulator assembly 702 may be excluded from medical system 700, and instruments 704 may be directly controlled by operator O. In some examples, manipulator assembly 702 can be manually controlled by operator O. Direct operator controls can include various handles and operator interfaces for handheld operation of instrument 704 .

[0103] The main assembly 706 can be located at a surgeon's console near a patient table T where the patient P is located (e.g., in the same room as the patient table T), such as to the side of the patient table T. In some examples, the main assembly 706 is located remote from the patient table T, such as in a different room or a different building than the patient table T. The main assembly 706 can include one or more control devices for controlling the manipulator assembly 702. The control devices can include any number of various input devices, such as a joystick, a trackball, a scroll wheel, a directional pad, buttons, a data glove, a trigger gun, a hand controller, a voice recognition device, a motion or presence sensor, etc. In some examples, the main assembly 706 can be or include an extended reality (XR) device, such as a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, or any other such device as described herein.

[0104] The manipulator assembly 702 supports the medical device 704 and may include a kinematic structure of a linkage that provides a set structure. The linkage may include one or more non-servo controlled linkages (e.g., one or more linkages that can be manually positioned and locked in place) and / or one or more servo controlled linkages (e.g., one or more linkages that can be controlled in response to commands from, for example, a control system 712). The manipulator assembly 702 may include a plurality of actuators (e.g., motors) that drive inputs on the medical device 704 in response to commands from, for example, the control system 712. The actuator may include a drive system that, when coupled to the medical device 704, causes the medical device 704 to move in various ways. For example, one or more actuators may advance the medical device 704 into a natural or surgically created anatomical orifice. The actuator may control the engagement of the medical device 704, for example, by moving the distal end (or any other portion) of the medical device 704 with multiple degrees of freedom. These degrees of freedom may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). One or more actuators may control the rotation of the medical device about the longitudinal axis. The actuator may also be used to move an engageable end effector of the medical device 704 (e.g., for grasping tissue in the jaws of a biopsy device, etc.), or may be used to move or otherwise control a tool inserted within the medical device 704 (e.g., an imaging tool, an ablation tool, a biopsy tool, an electroporation tool, etc.). According to an example, the manipulator assembly 702 may include or may be as described above with respect to Figures 1 to 4 Similarly, the medical device 704 may be or may include a robotic assisted platform as described above with respect to Figures 1 to 4 Components of the medical device described.

[0105] Medical system 700 may include a sensor system 708 having one or more subsystems for receiving information about manipulator assembly 702 and / or medical device 704. Such subsystems may include a position sensor system (e.g., using an electromagnetic (EM) sensor or other type of sensor that detects position or location); a shape sensor system for determining the position, orientation, speed, velocity, pose, and / or shape of one or more segments along the flexible body and / or the distal end of medical device 704; a visualization system (e.g., using a color imaging device, an infrared imaging device, an ultrasound imaging device, an x-ray imaging device, a fluoroscopic imaging device, a computed tomography (CT) imaging device, a magnetic resonance imaging (MRI) imaging device, or some other type of imaging device) for capturing images, for example, from the distal end of medical device 704 or from some other location; and / or an actuator position sensor (e.g., a resolver, an encoder, a potentiometer, etc.) that describes the rotation and / or orientation of an actuator that controls medical device 704.

[0106] The medical system 700 may include a display system 710 for displaying an image or representation of a surgical site and a medical instrument 704. The display system 710 and the main assembly 706 may be oriented so that the surgeon O can control the medical instrument 704 and the main assembly 706 using the perception of telepresence. In some examples, although the display system 710 and the main assembly 706 are Figure 7 706. Although depicted as separate blocks in FIG. 70, both display system 710 and main component 706 may be part of the same device and / or operating control system.

[0107] In some examples, the medical device 704 may include a visualization system that may include an image capture component that records simultaneous or real-time images of the surgical site and provides the images to the operator O via one or more displays of the display system 710. The image capture component may include various types of imaging devices. The simultaneous images may be two-dimensional or three-dimensional images, for example, captured by an endoscope positioned within the anatomical surgical site. In some examples, the visualization system may include an endoscope component that may be integrally or removably coupled to the medical device 704. Additionally or alternatively, a separate endoscope attached to a separate manipulator assembly may be used with the medical device 704 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as the control system 712.

[0108] Display system 710 can also display images of the surgical site and medical instrument, which can be captured by the visualization system. In some examples, medical system 700 provides operator O with a telepresence perception. For example, images captured by an imaging device at the distal portion of medical instrument 704 can be presented by display system 710 to provide operator O with a perception of the distal portion of medical instrument 704. Input provided by operator O to master assembly 706 can move the distal portion of medical instrument 704 in a manner corresponding to the nature of the input (e.g., when the trackball is rolled to the right, the distal tip turns to the right), resulting in a corresponding change in the perspective of the image captured by the imaging device at the distal portion of medical instrument 704. Thus, while medical instrument 704 is moved using master assembly 706, operator O's telepresence perception is maintained. Operator O can manipulate the hand controls of master assembly 706 and medical instrument 704 as if viewing the workspace in a substantially real-life setting, simulating the operator's experience of physically manipulating medical instrument 704 from within the patient's anatomy.

[0109] In some examples, the display system 710 can present a virtual image of the surgical site created using image data recorded preoperatively (e.g., before a procedure is performed by the medical device system 800) or intraoperatively (e.g., concurrently with a procedure performed by the medical device system 800), such as image data created using computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), fluoroscopy, thermal imaging, ultrasound, optical coherence tomography (OCT), thermography, impedance imaging, laser imaging, nanotube X-ray imaging, etc. The virtual image can include an image in two dimensions, three dimensions, or higher dimensions (e.g., including time-based or rate-based information). In some examples, one or more models are created based on the preoperative or intraoperative image data sets, and the virtual image is generated using the one or more models.

[0110] In some examples, for the purpose of imaging-guided medical procedures, the display system 710 can display a virtual image generated based on the tracked position of the medical instrument 704. For example, the tracked position of the medical instrument 704 can be registered (e.g., dynamically referenced) with a model generated using preoperative or intraoperative images, where different portions of the model correspond to different locations of the patient's anatomy. As the medical instrument 704 moves through the patient's anatomy, the registration is used to determine the portion of the model that corresponds to the position and / or perspective of the medical instrument 704, and a virtual image is generated using the determined portion of the model. This can be done to present the operator O with a virtual image of the internal surgical site corresponding to the tracked position of the medical instrument 704 from the viewpoint of the medical instrument 704.

[0111] The medical system 700 may also include a control system 712, which may include processing circuitry that implements some or all of the methods or functions discussed herein. The control system 712 may include at least one memory and at least one processor for controlling the operation of the manipulator assembly 702, the medical device 704, the main assembly 706, the sensor system 708, and / or the display system 710. The control system 712 may include instructions (e.g., a non-transitory machine-readable medium storing instructions) that, when executed by the at least one processor, configure the one or more processors to implement some or all of the methods or functions discussed herein. Although the control system 712 is Figure 7 702. Although shown as a single block in FIG. 703, control system 712 may include two or more separate data processing circuits, where a portion of the processing is performed at manipulator assembly 702, another portion of the processing is performed at master assembly 706, and so on. In some examples, control system 712 may include other types of processing circuitry, such as an application specific integrated circuit (ASIC) and / or a field programmable gate array (FPGA). Control system 712 may be implemented using hardware, firmware, software, or a combination thereof.

[0112] In some examples, control system 712 can receive feedback, such as force and / or torque feedback, from medical device 704. In response to the feedback, control system 712 can transmit a signal to main assembly 706. In some examples, control system 712 can transmit a signal instructing one or more actuators of manipulator assembly 702 to move medical device 704. In some examples, control system 712 can transmit information about the feedback to display system 710 for presentation or perform other types of actions based on the feedback.

[0113] The control system 712 may include a virtual visualization system to provide navigation assistance to the operator O when controlling the medical device 704 during an image-guided medical procedure. Virtual navigation using the virtual visualization system can be based on a preoperative or intraoperative dataset of the anatomical passages of the patient P. The control system 712 or a separate computing device can use programming instructions, alone or in combination with operator input, to convert the recorded images into a model of the patient's anatomical structure. The model can include a two-dimensional or three-dimensional synthetic representation of a segmentation of a portion or entire anatomical organ or anatomical region. The image dataset can be associated with the synthetic representation. The virtual visualization system can obtain sensor data from the sensor system 708 for calculating the (e.g., approximate) position of the medical device 704 relative to the anatomical structure of the patient P. The sensor system 708 can be used to register and display the medical device 704 with the preoperative or intraoperative recorded images. For example, PCT Publication WO 2016 / 161298 (published on December 1, 2016 and entitled "Systems and Methods of Registration for Image Guided Surgery") discloses an example system, which is incorporated herein by reference in its entirety.

[0114] During the virtual navigation process, the sensor system 708 can be used to calculate the (e.g., approximate) position of the medical device 704 relative to the anatomical structure of the patient P. This position can be used to generate both a macro-level (e.g., external) tracking image of the anatomical structure of the patient P and a virtual internal image of the anatomical structure of the patient P. The system can include one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to align and display the medical device and the preoperatively recorded medical image. For example, U.S. Patent No. 8,300,131 (filed on May 13, 2011 and entitled "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery") discloses an example system, which is incorporated herein by reference in its entirety.

[0115] The medical system 700 may also include operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some examples, the medical system 700 may include more than one manipulator assembly and / or more than one main assembly. The exact number of manipulator assemblies may depend on factors such as the medical procedure and space limitations within the operating room. Multiple main assemblies may be located in the same location or may be positioned in separate locations. Multiple main assemblies may enable more than one operator to control one or more manipulator assemblies in various combinations.

[0116] Figure 8A 800 is a simplified diagram of a medical device system 800 according to some examples. The medical device system 800 includes a flexible extension device 802 (also referred to as extension device 802), a drive unit 804, and a medical tool 826, which together serve as examples of a medical device 704 of the medical system 700. The medical system 700 can be a teleoperation system, a non-teleoperation system, or a hybrid teleoperation system and a non-teleoperation system, as described with reference to FIG. Figure 7 The visualization system 831, tracking system 830 and navigation system 832 are also described. Figure 8A , and is an example component of the control system 712 of the medical system 700. In some examples, the medical instrument system 800 can be used in non-teleoperated exploratory procedures or in procedures involving conventional manually operated medical instruments (e.g., endoscopy). The medical instrument system 800 can be used to collect (e.g., measure) a set of data points corresponding to locations within an anatomical passage of a patient (e.g., patient P).

[0117] Extension device 802 is coupled to drive unit 804. Extension device 802 includes a channel 821 through which a medical tool 826 can be inserted. Extension device 802 navigates within the patient's anatomy to deliver medical tool 826 to the surgical site. Extension device 802 includes a flexible body 816 having a proximal end 817 and a distal end 818. In some examples, flexible body 816 can have an outer diameter of approximately 3 mm. Other flexible bodies can have larger or smaller outer diameters.

[0118] The medical device system 800 may include a tracking system 830 for determining the position, orientation, speed, velocity, pose, and / or shape of the flexible body 816 at the distal end 818 and / or along one or more segments 824 of the flexible body 816, as will be described in further detail below. The tracking system 830 may include one or more sensors and / or imaging devices. The flexible body 816 (e.g., the length between the distal end 818 and the proximal end 817) may include multiple segments 824. The tracking system 830 may be implemented using hardware, firmware, software, or a combination thereof. In some examples, the tracking system 830 is Figure 7 A portion of control system 712 is shown.

[0119] The tracking system 830 can track the distal end 818 and / or one or more segments 824 of the flexible body 816 using a shape sensor 822. The shape sensor 822 can include an optical fiber aligned with the flexible body 816 (e.g., disposed within an internal channel of the flexible body 816 or mounted externally along the flexible body 816). In some examples, the optical fiber can have a diameter of approximately 800 μm. In other examples, the diameter can be larger or smaller. The optical fiber of the shape sensor 822 can form a fiber optic bend sensor for determining the shape of the flexible body 816. The optical fiber, including a fiber Bragg grating (FBG), can be used to provide strain measurements of the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions that may be suitable for use in some examples are described in U.S. Patent Application Publication No. 8006 / 0013523 (filed on July 13, 2005 and entitled “Fiber Optic Position and Shape Sensing Device and Method Relating Thereto”), U.S. Patent No. 7,772,541 (filed on March 12, 2008 and entitled “Fiber Optic Position and / or Shape Sensing Based on Rayleigh Scatter”), and U.S. Patent No. 8,773,350 (filed on September 2, 2010 and entitled “Optical Position and / or Shape Sensing”), which are incorporated herein by reference in their entireties. In some examples, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering.

[0120] In some examples, other techniques can be used to determine the shape of flexible body 816. For example, a history of the position and / or pose of distal end 818 of flexible body 816 can be used to reconstruct the shape of flexible body 816 over time intervals (e.g., as flexible body 816 advances or retracts within the patient's anatomy). In some examples, tracking system 830 can alternatively and / or additionally use position sensor system 820 to track distal end 818 of flexible body 816. Position sensor system 820 can be a component of an EM sensor system, wherein position sensor system 820 includes one or more position sensors. Although position sensor system 820 is shown proximate distal end 818 of flexible body 816 to track distal end 818, the number and position of position sensors of position sensor system 820 can vary to track different regions along flexible body 816. In one example, the position sensor includes a conductive coil that can be subjected to an externally generated electromagnetic field. Each coil of position sensor system 820 can generate an induced electrical signal having characteristics that depend on the position and orientation of the coil relative to the externally generated electromagnetic field. Position sensor system 820 can measure one or more position coordinates and / or one or more orientation angles associated with one or more portions of flexible body 816. In some examples, position sensor system 820 can be configured and positioned to measure six degrees of freedom, such as three position coordinates X, Y, and Z and three orientation angles indicating pitch, yaw, and roll of a base point. In some examples, position sensor system 820 can be configured and positioned to measure five degrees of freedom, such as three position coordinates X, Y, and Z and two orientation angles indicating pitch and yaw of a base point. Further description of position sensor systems that may be applicable in some examples is provided in U.S. Patent No. 6,380,432 (filed on August 11, 1999 and entitled "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0121] In some examples, tracking system 830 can alternatively and / or additionally rely on a collection of stored pose, position, and / or orientation data for points of elongated device 802 and / or medical tool 826 captured during one or more cycles of alternating motion (e.g., breathing). This stored data can be used to develop shape information about flexible body 816. In some examples, a series of position sensors (not shown), such as EM sensors like those in position sensors 820 or some other type of position sensors, can be positioned along flexible body 816 and used for shape sensing. In some examples, a history of data from one or more of these position sensors acquired during surgery can be used to represent the shape of elongated device 802, particularly in situations where anatomical pathways are typically static.

[0122] Figure 8B 8 is a simplified diagram of a medical tool 826 within an extension device 802 according to some examples. Flexible body 816 of extension device 802 may include a channel 821 sized and shaped to accommodate medical tool 826. In some examples, medical tool 826 may be used for procedures such as diagnosis, imaging, surgery, biopsy, ablation, illumination, irrigation, suction, electroporation, and the like. Medical tool 826 may be deployed through channel 821 of flexible body 816 and operated at a surgical site within an anatomical structure. Medical instrument 826 may be, for example, an image capture probe, a biopsy tool (e.g., a needle, grasper, brush, etc.), an ablation tool (e.g., a laser ablation tool, a radiofrequency (RF) ablation tool, a cryoablation tool, a thermal ablation tool, a heated liquid ablation tool, etc.), an electroporation tool, and / or another surgical, diagnostic, or therapeutic tool. In some examples, medical tool 826 may include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, and the like. Other end types of end effectors may include, for example, forceps, graspers, scissors, staplers, clip appliers, etc. Other end effectors may also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, etc.

[0123] Medical tool 826 can be a biopsy tool for removing a sample tissue or cell sample from a target anatomical location. In some examples, the biopsy tool is a flexible needle. The biopsy tool can also include a sheath that surrounds the flexible needle to protect the needle and the inner surface of channel 821 when the biopsy tool is located within channel 821. Medical tool 826 can be an image capture probe that includes a distal portion having a stereoscopic or monoscopic camera that can be positioned at or near the distal end 818 of flexible body 816 for capturing images (e.g., still or video images). The captured images can be processed by visualization system 831 for display and / or provided to tracking system 830 to support tracking of distal end 818 of flexible body 816 and / or one or more of segments 824 of flexible body 816. The image capture probe can include a cable for transmitting captured image data, the cable being coupled to an imaging device at the distal portion of the image capture probe. In some examples, the image capture probe can include a fiber optic bundle coupled to a more proximal imaging device of the visualization system 831, such as a fiberscope. The image capture probe can be single-spectral or multi-spectral, for example, capturing image data in one or more of the visible, near-infrared, infrared, and / or ultraviolet spectrums. The image capture probe can also include one or more light emitters that provide illumination to facilitate image capture. In some examples, the image capture probe can use ultrasound, x-ray, fluoroscopy, CT, MRI, or other types of imaging techniques.

[0124] In some examples, an image capture probe is inserted into the flexible body 816 of the elongated device 802 to facilitate visual navigation of the elongated device 802 to the surgical site, and then the image capture probe is replaced within the flexible body 816 with another type of medical tool 826 for performing the procedure. In some examples, the image capture probe can be positioned within the flexible body 816 of the elongated device 802 along with another type of medical tool 826 to facilitate simultaneous image capture and tissue intervention, for example, within the same channel 821 or in different channels. The medical tool 826 can be advanced from the opening of the channel 821 to perform the procedure (or some other function) and then retracted into the channel 821 when the procedure is complete. The medical tool 826 can be removed from the proximal end 817 of the flexible body 816 or from another optional instrument port (not shown) along the flexible body 816.

[0125] In some examples, the extension device 802 can include integrated imaging capabilities rather than utilizing a removable image capture probe. For example, an imaging device (or fiber optic bundle) and a light emitter can be located at the distal end 818 of the extension device 802. The flexible body 215 can include one or more dedicated channels for carrying cables and / or optical fibers between the distal end 818 and the visualization system 831. In this manner, the medical device system 800 can perform imaging and tool operations simultaneously.

[0126] In some examples, the medical tool 826 can be capable of controllable articulation. The medical tool 826 can house a cable (also referred to as a pull wire), linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of the medical tool 826, such as those discussed herein with respect to the flexible extension device 802. The medical tool 826 can be coupled to the drive unit 804 and the manipulator assembly 702. In these examples, the extension device 802 can be excluded from the medical device system 800 or can be a flexible device that does not have controllable articulation. Steerable instruments or tools suitable for use in some examples are further described in detail in U.S. Patent No. 7,916,681 (filed on October 4, 2005 and entitled “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent No. 9,259,274 (filed on September 30, 2008 and entitled “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated herein by reference in their entirety.

[0127] The flexible body 816 of the extension device 802 may also or alternatively house a cable, linkage, or other steering control (not shown) extending between the drive unit 804 and the distal end 818 to controllably bend the distal end 818, e.g., Figure 8A The distal end 818 is depicted as a dotted line 819. In some examples, at least four cables are used to provide independent up and down steering to control the pitch of the distal end 818 and left and right steering to control the yaw of the distal end 281. In these examples, the flexible elongate device 802 can be a steerable catheter. Examples of steerable catheters suitable for some examples are described in detail in PCT publication WO 2019 / 018436 (published on January 24, 2019 and entitled "Flexible Elongate Device Systems and Methods"), which is incorporated herein by reference in its entirety.

[0128] In examples where the extension device 802 and / or the medical tool 826 are actuated by a remotely operated assembly (e.g., manipulator assembly 702), the drive unit 804 can include a drive element (e.g., an actuator) that is removably coupled to the remotely operated assembly and receives a powered drive input from the drive element. In some examples, the extension device 802 and / or the medical tool 826 can include a gripping feature, a manual actuator, or other features for manually controlling the movement of the extension device 802 and / or the medical tool 826. The extension device 802 can be steerable, or alternatively, the extension device 802 can be non-steerable and lack an integrated mechanism for operator-controlled bending of the distal end 818. In some examples, one or more channels 821 (also referred to as lumens) can be defined by the inner wall of the flexible body 816 of the extension device 802, through which the medical tool 826 can be deployed and used at the target anatomical location.

[0129] In some examples, the medical device system 800 (e.g., the extension device 802 or the medical tool 826) can include a flexible bronchial instrument, such as a bronchoscope or a bronchial catheter, for inspection, diagnosis, biopsy, and / or treatment of the lungs. The medical device system 800 can also be adapted to navigate and treat other tissues in any of a variety of anatomical systems, including the colon, intestines, kidneys and calyces, brain, heart, circulatory system including vasculature, etc., via natural or surgically created connecting pathways.

[0130] Information from the tracking system 830 can be sent to the navigation system 832, where it can be combined with information from the visualization system 831 and / or pre-operatively acquired models to provide real-time position information to the physician, clinician, surgeon, or other operator. In some examples, the real-time position information can be displayed on the display system 710 for use in controlling the medical device system 800. In some examples, the navigation system 832 can use the position information as feedback for positioning the medical device system 800. Various systems for registering and displaying surgical instruments and surgical images using fiber optic sensors, which are applicable in some examples, are provided in U.S. Patent No. 8,300,131 (filed May 13, 2011 and entitled “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated herein by reference in its entirety.

[0131] Figure 9A and Figure 9Bis a simplified diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some examples. Figure 9A and Figure 9B As shown, surgical environment 900 may include patient P positioned on patient table T. Patient P may be stationary within surgical environment 900 because the patient's overall movement is limited by sedation, restraints, and / or other means. Periodic anatomical motion of patient P (including respiratory and cardiac motion) may continue. Within surgical environment 900, medical instrument 904 is used to perform a medical procedure, which may include, for example, surgery, biopsy, ablation, illumination, irrigation, suction, or electroporation. Medical instrument 904 may also be used to perform other types of procedures, such as a registration process that associates position, orientation, and / or pose data captured by sensor system 708 with a desired (e.g., anatomical or system) reference frame. Medical instrument 904 may be, for example, medical instrument 704. In some examples, medical instrument 904 may include an elongated device 910 (e.g., a catheter) coupled to an instrument body 912. Elongated device 910 includes one or more channels sized and shaped to accommodate a medical tool.

[0132] The extension device 910 may also include one or more sensors (e.g., components of the sensor system 708). In some examples, a shape sensor 914 may be affixed to the instrument body 912 at a proximal point 916. The proximal point 916 of the shape sensor 914 may move with the instrument body 912, and the position of the proximal point 916 relative to a desired reference frame may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 914 may measure the shape from the proximal point 916 to another point (e.g., the distal end 918 of the extension device 910). The shape sensor 914 may be aligned with the extension device 910 (e.g., disposed within an internal channel or mounted externally). In some examples, the shape sensor 914 may use optical fibers to generate shape information of the extension device 910.

[0133] In some examples, position sensors (e.g., EM sensors) can be incorporated into medical device 904. A series of position sensors can be positioned along flexible elongated device 910 and used for shape sensing. Position sensors can be used in place of or in addition to shape sensor 914, for example, to improve the accuracy of shape sensing or to verify shape information.

[0134] The extension device 910 can house cables, linkages, or other steering controls that extend between the instrument body 912 and the distal end 918 to controllably bend the distal end 918. In some examples, at least four cables are used to provide independent up and down steering to control the pitch of the distal end 918 and left and right steering to control the yaw of the distal end 918. The instrument body 912 can include a drive input that is removably coupled to a drive element (e.g., an actuator) of a manipulator assembly and receives power from the drive element.

[0135] The instrument body 912 can be coupled to an instrument bracket 906. The instrument bracket 906 can be mounted to an insertion table 908 that is fixed within the surgical environment 900. Alternatively, the insertion table 908 can be movable but have a known position within the surgical environment 900 (e.g., via a tracking sensor or other tracking device). The instrument bracket 906 can be a component of a manipulator assembly (e.g., manipulator assembly 702) that is coupled to the medical device 904 to control insertion motion (e.g., motion along the insertion axis A) and / or motion of the distal end 918 of the extension device 910 in multiple directions, such as yaw, pitch, and / or roll. The instrument bracket 906 or insertion table 908 can include an actuator, such as a servo motor, that controls the motion of the instrument bracket 906 along the insertion table 908.

[0136] The sensor device 920 can be a component of the sensor system 708 that can provide information about the position of the instrument body 912 as the instrument body 912 moves relative to the insertion table 908 along the insertion axis A. The sensor device 920 can include one or more rotary transformers, encoders, potentiometers, and / or other sensors that measure the rotation and / or orientation of the actuator that controls the movement of the instrument holder 906, thereby indicating the movement of the instrument body 912. In some examples, the insertion table 908 has a plurality of sensors such as Figure 9A and Figure 9B In some examples, the insertion station 908 can have a curved track or a combination of curved track sections and linear track sections.

[0137] Figure 9A The instrument body 912 and instrument bracket 906 are shown in a retracted position along the insertion table 908. In the retracted position, the proximal point 916 is located at position L0 on the insertion axis A. The position of the proximal point 916 can be set to a zero value and / or other reference value to provide a base reference (e.g., corresponding to the origin of the desired reference system) to describe the position of the instrument bracket 906 along the insertion table 908. In the retracted position, the distal end 918 of the extension device 910 can be positioned directly within the entry orifice of the patient P. Also in the retracted position, the data captured by the sensor device 920 can be set to a zero value and / or other reference value (e.g., I=0). Figure 9B908 , the instrument body 912 and the instrument holder 906 have been advanced along the linear rails of the insertion table 908, and the distal end 918 of the extension device 910 has been advanced into the patient P. In this advanced position, the proximal point 916 is located at a position L1 on the insertion axis A. In some examples, the rotation and / or orientation of the actuator measured by the sensor device 920 that indicates the movement of the instrument holder 906 along the insertion table 908 and / or one or more position sensors associated with the instrument holder 906 and / or the insertion table 908 can be used to determine the position L1 of the proximal point 916 relative to the position L0. In some examples, the position L1 can also be used as an indicator of the distance or insertion depth of the distal end 918 of the extension device 910 inserted into the passage of the anatomical structure of the patient P.

[0138] One or more components (e.g., control system 712) of the examples discussed in this disclosure can be implemented in software to be executed on one or more processors of a computer system. The software may include code that, when executed by one or more processors, configures one or more processors to perform various functions as discussed herein. The code can be stored in a non-transient computer-readable storage medium (e.g., memory, magnetic storage device, optical storage device, solid-state storage device, etc.). The computer-readable storage medium can be a part of a computer-readable storage device, such as an electronic circuit, a semiconductor device, a semiconductor memory device, a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM), a floppy disk, a CD-ROM, an optical disc, a hard disk, or other storage device. The code can be downloaded via a computer network such as the Internet, an intranet, etc. to be stored on a computer-readable storage medium. The code can be executed by any of a variety of centralized or distributed data processing architectures. The programming instructions of the code can be implemented as multiple separate programs or subroutines, or they can be integrated into multiple other aspects of the system described herein. The components of the computing system discussed herein can be connected using wired and / or wireless connections. In some examples, the wireless connection may use wireless communication protocols such as Bluetooth, Near Field Communication (NFC), Infrared Data Association (IrDA), Home Radio Frequency (HomeRF), IEEE 502.11, Digital Enhanced Cordless Telecommunications (DECT), and Wireless Medical Telemetry Service (WMTS).

[0139] Various general-purpose computer systems can be used to perform one or more of the processes, methods, or functions described herein. Additionally or alternatively, various special-purpose computer systems can be used to perform one or more of the processes, methods, or functions described herein. In addition, various programming languages ​​can be used to implement one or more of the processes, methods, or functions described herein.

[0140] While certain examples and embodiments have been described above and shown in the accompanying drawings, it should be understood that these examples and embodiments are merely illustrative and are not limited to the specific constructions and arrangements shown and described, since those skilled in the art will recognize various other alternatives, modifications, and equivalents.

Claims

1. A method for generating a three-dimensional (3D) navigation interface for a robot-assisted medical procedure, the method comprising: receiving, by one or more processors, a 3D model representing a volume of a patient; receiving, by the one or more processors, two-dimensional (2D) data representing one or more 2D images corresponding to at least a portion of the volume of the patient; generating, by the one or more processors, operational data for co-registering the 3D model with the 2D data; generating, by the one or more processors, a 3D navigation interface, comprising: generating a display of at least a portion of the 3D model based on the co-registered operational data; as well as The one or more processors cause a display device to present the 3D navigation interface to a user.

2. The method according to claim 1, wherein Generating the 3D navigation interface includes: A display of the one or more 2D images is generated based at least on the co-registered operational data.

3. The method according to claim 2, wherein: Generating the 3D navigation interface further includes: The one or more 2D images are oriented such that an orientation of the one or more 2D images is based on at least the 3D model.

4. The method according to claim 1, further comprising: receiving 3D sensor data from one or more sensors configured to generate data associated with the volume of the patient; Wherein, generating the co-registered operational data comprises generating co-registered 3D data based on the 3D sensor data.

5. The method according to claim 4, wherein The 3D sensor data includes at least one of: computed tomography (CT) data, cone beam computed tomography (CBCT) data, catheter data, endoscopic video data, or magnetic resonance imaging (MRI) data.

6. The method according to claim 1, further comprising: receiving 2D sensor data from one or more sensors configured to generate data associated with the volume of the patient; Wherein, generating the co-registered operational data comprises generating co-registered 2D data based on the 2D sensor data.

7. The method according to claim 6, wherein: The 2D sensor data includes at least one of: endoscopic video data, C-arm data, or radial endobronchial ultrasound (EBUS) data.

8. The method according to claim 1, further comprising: receiving navigation information including historical navigation information representing past navigation of a sensor in the volume of the patient; and Wherein, generating the 3D navigation interface further includes generating the 3D navigation interface based on the historical navigation information.

9. The method according to claim 1, further comprising: receiving navigation information including navigation guidance information indicating a recommended path for a sensor in the volume of the patient; and Wherein, generating the 3D navigation interface further includes generating the 3D navigation interface based on the navigation guidance information.

10. The method according to claim 1, further comprising: receiving navigation information comprising one or more visual or auditory elements representing landmarks in the volume of the patient; and Herein, generating the 3D navigation interface further comprises generating the 3D navigation interface based on the one or more visual or auditory elements.

11. The method according to claim 1 , further comprising: receiving navigation information including one or more sampled tissue locations in the volume of the patient; and Wherein, generating the 3D navigation interface further includes generating the 3D navigation interface based on the navigation information.

12. The method according to any one of claims 1 to 11, wherein The one or more 2D images include one or more 2D x-ray images.

13. The method according to claim 12, wherein: The one or more 2D x-ray images include fluoroscopic images captured by a C-arm.

14. The method according to claim 13, wherein Generating the co-registered operational data includes: Track the pose of the C-arm.

15. The method according to claim 14, wherein Tracking the posture of the C-arm includes: A head-mounted display (HMD) is used to track the pose of the C-arm.

16. The method according to claim 14, wherein Tracking the posture of the C-arm includes: The position of the C-arm is tracked using sensors on the C-arm.

17. The method according to any one of claims 1 to 11, wherein Enabling the display device to present the 3D navigation interface includes: A head-mounted display (HMD) is caused to present the 3D navigation interface as one of: (i) a mixed reality (MR) navigation interface, (ii) an augmented reality (AR) navigation interface, or (iii) a virtual reality (VR) navigation interface.

18. The method according to any one of claims 1 to 11, wherein The 3D model includes a 3D lung model, and wherein causing the display device to present the 3D navigation interface includes causing the display device to present at least the portion of the 3D lung model (i) above the patient, (ii) adjacent to the patient, or (iii) superimposed on the patient.

19. The method according to any one of claims 1 to 11, wherein The 3D model is generated prior to the medical procedure via at least one of: (i) a computed tomography (CT) scanning device; (ii) a magnetic resonance imaging (MRI) device; or (iii) a positron emission tomography (PET) scanning device.

20. The method according to any one of claims 1 to 11, wherein The 3D model is generated during the treatment procedure via at least one of: (i) a cone beam computed tomography (CBCT) scanning device or (ii) a tomosynthesis device.

21. The method according to any one of claims 1 to 11, wherein The 3D navigation interface includes at least one of a live 2D video feed or a live 3D video feed.

22. The method according to claim 21, wherein Generating the 3D navigation interface further includes: A camera view of at least one of the live 2D video feed or the live 3D video feed is aligned with the 3D model.

23. The method according to any one of claims 1 to 11, further comprising: As the user navigates the patient volume, at least a portion of the 3D navigation interface is modified based on control input received from the user.

24. The method according to claim 23, further comprising: The control input is received from a console operated by the user, wherein the control input comprises one or more inputs made by the user via at least one of: (i) a controller, (ii) a trackball, (iii) a keyboard, (iv) a mouse, or (v) a touch screen device.

25. The method of claim 23, further comprising: The control input is received based on user movement detected by at least one of: (i) a touch sensor, (ii) a motion sensor, (iii) an accelerometer, (iv) a gyroscope, or (v) a position sensor.

26. The method according to claim 25, wherein The user movement is movement representing the user dragging a virtual representation of a sensor within the confines of the 3D navigation interface.

27. The method according to claim 25, wherein The user movement includes an indication of a virtual target in the volume of the patient.

28. The method according to any one of claims 1 to 11, further comprising: The visibility of at least a portion of the 3D navigation interface is modified based on the user's context or current task.

29. The method of claim 28, wherein modifying the visibility comprises: determining at least one element is potentially distracting to the user based on the user's context or current task; as well as At least one of the following operations is performed to make the at least one element invisible to the user: reducing visibility of the at least one element or changing an augmented reality type of the display device.

30. The method according to any one of claims 1 to 11, wherein The user is a first user, the display device is a first display device, and the 3D navigation interface is a first 3D navigation interface, and wherein the method further comprises: generating a second 3D navigation interface, wherein the second 3D navigation interface includes a reduced set of information compared to the first 3D navigation interface; and The second display device is enabled to present the second 3D navigation interface to a second user.

31. A system for generating a three-dimensional (3D) navigation interface for a robotic-assisted medical procedure, the system comprising: one or more processors; Communication unit; display device; as well as A non-transitory computer-readable medium coupled to the one or more processors and the communication unit and having instructions stored thereon that, when executed by the one or more processors, cause the system to perform the method according to any one of claims 1 to 30.

32. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 30.

33. A method for generating an interactive view for a robot-assisted medical procedure, the method comprising: receiving, by one or more processors, a three-dimensional (3D) model representing a volume of a patient; generating, by the one or more processors, a two-dimensional (2D) view of the volume of the patient, the 2D view representing the 2D imaging modality when positioned at a particular device projection angle; determining a display orientation of the 2D view relative to the user; registering the 2D view to the 3D model such that both the 2D view and the 3D model share the display orientation; causing, by the one or more processors, a display device to simultaneously present the 2D view and the 3D model to a user according to the shared display orientation; receiving, by the one or more processors, a control input from the user; as well as At least the 2D view of the volume of the patient is updated, by the one or more processors, based on the control input.

34. The method of claim 33, further comprising: The 2D view and the 3D model are oriented relative to the user such that the shared display orientation represents at least one of: (i) a first-person view relative to the user, (ii) a third-person view relative to the user, (iii) a top view of the patient, or (iv) a task view representing the volume of the patient based on a current task being performed by the user.

35. The method of claim 34, further comprising: The 3D model is scaled relative to at least one of: (i) a velocity of movement of an elongated flexible device within the volume of the patient, (ii) a navigation context of the volume of the patient, (iii) a user-indicated accuracy preference, or (iv) a current user task.

36. The method according to claim 35, wherein Scaling the 3D model includes modifying an indication corresponding to a navigation path based on the movement speed.

37. The method of claim 33, further comprising: The 2D view and the 3D model are updated while maintaining the shared display orientation.

38. The method of claim 33, further comprising: The 2D histological sample image is embedded in the 3D model based on the location of the volume of the patient where the 2D histological sample image was captured.

39. The method of claim 33, further comprising: A procedure report is generated that includes navigation information representing at least one of the 2D view or the 3D model of the volume of the patient.

40. The method of claim 39, wherein The navigation information includes at least one of the following: (i) navigation guidance information representing a recommended path for a sensor in the volume of the patient, (ii) historical navigation information representing past navigations of the sensor in the volume of the patient, (iii) a shared display orientation of the 2D view and the 3D model, (iv) a slice of the 3D model, or (v) Histological data representing at least one of the 2D view or the 3D model.

41. The method of claim 33, wherein: The control input comprises user manipulation of a virtual representation of the elongated flexible device, and wherein the method further comprises: In response to receiving the control input, the elongated flexible device is caused to move according to the user manipulation of the virtual representation.

42. The method of claim 33, wherein: The control input comprises user interaction with a physical location or a virtual representation of the physical location, the method further comprising: In response to receiving the control input, a virtual path to the location is automatically generated.

43. The method of claim 33, further comprising: receiving a computed tomography (CT) image during the medical procedure; determining that the CT image deviates from the volume; as well as A request is sent to replace the cone beam computed tomography (CBCT) image with the CT image.

44. The method according to any one of claims 33 to 43, wherein The control input includes a control input for rotating the C-arm.

45. The method of claim 44, wherein: Updating at least the 2D view comprises: At least the 2D view is updated to correspond to the new angle of the C-arm after the rotation.

46. ​​The method of claim 44, wherein The control input for rotating the C-arm is a control input for rotating a virtual C-arm.

47. The method of claim 44, wherein The control input for rotating the C-arm is a control input for rotating a physical C-arm.

48. The method according to any one of claims 33 to 43, wherein The 2D view includes at least one of: (i) a synthetic fluoroscopic image, (ii) a 2D fluoroscopic image from a C-arm, or (iii) an endobronchial ultrasound (EBUS) image.

49. The method according to any one of claims 33 to 43, wherein Causing the display device to simultaneously present the 2D view and the 3D model includes causing a head-mounted display (HMD) to present the 2D view and the 3D model in extended reality (XR).

50. The method according to any one of claims 33 to 43, wherein The 3D model includes a 3D lung model, and wherein the 3D lung model is presented over a view of the patient.

51. The method of claim 50, wherein: The view of the patient is a view of the patient as an entity in an augmented reality (AR) view.

52. The method of claim 50, wherein: The view of the patient is a representation of the patient in a virtual reality (VR) view.

53. The method according to any one of claims 33 to 43, wherein The 3D model is generated prior to the medical procedure via at least one of: (i) a computed tomography (CT) scanning device; (ii) a magnetic resonance imaging (MRI) device; or (iii) a positron emission tomography (PET) scanning device.

54. The method according to any one of claims 33 to 43, wherein The 3D model is generated during the treatment procedure via at least one of: (i) a cone beam computed tomography (CBCT) scanning device or (ii) a tomosynthesis device.

55. The method according to any one of claims 33 to 43, wherein Enabling the display device to simultaneously present the 2D view and the 3D model comprises: The display device is caused to present an indication of how to rotate in at least one of the 2D view or the 3D model for reorientation.

56. The method according to any one of claims 33 to 43, wherein The 3D model includes a virtual device rotated to match the positioning of a corresponding physical device.

57. The method according to any one of claims 33 to 43, wherein The 2D imaging modality includes at least one of the following: (i) C-arm imaging, (ii) computed tomography (CT) imaging, or (iii) Endobronchial ultrasound (EBUS) imaging.

58. The method according to any one of claims 33 to 43, wherein The user is a first user, the display device is a first display device, the shared orientation is a first user orientation, and the method further comprises: generating a second user orientation for a second user, wherein the second user orientation is different from the first user orientation; and A second display is caused to present the 2D view and the 3D model to the second user based on the second user orientation.

59. A system for generating an interactive view for a robot-assisted medical procedure, the system comprising: one or more processors; Communication unit; display device; as well as A non-transitory computer-readable medium coupled to the one or more processors and the communication unit and having instructions stored thereon that, when executed by the one or more processors, cause the system to perform the method according to any one of claims 33 to 58.

60. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 33 to 58.

Images