Updating and displaying model of surgical environment

Abstract

A first method includes capturing, via a camera having a first pose, a first image of a surgical environment after a change has been made to the surgical environment. The method also includes capturing, via. a. camera having a second pose that is different from the first pose, a second image of the surgical environment after the change has been made to the surgical environment. Hie method also includes using a computational algorithm to process the first image, the second image, and a model representing the surgical environment, thereby updating the model to represent the change made to the surgical environment. A second method includes training the computational algorithm to perform aspects of the first method.

Description

Updating and Displaying Model of Surgical Environment CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 693,002, filed on September 10, 2024, the entire contents of which are incorporated by reference herein. BACKGROUND

[0002] Endoscopic sinus surgery (ESS) is very common, with over 300,000 surgeries performed yearly in the United States. Based on a patient’s pre-operative anatomic imaging, surgeons can use Image Guidance Systems (IGS) that facilitate instrument tracking within a sinonasal cavity during a surgery. However, current IGS does not accurately update in real- time, even though a patient’s anatomy changes incrementally with each surgical modification performed during a broader surgical procedure. SUMMARY

[0003] A first example is a method comprising: accessing a first model representing a first surgical environment; accessing first images of the first surgical environment captured by a first imaging system having a different pose for each of the first images; and using the first model and the first images to train a computational algorithm to perform functions comprising: accessing a second model representing a second surgical environment; accessing second images of the second surgical environment captured by a second imaging system after a change has been made to the second surgical environment; and using the second images to update the second model to represent the change made to the second surgical environment.

[0004] A second example is a non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of the first example.

[0005] A third example is a computing device comprising: one or more processors; and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of the first example.

[0006] A fourth example is a method comprising: capturing, via a camera having a first pose, a first image of a surgical environment after a change has been made to the surgical environment; capturing, via a camera having a second pose that is different from the first pose, a second image of the surgical environment after the change has been made to the surgical environment; and using a computational algorithm to process the first image, thesecond image, and a model representing the surgical environment, thereby updating the model to represent the change made to the surgical environment.

[0007] A fifth example is a non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of the fourth example.

[0008] A sixth example is an imaging system comprising: one or more cameras; one or more processors; and a computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of the fourth example.

[0009] When the term “substantially” or “about” is used herein, it is meant that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including, for example, tolerances, measurement error, measurement accuracy limitations, and other factors known to those of skill in the art may occur in amounts that do not preclude the effect the characteristic was intended to provide. In some examples disclosed herein, “substantially” or “about” means within + / - 0-5% of the recited value.

[0010] These, as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, it should be understood that this summary and other descriptions and figures provided herein are intended to illustrate by way of example only and, as such, that numerous variations are possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an imaging system and a computing device, according to an example.

[0012] Figure 2A shows a model of a surgical environment overlaid upon an image of the surgical environment, according to an example.

[0013] Figure 2B is an image of a surgical environment, according to an example.

[0014] Figure 3A shows a three-dimensional rendering of a model of a surgical environment, according to an example.

[0015] Figure 3B shows a three-dimensional rendering of a model of a surgical environment, according to an example.

[0016] Figure 4 is a schematic diagram of functionality related to training and using a computational algorithm to update a model of a surgical environment, according to an example.

[0017] Figure 5A is an image of a surgical environment, according to an example.

[0018] Figure 5B shows a model of a surgical environment overlaid upon an image of the surgical environment, according to an example.

[0019] Figure 6 shows a model of a changed surgical environment overlaid upon an image of the changed surgical environment, according to an example.

[0020] Figure 7 is a block diagram of a method, according to an example.

[0021] Figure 8 is a block diagram of a method, according to an example. DETAILED DESCRIPTION

[0022] There is an unmet need for a surgical guidance system that accurately updates in real-time, without increasing intraoperative time and requiring expensive ancillary hardware. This disclosure includes methods for training and using a computational algorithm to periodically update and display a model of a surgical environment as changes (e.g., surgical modifications) are made to the surgical environment. Using these systems and methods, surgeons can quickly apprise the progress of the surgery in real time.

[0023] More specifically, this disclosure includes a method for training a computational algorithm, such as a neural radiance field (NeRF) algorithm. The method includes accessing a first model representing a first surgical environment. In some examples, the first model could include point cloud data collected via magnetic resonance imaging (MRI) or computed tomography (CT). The first surgical environment could be a sinonasal cavity. The method also includes accessing first images of the first surgical environment captured by a first imaging system having a different pose for each of the first images, and using the first model and the first images to train a computational algorithm to perform a method of using the computational algorithm.

[0024] The method of using the computational algorithm includes capturing, via a camera having a first pose, a first image of a surgical environment after a change has been made to the surgical environment. The change is typically a surgical modification. The method also includes capturing, via a camera having a second pose that is different from the first pose, a second image of the surgical environment after the change has been made to the surgical environment. The method also includes using the computational algorithm to process the first image, the second image, and a model representing the surgical environment, thereby updating the model to represent the change made to the surgical environment.

[0025] Figure 1 is a block diagram of an imaging system 10 and a computing device 100A. The imaging system 10 includes a computing device 100B and an endoscope 203. The endoscope 203 includes a light source 204, a camera 202A, and a camera 202B.

[0026] The endoscope 203 takes the form of a cable or a flexible scope that can be non- invasively inserted into surgical environments within a human body. The light source 204, the camera 202A, and the camera 202B are typically positioned at the distal end of the endoscope 203 to face the surgical environment. The distal end of the endoscope 203 is typically inserted into a surgical environment while the proximal end of the endoscope 203 remains outside of the surgical environment.

[0027] The cameras 202 can each include a digital image sensor and various optical components. In some examples, the imaging system 10 includes a monocular camera (e.g., only the camera 202A).

[0028] The light source 204 can include one or more light emitting diodes, among other examples.

[0029] The computing device 100A and the computing device 100B each includes one or more processors 102, a non-transitory computer readable medium 104, a communication interface 106, and a user interface 108. Components of the computing device 100A and the computing device 100B are linked together by a system bus, network, or other connection mechanism 112.

[0030] The one or more processors 102 may be any type of processor(s), such as a microprocessor, a field programmable gate array (FPGA), a digital signal processor, a multicore processor, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), one or more logic gates, etc., which can be coupled to the non-transitory computer readable medium 104 in some examples.

[0031] The non-transitory computer readable medium 104 may be any type of memory, such as volatile memory like random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), or non-volatile memory like read- only memory (ROM), flash memory, magnetic or optical disks, or compact-disc read-only memory (CD-ROM), among other devices used to store data or programs on a temporary or permanent basis.

[0032] Additionally, the non-transitory computer readable medium 104 stores instructions 111. The instructions 111 are executable by the one or more processors 102 to cause the computing device 100 to perform any of the functions or methods described herein. The non- transitory computer readable medium 104 can also store the computational algorithm 113 which can take the form of a machine learning model, an artificial neural network (ANN), a convolutional neural network (CNN), a diffusion model, a message-passing neural network(MPNN), a regression model, a multilayer perceptron (MLP) and / or a neural radiance field (NeRF) algorithm.

[0033] The communication interface 106 may include hardware to enable communication within the computing device 100 and / or between the computing device 100 and one or more other devices. The hardware can include any type of input and / or output interfaces, a universal serial bus (USB), PCI Express, transmitters, receivers, and antennas, for example. The communication interface 106 may be configured to facilitate communication with one or more other devices, in accordance with one or more wired or wireless communication protocols. For example, the communication interface 106 may be configured to facilitate wireless data communication for the computing device 100 according to one or more wireless communication standards, such as one or more Institute of Electrical and Electronics Engineers (IEEE) 801.11 standards, ZigBee standards, Bluetooth standards, etc. As another example, the communication interface 106 can be configured to facilitate wired data communication with one or more other devices. The communication interface 106 may also include analog-to-digital converters (ADCs) or digital-to-analog converters (DACs) that the computing device 100 can use to control various components of the computing device 100 or external devices.

[0034] The user interface 108 may include any type of display component configured to display data. As one example, the user interface 108 can include a touchscreen display. As another example, the user interface 108 can include a flat-panel display, such as a liquid- crystal display (LCD) or a light-emitting diode (LED) display. The user interface 108 can include one or more pieces of hardware used to provide data and control signals to the computing device 100. For instance, the user interface 108 can include a mouse or a pointing device, a keyboard or a keypad, a microphone, a touchpad, or a touchscreen, among other possible types of user input devices. Generally, the user interface 108 can enable an operator to interact with a graphical user interface (GUI) provided by the computing device 100 (e.g., displayed by the user interface 108).

[0035] Figure 2A shows a model 302A of a surgical environment 304A overlaid upon an image of the surgical environment 304A. Figure 2B is an image of the surgical environment 304A without the model 302A overlaid. An area of interest of the surgical environment 304A is encircled in Figure 2B. In various examples, the surgical environment 304A can be part of a living human or a cadaver.

[0036] The model 302A can take the form of a three-dimensional point cloud (e.g., a set of three-dimensional coordinates) and can include data collected using a computed tomography(CT) scan of the surgical environment 304A or data collected using a magnetic resonance imaging (MRI) scan of the surgical environment 304A, for example. The surgical environment 304A generally includes soft tissues and / or bones and can be a human sinonasal cavity, but other examples are possible. The surgical environment 304A is generally used for training the computational algorithm 113.

[0037] Figure 3A shows a three-dimensional rendering of the model 302A of the surgical environment 304A.

[0038] Figure 3B shows a three-dimensional rendering of a model 302B of a surgical environment 304B. The model 302B can take the form of a three-dimensional point cloud and can include data collected using a computed tomography (CT) scan of the surgical environment 304B or data collected using a magnetic resonance imaging (MRI) scan of the surgical environment 304B, for example. The surgical environment 304B generally includes soft tissues and / or bones and can be a human sinonasal cavity, but other examples are possible. In various examples, the surgical environment 304B can be part of a living human or a cadaver.

[0039] The surgical environment 304B is also shown in Figure 5A, Figure 5B, and Figure 6. In some examples, the surgical environment 304B is the same surgical environment as the surgical environment 304A. In other examples, the surgical environment 304B is not the same surgical environment as the surgical environment 304A. The surgical environment 304B is generally the surgical environment being subjected to a medical procedure. That is, the model 302B is generally updated and / or displayed as surgical modifications or other changes are made to the surgical environment 304B.

[0040] Figure 4 is a schematic diagram of functionality related to training and using the computational algorithm 113 to update the model 302B of the surgical environment 304B. To train the computational algorithm 113, the computing device 100A accesses the model 302A representing the surgical environment 304A and accesses images 306A of the surgical environment 304A. The model 302A and the images 306A can be generated and then stored on the non-transitory computer readable medium 104 prior to the computing device 100A accessing the model 302A and the images 306A, for example. The images 306A are captured by an imaging system 10A having a different pose for each of the images 306A. That is, each of the images 306A are generally captured while the imaging system 10A (e.g., a tip of the endoscope 203) has a different position and / or a different orientation within the surgical environment 304A. Accordingly, the pose for each of the images 306A can be defined by five coordinates: x, y, and z defining a position along three orthogonal axes and and defining aviewing direction by an azimuthal angle and a polar angle. The poses of the images 306A can accompany the images 306A as metadata. In some examples, a field of view of each of the images 306A overlaps with a field of view of at least one of the other images 306A by at least 90%.

[0041] Next, the computing device 100A uses the model 302A and the images 306A to train the computational algorithm 113 to perform runtime functions. The runtime functions include accessing the model 302B representing the surgical environment 304B, accessing images 306B of the surgical environment 304B captured by an imaging system 10B after a change has been made to the surgical environment 304B, and using the images 306B to update the model 302B to represent the change made to the surgical environment 304B.

[0042] To train the computational algorithm 113, the computing device 100A provides the computational algorithm 113 with the model 302A, the images 306A, and the metadata indicating the pose of the imaging system 10A when the images 306A were captured. The computational algorithm 113 can be formed of multiple layers of multiple nodes. The computing device 100 performs a supervised or unsupervised learning algorithm such that the weights between various nodes of the computational algorithm 113 are adjusted such that the computational algorithm 113 better maps the images 306A (i.e., two-dimensional images) associated with the poses indicated by the metadata to the model 302A. This trains the computational algorithm 113 to update the model 302B based on the newly encountered images 306B as described in more detail below.

[0043] In some examples, the computing device 100A generates virtual images 307 of the surgical environment 304A that form stereo pairs respectively with the images 306A. The computing device 100 additionally uses the stereo pairs made up of the images 306A and the virtual images 307 to train the computational algorithm 113 to update the model 302B as described above. In some examples, the computing device 100A determines a direction of movement of the imaging system 10A based on two or more of the 306A images and generates the virtual images 307 based on the direction of movement of the imaging system 10A.

[0044] Figure 5A is an image of the surgical environment 304B before a change is made to the surgical environment 304B. Figure 5B shows the model 302B of the surgical environment 304B overlaid upon an image of the surgical environment 304B.

[0045] Figure 6 shows the model 302B of the surgical environment 304B updated to reflect a change that has been made to the surgical environment 304B (e.g. a resection). The model 302B is overlaid upon the image of the surgical environment 304B in Figure 6.

[0046] Figure 5A, Figure 5B, and Figure 6 are related to actions of an imaging system 10B. In some examples, the imaging system 10B and the imaging system 10A are the same imaging system. In other examples, the imaging system 10B and the imaging system 10A are not the same imaging system.

[0047] Referring to Figures 4-6, the imaging system 10B captures two or more images 306B of the surgical environment 304B after a change has been made to the surgical environment 304B. Figure 5A and Figure 6 show that the change made to the surgical environment 304B is a resection in this example. The two or more images 306B were captured by the imaging system 10B with different poses within the surgical environment 304B. In some examples, a field of view of each of the images 306B overlaps with a field of view of at least one of the other images 306B by at least 90%.

[0048] The computing device 100B uses the computational algorithm 113 to process the two or more images 306B and the model 302B shown in Figure 5B, thereby updating the model 302B to represent the change made to the surgical environment 304B, as shown in Figure 6. Generally, updating the model 302B involves rewriting data to the non-transitory computer readable medium 104 of the computing device 100B. In various examples, the model 302B can be updated to include a structure added by a medical procedure or to remove a structure removed by a medical procedure. That is, the change to the surgical environment 304B can be a medical procedure such as an excision, an incision, a resection, a displacement, or an injection.

[0049] In some examples, the computing device 100B determines poses of the imaging system 10B corresponding to the images 306B using a simultaneous localization and mapping algorithm or an electromagnetic tracking sensor. In this context, the computing device 100B updates the model 302B based on the images 306B and the poses of the imaging system 10B corresponding to the images 306B.

[0050] In some examples, updating the model 302B can include an intermediate step. That is, the computing device 100B generates an intermediate model by processing the images 306B using the computational algorithm 113. The intermediate model represents the surgical environment 304B after the change has been made to the surgical environment 304B. The computing device 100B uses the intermediate model to update the model 302B shown in Figure 5B to represent the change made to the surgical environment 304B shown in Figure 6.

[0051] In some examples, it is useful for the computing device 100B to display the model 302B in real-time as the model 302B is updated to reflect surgical modifications.Furthermore, it can be useful to toggle the display between “before” and “after” versions of the model 302B.

[0052] Thus, the computing device 100B can receive, via the user interface 108, a command to display the model 302B as the model 302B existed prior to the updating, such as in Figure 5B. Additionally, the computing device 108B can display, in response to receiving the command, the model 302B as the model 302B existed prior to the updating, such as in Figure 5B.

[0053] Similarly, the computing device 100B can receive, via the user interface 108, a command to display the model 302B as altered by the updating and display, in response to receiving the command, the model 302B as altered by the updating, as shown in Figure 6.

[0054] In some examples, the computing device 100B uses one or more of the images 306B and the model 302B to determine that a distance between a surgical instrument and an object within the surgical environment 304B is less than a threshold distance. Responsively, the computing device 100B generates, via the user interface 108, an indication that the distance between the surgical instrument and the object is less than the threshold distance.

[0055] Figure 7 and Figure 8 are block diagrams of a method 400 and a method 500. As shown in Figure 7 and Figure 8, the method 400 and the method 500 include one or more operations, functions, or actions as illustrated by blocks 402, 404, 406, 502, 504, and 506. Although the blocks are illustrated in a sequential order, these blocks may also be performed in parallel, and / or in a different order than those described herein. Also, the various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based upon the desired implementation.

[0056] At block 402, the method 400 includes accessing the model 302A representing the surgical environment 304A. Functionality related to block 402 is described above with reference to Figure 2A, Figure 2B, Figure 3, and Figure 4.

[0057] At block 404, the method 400 includes accessing the images 306A of the surgical environment 304A captured by the imaging system 10A having a different pose for each of the images 306A. Functionality related to block 404 is described above with reference to Figure 4.

[0058] At block 406, the method 400 includes using the model 302A and the images 306A to train the computational algorithm 113 to perform functions comprising: accessing the model 302B representing the surgical environment 304B, accessing images 306B of the surgical environment 304B captured by the imaging system 10B after a change has been made to the surgical environment 304B, and using the images 306B to update the model302B to represent the change made to the surgical environment 304B. Functionality related to block 406 is described above with reference to Figures 2-6.

[0059] At block 502, the method 500 includes the imaging system 10B capturing, via a camera having a first pose, a first image 306B of the surgical environment 304B after a change has been made to the surgical environment 304B. Functionality related to block 502 is described above with reference to Figures 4-6.

[0060] At block 504, the method 500 includes the imaging system 10B capturing, via a camera having a second pose that is different from the first pose, a second image 306B of the surgical environment 304B after the change has been made to the surgical environment 304B. Functionality related to block 504 is described above with reference to Figures 4-6.

[0061] At block 506, the method 500 includes the computing device 100B using the computational algorithm 113 to process the first image 306B, the second image 306B, and the model 302B representing the surgical environment 304B, thereby updating the model 302B to represent the change made to the surgical environment 304B. Functionality related to block 506 is described above with reference to Figures 4-6.

[0062] ENUMERATED EXAMPLE EMBODIMENTS (EEEs)

[0063] EEE 1 is a method comprising: accessing a first model representing a first surgical environment; accessing first images of the first surgical environment captured by a first imaging system having a different pose for each of the first images; and using the first model and the first images to train a computational algorithm to perform functions comprising: accessing a second model representing a second surgical environment; accessing second images of the second surgical environment captured by a second imaging system after a change has been made to the second surgical environment; and using the second images to update the second model to represent the change made to the second surgical environment.

[0064] EEE 2 is the method of EEE 1, wherein the first surgical environment and the second surgical environment are the same surgical environment.

[0065] EEE 3 is the method of EEE 1, wherein the first surgical environment and the second surgical environment are not the same surgical environment.

[0066] EEE 4 is the method of any one of EEEs 1-3, wherein the first imaging system and the second imaging system are the same imaging system.

[0067] EEE 5 is the method of any one of EEEs 1-3, wherein the first imaging system and the second imaging system are not the same imaging system.

[0068] EEE 6 is the method of any one of EEEs 1-5, wherein the first model includes data collected using a computed tomography (CT) scan of the first surgical environment.

[0069] EEE 7 is the method of any one of EEEs 1-6, wherein the first model includes data collected using a magnetic resonance imaging (MRI) scan of the first surgical environment.

[0070] EEE 8 is the method of any one of EEEs 1-7, wherein the first model comprises a point cloud.

[0071] EEE 9 is the method of any one of EEEs 1-8, wherein the first surgical environment comprises soft tissue or bone.

[0072] EEE 10 is the method of any one of EEEs 1-9, wherein the first surgical environment comprises a sinonasal cavity.

[0073] EEE 11 is the method of any one of EEEs 1-10, wherein the first imaging system comprises a monocular camera.

[0074] EEE 12 is the method of any one of EEEs 1-11, wherein the first imaging system includes an endoscope.

[0075] EEE 13 is the method of any one of EEEs 1-12, wherein the second model includes data collected using a computed tomography (CT) scan of the second surgical environment.

[0076] EEE 14 is the method of any one EEEs 1-13, wherein the second model includes data collected using a magnetic resonance imaging (MRI) scan of the second surgical environment.

[0077] EEE 15 is the method of any one of EEEs 1-14, wherein the second model comprises a point cloud.

[0078] EEE 16 is the method of any one of EEEs 1-15, wherein the second surgical environment comprises soft tissue or bone.

[0079] EEE 17 is the method of any one of EEEs 1-16, wherein the second surgical environment comprises a sinonasal cavity.

[0080] EEE 18 is the method of any one of EEEs 1-17, wherein the second imaging system comprises a monocular camera.

[0081] EEE 19 is the method of any one of EEEs 1-18, wherein the second imaging system includes an endoscope.

[0082] EEE 20 is the method of any one of EEEs 1-19, wherein the change to the second surgical environment is caused by a medical procedure performed upon the second surgical environment.

[0083] EEE 21 is the method of EEE 20, wherein the medical procedure comprises one or more of an excision, an incision, a resection, a displacement, or an injection.

[0084] EEE 22 is the method of any one of EEEs 1-21, the functions further comprising determining poses of the second imaging system corresponding to the second images using asimultaneous localization and mapping algorithm or an electromagnetic tracking sensor, wherein using the second images to update the second model comprises updating the second model based on the second images and the poses of the second imaging system corresponding to the second images.

[0085] EEE 23 is the method of any one of EEEs 1-22, wherein a field of view of each of the first images overlaps with a field of view of at least one of the other first images by at least 90%.

[0086] EEE 24 is the method of any one of EEEs 1-23, wherein a field of view of each of the second images overlaps with a field of view of at least one of the other second images by at least 90%.

[0087] EEE 25 is the method of any one of EEEs 1-24, further comprising: generating virtual images of the first surgical environment that form stereo pairs respectively with the first images; and additionally using the stereo pairs to train the computational algorithm to perform the functions.

[0088] EEE 26 is the method of EEE 25, further comprising: determining a direction of movement of the first imaging system based on two or more of the first images; and generating the virtual images based on the direction of movement.

[0089] EEE 27 is the method of any one of EEEs 1-26, wherein the computational algorithm comprises a neural radiance field (NeRF) algorithm.

[0090] EEE 28 is a non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of any one of EEEs 1-27.

[0091] EEE 29 is a computing device comprising: one or more processors; and a non- transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of any one of EEEs 1- 27.

[0092] EEE 30 is a method comprising: capturing, via a camera having a first pose, a first image of a surgical environment after a change has been made to the surgical environment; capturing, via a camera having a second pose that is different from the first pose, a second image of the surgical environment after the change has been made to the surgical environment; and using a computational algorithm to process the first image, the second image, and a model representing the surgical environment, thereby updating the model to represent the change made to the surgical environment.

[0093] EEE 31 is the method of EEE 30, wherein the model is a first model and using the computational algorithm comprises: generating a second model by processing the first image and the second image using the computational algorithm, wherein the second model represents the surgical environment after the change has been made to the surgical environment; and using the second model to update the first model to represent the change made to the surgical environment.

[0094] EEE 32 is the method of EEE 31, further comprising co-registering the first model and the second model, wherein updating the first model comprises updating the first model using the second model after co-registering the first model and the second model.

[0095] EEE 33 is the method of any one of EEEs 30-32, wherein the model includes data collected using a computed tomography (CT) scan or a magnetic resonance imaging (MRI) scan of the surgical environment.

[0096] EEE 34 is the method of any one of EEEs 30-33, wherein the computational algorithm is a neural radiance field (NeRF) algorithm.

[0097] EEE 35 is the method of any one of EEEs 30-34, wherein updating the model comprises rewriting data to a computer readable medium storing the model.

[0098] EEE 36 is the method of any one of EEEs 30-35, further comprising determining the first pose and the second pose using a simultaneous localization and mapping algorithm or an electromagnetic tracking sensor, wherein using the computational algorithm comprises using the computational algorithm to additionally process the first pose and the second pose to update the model.

[0099] EEE 37 is the method of any one of EEEs 30-36, further comprising displaying the model as altered by the updating. [000100] EEE 38 is the method of EEE 37, further comprising: receiving, via a user interface, a command to display the model as the model existed prior to the updating; and displaying, in response to receiving the command, the model as the model existed prior to the updating. [000101] EEE 39 is the method of EEE 38, further comprising: receiving, via the user interface, a second command to display the model as altered by the updating; and displaying, in response to receiving the second command, the model as altered by the updating. [000102] EEE 40 is the method of any one of EEEs 30-39, further comprising: using the first image, the second image, and the model to make a determination that a distance between a surgical instrument and an object within the surgical environment is less than a threshold distance; and generating, via a user interface and in response to making the determination, anindication that the distance between the surgical instrument and the object is less than the threshold distance. [000103] EEE 41 is the method of any one of EEEs 30-40, wherein updating the model comprises updating the model to include a structure added by a medical procedure. [000104] EEE 42 is the method of any one of EEEs 30-41, wherein updating the model comprises updating the model to remove a structure removed by a medical procedure. [000105] EEE 43 is a non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of any one of EEEs 30-42. [000106] EEE 44 is an imaging system comprising: one or more cameras; one or more processors; and a computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of any one of EEEs 30-42. [000107] While various example aspects and example embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various example aspects and example embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

CLAIMS What is claimed is:

1. A method comprising: accessing a first model representing a first surgical environment; accessing first images of the first surgical environment captured by a first imaging system having a different pose for each of the first images; and using the first model and the first images to train a computational algorithm to perform functions comprising: accessing a second model representing a second surgical environment; accessing second images of the second surgical environment captured by a second imaging system after a change has been made to the second surgical environment; and using the second images to update the second model to represent the change made to the second surgical environment.

2. The method of claim 1, wherein the first surgical environment and the second surgical environment are the same surgical environment.

3. The method of claim 1, wherein the first surgical environment and the second surgical environment are not the same surgical environment.

4. The method of claim 1, wherein the first imaging system and the second imaging system are the same imaging system.

5. The method of claim 1, wherein the first imaging system and the second imaging system are not the same imaging system.

6. The method of claim 1, wherein the first model includes data collected using a computed tomography (CT) scan of the first surgical environment.

7. The method of claim 1, wherein the first model includes data collected using a magnetic resonance imaging (MRI) scan of the first surgical environment.

8. The method of claim 1, wherein the first model comprises a point cloud.

9. The method of claim 1, wherein the first surgical environment comprises soft tissue or bone.

10. The method of claim 1, wherein the first surgical environment comprises a sinonasal cavity.

11. The method of claim 1, wherein the first imaging system comprises a monocular camera.

12. The method of claim 1, wherein the first imaging system includes an endoscope.

13. The method of claim 1, wherein the second model includes data collected using a computed tomography (CT) scan of the second surgical environment.

14. The method of claim 1, wherein the second model includes data collected using a magnetic resonance imaging (MRI) scan of the second surgical environment.

15. The method of claim 1, wherein the second model comprises a point cloud.

16. The method of claim 1, wherein the second surgical environment comprises soft tissue or bone.

17. The method of claim 1, wherein the second surgical environment comprises a sinonasal cavity.

18. The method of claim 1, wherein the second imaging system comprises a monocular camera.

19. The method of claim 1, wherein the second imaging system includes an endoscope.

20. The method of claim 1, wherein the change to the second surgical environment is caused by a medical procedure performed upon the second surgical environment.

21. The method of claim 20, wherein the medical procedure comprises one or more of an excision, an incision, a resection, a displacement, or an injection.

22. The method of claim 1, the functions further comprising determining poses of the second imaging system corresponding to the second images using a simultaneous localization and mapping algorithm or an electromagnetic tracking sensor, wherein using the second images to update the second model comprises updating the second model based on the second images and the poses of the second imaging system corresponding to the second images.

23. The method of claim 1, wherein a field of view of each of the first images overlaps with a field of view of at least one of the other first images by at least 90%.

24. The method of claim 1, wherein a field of view of each of the second images overlaps with a field of view of at least one of the other second images by at least 90%.

25. The method of claim 1, further comprising: generating virtual images of the first surgical environment that form stereo pairs respectively with the first images; and additionally using the stereo pairs to train the computational algorithm to perform the functions.

26. The method of claim 25, further comprising: determining a direction of movement of the first imaging system based on two or more of the first images; and generating the virtual images based on the direction of movement.

27. The method of claim 1, wherein the computational algorithm comprises a neural radiance field (NeRF) algorithm.

28. A non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of claim 1.

29. A computing device comprising: one or more processors; and a non-transitory computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of any one of claim 1.

30. A method comprising: capturing, via a camera having a first pose, a first image of a surgical environment after a change has been made to the surgical environment; capturing, via a camera having a second pose that is different from the first pose, a second image of the surgical environment after the change has been made to the surgical environment; and using a computational algorithm to process the first image, the second image, and a model representing the surgical environment, thereby updating the model to represent the change made to the surgical environment.

31. The method of claim 30, wherein the model is a first model and using the computational algorithm comprises: generating a second model by processing the first image and the second image using the computational algorithm, wherein the second model represents the surgical environment after the change has been made to the surgical environment; and using the second model to update the first model to represent the change made to the surgical environment.

32. The method of claim 31, further comprising co-registering the first model and the second model, wherein updating the first model comprises updating the first model using the second model after co-registering the first model and the second model.

33. The method of claim 30, wherein the model includes data collected using a computed tomography (CT) scan or a magnetic resonance imaging (MRI) scan of the surgical environment.

34. The method of claim 30, wherein the computational algorithm is a neural radiance field (NeRF) algorithm.

35. The method of claim 30, wherein updating the model comprises rewriting data to a computer readable medium storing the model.

36. The method of claim 30, further comprising determining the first pose and the second pose using a simultaneous localization and mapping algorithm or an electromagnetic tracking sensor, wherein using the computational algorithm comprises using the computational algorithm to additionally process the first pose and the second pose to update the model.

37. The method of claim 30, further comprising displaying the model as altered by the updating.

38. The method of claim 37, further comprising: receiving, via a user interface, a command to display the model as the model existed prior to the updating; and displaying, in response to receiving the command, the model as the model existed prior to the updating.

39. The method of claim 38, further comprising: receiving, via the user interface, a second command to display the model as altered by the updating; and displaying, in response to receiving the second command, the model as altered by the updating.

40. The method of claim 30, further comprising:using the first image, the second image, and the model to make a determination that a distance between a surgical instrument and an object within the surgical environment is less than a threshold distance; and generating, via a user interface and in response to making the determination, an indication that the distance between the surgical instrument and the object is less than the threshold distance.

41. The method of claim 30, wherein updating the model comprises updating the model to include a structure added by a medical procedure.

42. The method of claim 30, wherein updating the model comprises updating the model to remove a structure removed by a medical procedure.

43. A non-transitory computer readable medium storing instructions that, when executed by a computing device, cause the computing device to perform the method of claim 30.

44. An imaging system comprising: one or more cameras; one or more processors; and a computer readable medium storing instructions that, when executed by the one or more processors, cause the computing device to perform the method of claim 30.

Images