Method and system for registering preoperative image data to intraoperative image data of a scene, such as a surgical scene - Patents.com

Abstract

Disclosed herein are mediated reality imaging systems, methods, and devices. In some embodiments, the imaging system includes (i) a camera array configured to capture intraoperative image data of a surgical scene in substantially real-time, and (ii) a processing device communicatively coupled to the camera array. The processing device can be configured to synthesize a three-dimensional (3D) image corresponding to a virtual view of the scene based on the intraoperative image data from the cameras. The imaging system is further configured to receive and / or store preoperative image data, such as medical scan data, corresponding to a portion of a patient in the scene. The image processing device can register the preoperative image data to the intraoperative image data and overlay the registered preoperative image data on a corresponding portion of the 3D image of the scene to present a mediated reality view.

Description

[Technical field]

[0001] The present technology generally relates to methods and systems for generating a real-time or near real-time three-dimensional (3D) virtual view of a scene for a mediated reality viewer and registering previously captured image data, such as pre-operative medical images (e.g., computed tomography (CT) scan data), to the 3D virtual view. [Background technology]

[0002] In a mediated reality system, an image processing system adds, subtracts, and / or modifies visual information that represents the environment. In surgical applications, a mediated reality system can enable a surgeon to view a surgical site from a desired perspective along with contextual information that helps the surgeon perform the surgical task more efficiently and accurately. When performing surgery, surgeons often rely on preoperative three-dimensional images of the patient's anatomy, such as CT scan images. However, the usefulness of such preoperative images is limited because the images cannot be easily incorporated into the surgical procedure. For example, because the images are captured in a preoperative session, the relative anatomical positions captured in the preoperative images may change from their actual positions during the surgical procedure. Furthermore, to utilize the preoperative images during surgery, the surgeon must split his or her attention between the surgical field and the display of the preoperative images. Also, navigating between different layers of the preoperative images may also require significant attention that detracts from the surgeon's focus on the operation.

[0003] Many aspects of the present disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale. Instead, emphasis is placed upon clearly illustrating the principles of the present disclosure. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a schematic diagram of an imaging system according to an embodiment of the present technology.

[0005] [Diagram 2] 2 is a perspective view of a surgical environment employing the imaging system of FIG. 1 for surgical applications in accordance with an embodiment of the present technology.

[0006] [Diagram 3] FIG. 1 is a flow diagram of a process or method for registering pre-operative image data to intra-operative image data to generate a mediated reality view of a surgical scene, in accordance with an embodiment of the present technology.

[0007] [Figure 4A] 4 is a schematic diagram of (i) intraoperative image data of an object within the field of view of the camera array of the imaging system of FIG. 1 and (ii) preoperative image data of the object, illustrating various stages of the method of FIG. 3 in accordance with an embodiment of the present technology. [Figure 4B] 4 is a schematic diagram of (i) intraoperative image data of an object within the field of view of the camera array of the imaging system of FIG. 1 and (ii) preoperative image data of the object, illustrating various stages of the method of FIG. 3 in accordance with an embodiment of the present technology. [Figure 4C] 4 is a schematic diagram of (i) intraoperative image data of an object within the field of view of the camera array of the imaging system of FIG. 1 and (ii) preoperative image data of the object, illustrating various stages of the method of FIG. 3 in accordance with an embodiment of the present technology.

[0008] [Diagram 5] FIG. 1 is a flow diagram of a process or method for registering pre-operative image data to intra-operative image data in accordance with an embodiment of the present technique.

[0009] [Figure 6A] 6 is a perspective view of an output image of a surgical scene produced by the imaging system of FIG. 1 illustrating various stages of the method of FIG. 5 in accordance with an embodiment of the present technique. [Figure 6B]6 is a perspective view of an output image of a surgical scene produced by the imaging system of FIG. 1 illustrating various stages of the method of FIG. 5 in accordance with an embodiment of the present technique. [Figure 6C] 6 is a perspective view of an output image of a surgical scene produced by the imaging system of FIG. 1 illustrating various stages of the method of FIG. 5 in accordance with an embodiment of the present technique. [Figure 6D] 6 is a perspective view of an output image of a surgical scene produced by the imaging system of FIG. 1 illustrating various stages of the method of FIG. 5 in accordance with an embodiment of the present technique. [Figure 6E] 6 is a perspective view of an output image of a surgical scene produced by the imaging system of FIG. 1 illustrating various stages of the method of FIG. 5 in accordance with an embodiment of the present technique.

[0010] [Figure 7] FIG. 11 is a flow diagram of a process or method for registering pre-operative image data to intra-operative image data in accordance with an additional embodiment of the present technique.

[0011] [Figure 8] 2 is an image of a patient's spine captured by the camera array of the imaging system of FIG. 1 in accordance with an embodiment of the present technology.

[0012] [Figure 9] FIG. 11 is a flow diagram of a process or method for registering pre-operative image data to intra-operative image data to generate a mediated reality view of a surgical scene, in accordance with an additional embodiment of the present technique.

[0013] [Figure 10] FIG. 2 is a flow diagram of a process or method for registering a point cloud depth map of a scene to pre-operative image data of a portion of the scene in accordance with an embodiment of the present technique.

[0014] [Figure 11]1 is a graph of the accuracy of a registration algorithm over time for processing different numbers / densities of points in a point cloud in accordance with an embodiment of the present technology.

[0015] [Figure 12] FIG. 2 is a flow diagram of a process or method for determining the accuracy of registration between intraoperative image data and preoperative image data, in accordance with an embodiment of the present technique. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Aspects of the present technology are generally directed to a mediated reality imaging system, such as for use in a surgical procedure, and an associated method for registering pre-operative image data to intra-operative image data for displaying together. In some embodiments described below, for example, the imaging system includes (i) a camera array configured to capture intra-operative image data (e.g., light field data and / or depth data) of a surgical scene, and (ii) a processing device communicatively coupled to the camera array. The processing device can be configured to synthesize / generate in real time or near real time a three-dimensional (3D) virtual image corresponding to a virtual view of the scene based on image data from at least a subset of the cameras. The processing device can output the 3D virtual image to a display device (e.g., a head mounted display (HMD)) for viewing by a viewer, such as a surgeon or other operator of the imaging system. The imaging system is further configured to receive and / or store pre-operative image data. The pre-operative image data can be medical scan data (e.g., computed tomography (CT) scan data) corresponding to a portion of a patient in a scene, such as the spine of a patient undergoing a spinal surgical procedure.

[0017] The processing device can globally and / or locally register the preoperative image data to the intraoperative image data, for example, by registering / matching fiducial markers and / or other feature points visible in the 3D data sets representing both the preoperative image data and the intraoperative image data. The processing device can further apply a transformation to the preoperative image data based on the registration, for example, to substantially align (e.g., in a common coordinate frame) the preoperative image data with the real-time or near real-time intraoperative image data captured by the camera array. The processing device can then display the preoperative image data and the intraoperative image data together to provide a mediated reality view of the surgical scene. More specifically, the processing device can overlay a 3D graphical representation of the preoperative image data on a corresponding portion of a 3D virtual image of the scene to present the mediated reality view, for example, allowing a surgeon to simultaneously view the surgical site in the scene and the underlying 3D anatomical structure of the patient during surgery.

[0018] In some embodiments, the processing device of the imaging system can perform a method for registering pre-operative image data, such as medical scan data, to intra-operative image data, including overlaying unregistered medical scan data on top of the 3D virtual image. The method can further include receiving a user input to move the medical scan data into alignment with a corresponding portion of the patient (e.g., a selected anatomical structure of the patient) at least partially visible in the 3D virtual image. For example, the medical scan data can be segmented vertebrae from a CT scan, and the user can virtually "drag-and-drop" the vertebrae to align with the corresponding vertebrae shown in the 3D virtual image by moving a tool through the scene. Once the medical scan data is manually aligned by the user, the method can include registering the medical scan data to the intra-operative image based on the alignment. In some embodiments, the registration can be a local registration that further aligns the medical scan data to the intra-operative image data. Such a local registration can be visually represented in the 3D virtual image by "snapping" the medical scan data to a location on a corresponding anatomical structure of the patient in the 3D virtual image.

[0019] In some embodiments, a processing device of the imaging system may implement a method of registering pre-operative medical scan data to intra-operative image data based on one or more characteristics of the intra-operative image data, such as color, specularity, etc. More specifically, the method may include analyzing the intra-operative light field image data to determine one or more characteristics, and determining based on the determined one or more characteristics that (i) a first portion of the intra-operative image data corresponds to a first type of anatomical structure of the patient, and (ii) a second portion of the intra-operative image data corresponds to a second type of anatomical structure of the patient. The first type of anatomical structure may correspond to the medical scan data. For example, the medical scan data may be a CT scan of the patient's spine, and the first type of anatomical structure of the patient may be a vertebra. In some embodiments, the method may include adjusting weights of a registration algorithm based on whether a point in the intra-operative image data is the first type of anatomical structure or the second type of anatomical structure. For example, points that are likely to be bone may be weighted higher than points that are likely to be flesh or other anatomical structures of the patient that do not correspond to the medical scan data.

[0020] In some embodiments, a processing device of the imaging system can implement a method for registering pre-operative medical scan data to intra-operative image data, the method including processing intra-operative depth data of a scene. More specifically, the method can include processing the intra-operative image data to generate a point cloud depth map of the scene. The method can then utilize a registration algorithm that maps the point cloud depth map to the pre-operative medical scan data. In some embodiments, the processing device of the imaging system can generate a 3D mesh based on the point cloud depth map that can be used, for example, in generating a 3D virtual image of the scene. Thus, the registration algorithm can begin based on the point cloud depth map rather than the 3D mesh. In some aspects of the present technology, the point cloud depth map can be utilized to perform registration in parallel with the generation of the 3D mesh and subsequent synthesis of the 3D virtual image, thereby improving the processing speed of the imaging system.

[0021] In some embodiments, the processing device of the imaging system may implement / utilize a registration algorithm that processes an increasing number / density of points in the point cloud depth map in a stepwise manner until sufficient registration accuracy is achieved. For example, the registration algorithm may first process a first number of points in the point cloud and, after reaching a predefined accuracy, continue the registration based on a second, higher number of points in the point cloud. In some embodiments, the method may include processing (e.g., increasing the number of) an increasing number of points in the point cloud until sufficient registration accuracy is reached. In some aspects of the present technology, such stepwise processing may improve the processing speed of the imaging system.

[0022] In some embodiments, a processing device (and / or another processing device) of the imaging system can implement a method for evaluating the accuracy of a calculated intraoperative registration transformation that defines a mapping between intraoperative image data and preoperative image data. More specifically, the method can include (i) receiving historical registration data including the historical registration transformation, (ii) defining a spatial neighborhood around the registration transformation, (iii) classifying / labeling the registration transformation (e.g., as a "good" or "bad" transformation), and (iv) training a machine learning model based on the spatial neighborhood and the classification. The method can further include determining the accuracy of the intraoperative registration transformation by defining a spatial neighborhood around the intraoperative registration transformation, and inputting the intraoperative registration transformation to a machine learning model that can output a goodness-of-fit score (e.g., "good", "bad") of the registration. In some aspects of the present technology, evaluating a neighborhood around a given registration transformation rather than the transformation alone increases confidence in the evaluation of registration accuracy.

[0023] Specific details of some embodiments of the technology of the present invention are described herein with reference to Figures 1-12. However, the technology of the present invention can be practiced without some of these specific details. In some cases, well-known structures and techniques often associated with camera arrays, light field cameras, image reconstruction, registration processes, etc., are not shown in detail so as not to obscure the technology of the present invention. The terms used in the description presented below are intended to be interpreted in the broadest and most reasonable manner, even when used in connection with the detailed description of certain specific embodiments of the present disclosure. Although certain terms may be highlighted below, any terms intended to be interpreted in any limited manner are thus clearly and specifically defined in this detailed description section.

[0024] Additionally, while the registration techniques of the present technology are often described in the context of registering pre-operative image data to intra-operative image data of a surgical scene, they can also be used to register other types of image data. For example, the systems and methods of the present technology can be used more generally to register any previously captured data to corresponding real-time or near real-time image data of the scene to generate a mediated reality view of the scene that includes a combination / fusion of the previously captured data with the real-time imagery.

[0025] The accompanying figures illustrate embodiments of the technology of the present invention and are not intended to limit its scope. The sizes of the various elements depicted are not necessarily drawn to scale, and these various elements may be arbitrarily enlarged to improve visibility. Details of components may be generalized in the figures to exclude details such as the location of components and specific precise connections between such components when such details are not necessary for a complete understanding of how to make and use the technology of the present invention. Many of the details, dimensions, angles, and other features shown in the figures are merely illustrative of certain embodiments of the present disclosure. Thus, other embodiments may have other details, dimensions, angles, and features without departing from the spirit or scope of the technology of the present invention.

[0026] The headings provided herein are for convenience only and are not to be construed as limiting the subject matter disclosed.

[0027] I. Selected Embodiments of the Imaging System 1 is a schematic diagram of an imaging system 100 ("system 100") in accordance with embodiments of the present technology. In some embodiments, system 100 can be a synthetic augmented reality system, a mediated reality imaging system, and / or a computational imaging system. In the illustrated embodiment, system 100 includes a processing device 102 operatively / communicatively coupled to one or more display devices 104, one or more input controllers 106, and a camera array 110. In other embodiments, system 100 can include additional, fewer, or different components. In some embodiments, system 100 may include certain features that are substantially similar to or identical to the mediated reality imaging systems disclosed in (i) U.S. patent application Ser. No. 16 / 586,375, filed on September 27, 2019, and entitled "CAMERA ARRAY FOR A MEDIATED-REALITY SYSTEM," and / or (ii) U.S. patent application Ser. No. 15 / 930,305, filed on May 12, 2020, and entitled "METHODS AND SYSTEMS FOR IMAGING A SCENE, SUCH AS A MEDICAL SCENE, AND TRACKING OBJECTS WITHIN THE SCENE," each of which is incorporated by reference in its entirety herein.

[0028] In the illustrated embodiment, the camera array 110 includes a number of cameras 112 (individually identified as cameras 112a-112n and may also be referred to as first cameras), each configured to capture an image (e.g., first image data) of a scene 108 from a different perspective. The scene 108 may be, for example, a surgical scene including a patient undergoing surgery or another medical procedure. In other embodiments, the scene 108 may be other types of scenes. The camera array 110 further includes a number of dedicated object trackers 113 (individually identified as trackers 113a-113n) configured to capture position data of one or more objects, such as a tool 101 (e.g., a surgical tool) having a tip 109, and to track the movement and / or orientation of the objects through / within the scene 108. In some embodiments, the cameras 112 and trackers 113 are disposed at fixed positions and orientations (e.g., poses) relative to one another. For example, the cameras 112 and trackers 113 may be structurally fixed by / to a mounting structure (e.g., a frame) at predefined, fixed positions and orientations. In some embodiments, the cameras 112 may be positioned such that adjacent cameras 112 share overlapping views of the scene 108. Similarly, the trackers 113 may be positioned such that adjacent trackers 113 share overlapping views of the scene 108. Thus, all or a subset of the cameras 112 and trackers 113 may have different extrinsic parameters, such as positions and orientations.

[0029] In some embodiments, the cameras 112 in the camera array 110 are synchronized to capture images of the scene 108 substantially simultaneously (e.g., within a threshold time error). In some embodiments, all or a subset of the cameras 112 may be light field / plenoptic / RGB cameras configured to capture information about the light field emanating from the scene 108 (e.g., information about the intensity of light rays in the scene 108, and also information about the direction in which the light rays travel through space). Thus, in some embodiments, the images captured by the cameras 112 may encode depth information that represents the surface shape of the scene 108. In some embodiments, the cameras 112 are substantially identical. In other embodiments, the cameras 112 may include multiple cameras of different types. For example, different subsets of the cameras 112 may have different intrinsic parameters such as focal length, sensor type, optical components, and the like. The cameras 112 may have charge-coupled device (CCD) and / or complementary metal-oxide semiconductor (CMOS) image sensors and associated optics. Such optics can include a variety of configurations, including lensed or bare individual image sensors in combination with larger macro lenses, microlens arrays, prisms, and / or negative lenses. For example, the cameras 112 can be separate light field cameras, each with its own image sensor and optics. In some embodiments, some or all of the cameras 112 can comprise separate microlenslets (e.g., lenslets, lenses, microlenses) of a microlens array (MLA) that share a common image sensor.

[0030] In some embodiments, the trackers 113 are imaging devices such as infrared (IR) cameras, each configured to capture an image of the scene 108 from a different perspective compared to the others of the trackers 113. Thus, the trackers 113 and the cameras 112 may have different spectral sensitivities (e.g., infrared vs. visible wavelengths). In some embodiments, the trackers 113 are configured to capture image data of multiple optical markers (e.g., fiducial markers, marker balls) in the scene 108, such as the marker 111 coupled to the tool 101.

[0031] In the illustrated embodiment, the camera array 110 further includes a depth sensor 114. In some embodiments, the depth sensor 114 includes (i) one or more projectors 116 configured to project a structured light pattern onto / into the scene 108, and (ii) one or more depth cameras 118 (which may also be referred to as second cameras) configured to capture second image data of the scene 108 including the structured light projected onto the scene 108 by the projectors 116. The projectors 116 and the depth cameras 118 can operate at the same wavelength, or in some embodiments, at a different wavelength than the cameras 112. For example, the cameras 112 can capture first image data in the visible spectrum, while the depth cameras 118 capture second image data in the infrared spectrum. In some embodiments, the depth cameras 118 have a resolution that is smaller than the resolution of the cameras 112. For example, depth camera 118 may have a resolution that is less than 70%, 60%, 50%, 40%, 30%, or 20% of the resolution of camera 112. In other embodiments, depth sensor 114 may include other types of dedicated depth-sensing hardware (e.g., a LiDAR detector) for determining the surface geometry of scene 108. In other embodiments, camera array 110 may omit projector 116 and / or depth camera 118.

[0032] In the illustrated embodiment, the processing device 102 includes an image processing device 103 (e.g., an image processor, an image processing module, an image processing unit), a registration processing device 105 (e.g., a registration processor, a registration processing module, a registration processing unit), and a tracking processing device 107 (e.g., a tracking processor, a tracking processing module, a tracking processing unit). The image processing device 103 is configured to (i) receive first image data (e.g., a light field image, a light field image data, an RGB image) captured by the camera 112 and depth information (e.g., a second image data captured by the depth camera 118) from the depth sensor 114, and (ii) process the image data and the depth information to synthesize (e.g., generate, reconstruct, render) a three-dimensional (3D) output image of the scene 108 corresponding to a virtual camera viewpoint. The output image can correspond to an approximation of an image of the scene 108 that would be captured by a camera positioned at any position and orientation corresponding to the virtual camera viewpoint. In some embodiments, image processing device 103 is further configured to receive and / or store calibration data for camera 112 and / or depth camera 118 and synthesize an output image based on the image data, the depth information, and / or the calibration data. More specifically, the depth information and the calibration data can be used / combined with the image from camera 112 to synthesize an output image as a 3D (or stereoscopic 2D) rendering of scene 108 from the perspective of the virtual camera. In some embodiments, image processing device 103 can synthesize an output image using any of the methods disclosed in patent application Ser. No. 16 / 457,780, entitled “SYNTHESIZING AN IMAGE FROM A VIRTUAL PERSPECTIVE USING PIXELS FROM A PHYSICAL IMAGER ARRAY WEIGHTED BASED ON DEPTH ERROR SENSITIVITY,” which is incorporated herein by reference in its entirety.In other embodiments, the image processing device 103 is configured to generate the virtual camera viewpoint based solely on images captured by the cameras 112, without utilizing depth information from the depth sensor 114. For example, the image processing device 103 can generate the virtual camera view by interpolating between different images captured by one or more cameras 112.

[0033] The image processing device 103 may synthesize an output image from images captured by a subset (e.g., two or more) of the cameras 112 in the camera array 110, and may not necessarily utilize images from all of the cameras 112. For example, for a given virtual camera viewpoint, the processing device 102 may select a stereoscopic pair of images from two of the cameras 112 positioned and oriented to most closely match the virtual camera viewpoint. In some embodiments, the image processing device 103 (and / or the depth sensor 114) is configured to estimate the depth of each surface point of the scene 108 relative to a common origin and generate a point cloud and / or a 3D mesh that represents the surface shape of the scene 108. For example, in some embodiments, the depth camera 118 of the depth sensor 114 may detect structured light projected by the projector 116 onto the scene 108 to estimate depth information of the scene 108. In some embodiments, the image processing device 103 can estimate depth from multi-view image data from the cameras 112 using techniques such as light field correspondence, stereo block matching, photometric symmetry, correspondence, defocus, block matching, texture-assisted block matching, structured light, etc., with or without the use of information collected by the depth sensor 114. In other embodiments, depth can be obtained by a specialized set of cameras 112 performing the aforementioned methods at a different wavelength.

[0034] In some embodiments, the registration processing device 105 is configured to receive and / or store previously captured image data, such as pre-operative image data of a three-dimensional volume of a patient. The pre-operative image data may include, for example, computed tomography (CT) scan data, magnetic resonance imaging (MRI) scan data, ultrasound images, fluoroscopic images, etc. As described in more detail below with reference to FIGS. 3-12, the registration processing device 105 is further configured to register the pre-operative image data to real-time images captured by the camera 112 and / or depth sensor 114, for example, by determining one or more transformations / transformations / mappings between the two. The processing device 102 (e.g., image processing device 103) can then apply one or more transformations to the pre-operative image data such that the pre-operative image data can be aligned (e.g., overlaid) with the output image of the scene 108 in real-time or near real-time on a frame-by-frame basis, even as the virtual field of view changes. That is, the image processing device 103 can fuse pre-operative image data with a real-time output image of the scene 108 to present a mediated reality view that allows, for example, a surgeon to simultaneously view the surgical site in the scene 108 and the underlying 3D anatomy of the patient during surgery.

[0035] In some embodiments, the tracking processing device 107 can process position data captured by the trackers 113 to track objects (e.g., tools 101) within the vicinity of the scene 108. For example, the tracking processing device 107 can determine the position of the marker 111 within 2D images captured by one or more trackers 113 and can calculate the 3D position of the marker 111 through triangulation of the 2D position data. More specifically, in some embodiments, the trackers 113 include dedicated processing hardware for determining position data from the captured images, such as the center of gravity of the marker 111 within the captured images. The trackers 113 can then transmit the position data to the tracking processing device 107 to determine the 3D position of the marker 111. In other embodiments, the tracking processing device 107 can receive raw image data from the trackers 113. For example, in a surgical application, the tracked objects can include a surgical instrument, a doctor's or assistant's hand or arm, and / or another object to which the marker 111 is attached. In some embodiments, the processing device 102 may recognize that the tracked object is separate from the scene 108 and may apply visual effects to the 3D output image to distinguish the tracked object, for example, by highlighting the object, labeling the object, and / or applying transparency to the object.

[0036] In some embodiments, the functionality attributed to the processing device 102, the image processing device 103, the registration processing device 105, and / or the tracking processing device 107 may be substantially implemented by two or more physical devices. For example, in some embodiments, a synchronization controller (not shown) controls the images displayed by the projector 116 and sends synchronization signals to the cameras 112 to ensure synchronization between the cameras 112 and the projector 116, enabling high speed multi-frame, multi-camera structured light scanning. Furthermore, such a synchronization controller may act as a parameter server that stores hardware specific configurations such as parameters for structured light scanning, camera settings, and camera calibration data specific to the camera configuration of the camera array 110. The synchronization controller may be implemented in a separate physical device from the display controller that controls the display device 104, or the devices may be integrated together.

[0037] The processing device 102 may comprise a processor and a non-transitory computer-readable storage medium that stores instructions that, when executed by the processor, perform functions attributed to the processing device 102 as described herein. Although not required, aspects and embodiments of the present technology may be described in the general context of computer-executable instructions, such as routines executed by a general purpose computer, e.g., a server or a personal computer. Those skilled in the relevant art will appreciate that the present technology may be implemented in other computer system configurations, including Internet appliances, handheld devices, wearable computers, mobile or cellular phones, multiprocessor systems, microprocessor-based or programmable appliances, set-top boxes, network PCs, minicomputers, mainframe computers, and the like. The present technology may be embodied in a special purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described in detail below. Indeed, the term "computer" (and similar terms) as generally used herein refers to any data processor or any device capable of communicating with a network, including the above devices as well as consumer electronic products, such as gaming devices, cameras, or other electronic devices having a processor and other components, e.g., network communication circuitry.

[0038] The inventive technology may also be practiced in a distributed computing environment where tasks or modules are performed by remote processing devices that are linked through a communications network, such as a local area network ("LAN"), a wide area network ("WAN"), or the Internet. In a distributed computing environment, program modules or subroutines may be located in both local and remote memory storage devices. Aspects of the inventive technology described below may be stored or distributed on computer readable media, including stored in a chip (e.g., an EEPROM or flash memory chip), magnetically and optically readable, and removable computer disks. Alternatively, aspects of the inventive technology may be distributed electronically over the Internet or other networks (including wireless networks). Those skilled in the relevant art will recognize that parts of the inventive technology may reside on a server computer and corresponding parts may reside on a client computer. Data structures and data transmission specific to aspects of the inventive technology are also encompassed within the scope of the inventive technology.

[0039] The virtual camera viewpoint can be controlled by an input controller 106, which provides control inputs corresponding to the position and orientation of the virtual camera viewpoint. An output image corresponding to the virtual camera viewpoint can be output to a display device 104. In some embodiments, the image processing device 103 can vary (e.g., based on input from the input controller) the viewpoint, depth of field (e.g., aperture), focal plane, and / or another parameter of the virtual camera to generate different 3D output images without physically moving the camera array 110. The display device 104 is configured to receive the output images (e.g., a composited 3D rendering of the scene 108) and display the output images for viewing by one or more viewers. In some embodiments, the processing device 102 can receive and process inputs from the input controller 106 and process the captured images from the camera array 110 to generate output images corresponding to the virtual field of view in substantially real-time as perceived by a viewer of the display device 104 (e.g., at least as fast as the frame rate of the camera array 110). Additionally, the display device 104 may display a graphical representation of any (i) tracked objects (e.g., surgical tools) within the scene 108 and / or (ii) registered or unregistered pre-operative image data within the virtual field of view image.

[0040] The display device 104 may comprise, for example, a head-mounted display device, a monitor, a computer display, and / or another display device. In some embodiments, the input controller 106 and the display device 104 are integrated into a head-mounted display device, and the input controller 106 comprises a motion sensor that detects the position and orientation of the head-mounted display device. The viewpoint of the virtual camera may then be derived to correspond to the position and orientation of the head-mounted display device 104 at a calculated depth (e.g., calculated by the depth sensor 114) in the same reference frame such that the virtual field of view corresponds to the viewpoint seen by a viewer wearing the head-mounted display device 104. Thus, in such an embodiment, the head-mounted display device 104 may provide a real-time rendering of the scene 108 as seen by an observer not wearing the head-mounted display device 104. Alternatively, the input controller 106 may comprise a user-controlled control device (e.g., a mouse, a pointing device, a handheld controller, a gesture-recognition controller, etc.) that allows the viewer to manually control the virtual field of view displayed by the display device 104.

[0041] FIG. 2 is a perspective view of a surgical environment employing a system 100 for surgical applications according to an embodiment of the present technology. In the illustrated embodiment, the camera array 110 is positioned over the scene 108 (e.g., a surgical site) and supported / positioned via a movable arm 222 operably coupled to a workstation 224. In some embodiments, the arm 222 can be moved manually to position the camera array 110, while in some embodiments, the arm 222 can be robotically controlled in response to the input controller 106 (FIG. 1) and / or another controller. In the illustrated embodiment, the display device 104 is a head-mounted display device (e.g., a virtual reality headset, an augmented reality headset, etc.). The workstation 224 can include a computer for controlling various functions of the processing device 102, the display device 104, the input controller 106, the camera array 110, and / or other components of the system 100 shown in FIG. 1. Thus, in some embodiments, the processing device 102 and the input controller 106 are each integrated into the workstation 224. In some embodiments, the workstation 224 includes a secondary display 226 capable of displaying a user interface for performing various configuration functions, a mirror image of the display on the display device 104, and / or other useful visual images / displays.

[0042] II. Selected embodiments of the registration technique 3 is a flow diagram of a process or method 330 for registering pre-operative image data to / with intra-operative image data to generate a mediated reality view of a surgical scene, according to an embodiment of the present technology. Although some features of the method 330 are described in the context of the system 100 shown in FIGS. 1 and 2 for illustrative purposes, one of ordinary skill in the art will readily appreciate that the method 330 can be implemented using other suitable systems and / or devices described herein. Similarly, although reference is made herein to pre-operative image data, intra-operative image data, and a surgical scene, the method 330 can be used to register and display other types of information relating to other scenes. For example, the method 330 can be used more generally to register any previously captured image data to corresponding real-time or near real-time image data of the scene to generate a mediated reality view of the scene, including a combination / fusion of the previously captured image data and the real-time image data. 4A-4C are schematic diagrams of intraoperative image data 440 of an object within the field of view of camera array 110 and corresponding preoperative image data 442 of the object, illustrating various stages of method 330 of FIG. 3 in accordance with an embodiment of the present technology. Accordingly, several aspects of method 330 will be described in relation to FIGS. 4A-4C.

[0043] At block 331, the method 330 includes receiving pre-operative image data. As described in detail above, the pre-operative image data may be medical scan data representing a three-dimensional volume of a patient, such as, for example, computed tomography (CT) scan data, magnetic resonance imaging (MRI) scan data, ultrasound images, fluoroscopic images, etc. In some embodiments, the pre-operative image data may comprise a point cloud or a three-dimensional (3D) mesh.

[0044] At block 332, the method 330 includes receiving intraoperative image data of the surgical scene 108, for example from the camera array 110. The intraoperative image data may include real-time or near real-time images of the patient in the scene 108 captured by the camera 112 and / or the depth camera 118. In some embodiments, the intraoperative image data includes (i) a light field image from the camera 112 and (ii) an image from the depth camera 118 that includes encoded depth information about the scene 108. In some embodiments, the preoperative image data corresponds to at least some of the features in the intraoperative image data. For example, the scene 108 may include a patient undergoing spinal surgery with the spine at least partially exposed. The preoperative image data may include CT scan data of the patient's spine taken prior to surgery, which comprises a complete 3D data set of at least a portion of the spine. Thus, various vertebrae or other features in the preoperative image data may correspond to portions of the patient's spine represented in the image data from the cameras 112, 118. In other embodiments, scene 108 may include a patient undergoing another type of surgery, such as knee surgery, skull surgery, etc., and the pre-operative image data may include CT or other scan data of ligaments, bones, flesh, and / or other anatomical structures relevant to the particular surgical procedure.

[0045] More specifically, with reference to FIG. 4A, the object may include multiple portions 441 (individually identified as first through fifth portions 441a-441e, respectively) that are represented in both the intraoperative image data 440 and the preoperative image data 442. The object may be, for example, a patient's spine, and the sub-portions 441 may include individual vertebrae of the spine. The preoperative image data 442 and the intraoperative image data 440 of the object typically exist in different coordinate systems such that the same features (e.g., sub-portions 441) in both data sets are represented differently. In the illustrated embodiment, for example, each of the sub-portions 441 of the preoperative image data 442 may be rotated, scaled, and / or translated relative to a corresponding one of the sub-portions 441 of the intraoperative image data 440 of the object.

[0046] Thus, in block 333, the method 330 includes globally registering the preoperative image data to the intraoperative image data, for example to establish a transformation / mapping / conversion between the intraoperative image data and the preoperative image data, so that these data sets are expressed in the same coordinate system and can then be displayed together. For example, FIG. 4B shows intraoperative image data 440 and preoperative image data 442 of an object after global registration. In the illustrated embodiment, after globally registering the preoperative image data 442 to the intraoperative image data 440 of the object, the sub-portions 441 can be at least roughly aligned (e.g., in intraoperative image space, coordinate system, and / or frame) in each data set. In some embodiments, the global registration process (i) matches 3D points in a point cloud or 3D mesh representing the preoperative image data to (ii) 3D points in a point cloud or 3D mesh representing the intraoperative image data. In some embodiments, the system 100 (e.g., the registration processing device 105) can generate a 3D point cloud from intraoperative image data from the depth camera 118 of the depth sensor 114 and can register the point cloud to the preoperative image data by detecting the locations of fiducial markers and / or feature points visible in both data sets. For example, if the preoperative image data includes CT scan data, rigid bodies of bone surfaces calculated from the CT scan data can be registered to corresponding points / surfaces in the point cloud. In other embodiments, the system 100 can employ other registration processes based on other methods of shape correspondence and / or registration processes that do not rely on fiducial markers (e.g., markerless registration processes).In some embodiments, the registration / alignment process may include features generally similar or identical to the registration / alignment process disclosed in U.S. Provisional Patent Application No. 16 / 749,963, entitled "ALIGNING PREOPERATIVE SCAN IMAGES TO REAL-TIME OPERATIVE IMAGES FOR A MEDIATED-REALITY VIEW OF A SURGICAL SITE," filed on January 22, 2020, which is incorporated herein by reference in its entirety. In still other embodiments, global registration may be performed using any of the registration methods described in detail below, for example, with reference to Figures 5-6E.

[0047] In some aspects of the present technology, the algorithm used to globally register the pre-operative image data to the intra-operative image data does not require alignment for initialization. That is, the global registration algorithm can generate a transformation between the pre-operative image data and the intra-operative image data even when the initial mapping is not known. In some embodiments, referring again to FIG. 4B, the global registration process can result in a relatively loose alignment, for example, where some of the sub-portions 441 are rotated, translated, and / or scaled differently from one another in a common coordinate space. Thus, in block 334, the method 330 can include locally registering at least a portion of the pre-operative image data to the intra-operative image data. For example, FIG. 4C illustrates intra-operative image data 440 and pre-operative image data 442 of an object after local registration. In the illustrated embodiment, each of the sub-portions 441 is locally registered to provide a tighter alignment than the global registration illustrated in FIG. 4B. In other embodiments, fewer than all of the sub-portions 441 and / or different subsets of the sub-portions 441 can be locally registered. For example, only the vertebra or vertebrae being operated on may be locally registered, while the others of the vertebrae remain only in global registration or may not be registered at all. In some embodiments, the registration processing device 105 may utilize a local registration algorithm that requires a rough alignment for initialization, such as the result of the global registration (block 333). For example, the registration processing device 105 may utilize any feature or surface matching registration method to achieve exact registration, such as iterative closest point (ICP), coherent point drift (CPD), or algorithms based on probability density estimation, such as Gaussian mixture model (GMM).

[0048] At block 335, the method 330 may include generating one or more transformations for the pre-operative image data based on the global registration and the local registration (blocks 333 and 334). The one or more transformations may be functions that define a mapping between a coordinate system of the pre-operative image data and a coordinate system of the intra-operative image data. At block 336, the registration processing device 105 may include applying the transformations to the pre-operative image data in real-time or near real-time. Applying the transformations to the pre-operative image data may substantially align the pre-operative image data with a real-time or near real-time image of the scene 108 captured by the camera array 110.

[0049] Finally, in block 337, the method 330 can include displaying the transformed pre-operative image data and the intra-operative image data together to provide a mediated reality view of the surgical scene. This view can be provided to a viewer, such as a surgeon, on the display device 104. More specifically, the image processing device 102 can overlay the aligned pre-operative image data onto the output image of the scene 108 in real-time or near real-time on a frame-by-frame basis, even as the virtual field of view changes. That is, the image processing device 103 can overlay the pre-operative image data with the real-time output image of the scene 108 to present a mediated reality view that allows, for example, a surgeon to simultaneously view the surgical site in the scene 108 and the underlying 3D anatomy of the patient during surgery.

[0050] 3-4C together, in some embodiments, the position and / or shape of objects in the scene 108 may change over time. For example, the relative positions and orientations of sub-portions 441, such as vertebrae, may change during a surgical procedure as a patient is operated on. Thus, the method 330 may include periodically re-registering the pre-operative image data to the intra-operative image data globally (block 333) and / or locally (block 334). In some embodiments, the re-registration may be triggered when the accuracy (e.g., score, level) of the registration falls below a threshold level. In some embodiments, for example, such accuracy determination may be performed using a method for evaluating registration accuracy, which is described in more detail below with reference to FIG. 12.

[0051] FIG. 5 is a flow diagram of a process or method 550 for registering pre-operative image data to intra-operative image data in accordance with an embodiment of the present technology. In some embodiments, the method 550 may be used to globally register pre-operative image data to intra-operative image data in block 333 of the method 330 described in detail above with reference to FIGS. 3-4C. Although some features of the method 550 are described in the context of the system 100 shown in FIGS. 1 and 2 for illustrative purposes, one of ordinary skill in the art will readily appreciate that the method 550 may be implemented using other suitable systems and / or devices described herein. FIGS. 6A-6E are perspective views of output images of a scene 108 (e.g., a surgical scene) generated by the system 100 and viewable by a viewer, illustrating various stages of the method 550 of FIG. 5 in accordance with an embodiment of the present technology. Accordingly, some aspects of the method 550 are described in the context of FIGS. 6A-6E.

[0052] At block 551, the method 550 includes receiving pre-operative image data. As described in detail above, the pre-operative image data may include medical scan data representing a three-dimensional volume of a patient, such as computed tomography CT scan data. At block 552, the method 550 includes receiving intra-operative image data of the surgical scene 108 from the camera array 110. As described in detail above, the intra-operative image data may include real-time or near real-time images from the camera 112 and / or the depth camera 118, such as images of the spine of a patient undergoing spinal surgery.

[0053] At block 553, the method 550 includes generating and displaying a 3D output image / view of the surgical scene based on the intraoperative image data. As described in detail above with reference to FIG. 1, the processing device 102 can receive the intraoperative image data from the depth sensor 114 and the camera 112 and process the intraoperative image data to synthesize (e.g., generate, reconstruct, render) a 3D output image of the scene 108 corresponding to a virtual camera viewpoint selected, for example, by the input controller 106. The 3D output image can correspond to an approximation of an image of the scene 108 that would be captured by a camera positioned at any position and orientation corresponding to the virtual camera viewpoint, and can be updated and displayed to the user via the display device 104 in substantially real-time as perceived by the user. FIG. 6A illustrates, for example, a 3D output image of the scene 108 that is visible to a user (e.g., a surgeon) viewing the display device 104. In some embodiments, the scene 108 can include an object of interest (e.g., for registration purposes). In the illustrated embodiment, for example, scene 108 is a spinal surgery scene including vertebrae 659 (individually identified as first through fourth vertebrae 659a through 659d, respectively) exposed from flesh 665 of a patient during, for example, a spinal fusion or other spinal surgical procedure.

[0054] At block 554, the method 550 includes displaying at least a portion of the pre-operative image data in the 3D output image of the surgical scene. The pre-operative image data may be of / corresponding to an object of interest in the scene 108 and may be unregistered to the intra-operative image data. In some embodiments, the pre-operative image data may be overlaid on the 3D output image of the surgical scene so that they can be simultaneously viewed by a user. For example, FIG. 6B shows pre-operative image data 642 overlaid on the output image of the scene 108. In the illustrated embodiment, the pre-operative image data 642 includes CT scan data of the second vertebra 659b. In some embodiments, the displayed pre-operative image data 642 may be a segmented portion of the CT scan that includes information about multiple ones of the vertebrae. That is, the pre-operative image data 642 overlaid on the virtual rendering of the scene 108 may be a portion or segment of a larger set of pre-operative image data. In some embodiments, the system 100 may display the pre-operative image data 642 based on the position of the tool 101 in the scene. In the illustrated embodiment, for example, the pre-operative image data 642 is displayed on the tip 109 of the tool 101, which is therefore movable within the scene 108. In some embodiments, the system 100 can render all or a portion of the tool 101 within the scene 108 (e.g., as shown in FIG. 6B), while in other embodiments, the tool 101 can be omitted from the 3D output image.

[0055] At block 555, the method 550 includes receiving a first user input for moving the displayed pre-operative image data relative to the 3D output image of the surgical scene. The first user input can be for manually aligning the displayed pre-operative image data over a corresponding portion of the 3D output image of the surgical scene. With reference to both FIG. 6C and FIG. 6D, for example, the user can move the tool 101 to translate (e.g., drag), rotate, and / or otherwise move the pre-operative image data 642 relative to (e.g., onto) the rendering of the scene 108 until the pre-operative image data 642 is generally aligned with the corresponding second vertebra 659b. That is, the user can physically manipulate the tool 101 relative to the surgical scene 108 to generally align / register the pre-operative image data 642 with the intra-operative image data (e.g., the second vertebra 659b). In some embodiments, the system 100 can track the movement of the tool 101 relative to the scene 108 via the tracker 113 and translate that movement into a virtual movement of the pre-operative image data 642. In other embodiments, the system 100 can track other objects in the scene 108, such as a user's hand (e.g., one or more fingers of the user), and translate that movement into a movement of the pre-operative image data 642.

[0056] At block 556, the method 550 includes receiving a second user input indicating that the displayed pre-operative image data is aligned over the corresponding portion of the 3D output image of the surgical scene. For example, referring to FIG. 6D, the user may instruct the system 100 to detach the tool 101 (FIG. 6C) from the pre-operative image data 642 after generally aligning the pre-operative image data 642 with the intra-operative image data of the second vertebra 659b. In some embodiments, the second user input may include a button press (e.g., of a button on the tool 101), a voice command, a hand movement, and / or another suitable instruction recognizable by the system 100. That is, after dragging the pre-operative image data 642 to a predetermined location, the user may "drop" the pre-operative image data at the location by instructing the system 100. In other embodiments, the system 100 may automatically detect that the pre-operative image data is aligned over the corresponding portion of the 3D output image.

[0057] At block 557, the method 550 may include generating a registration transformation between the pre-operative image data and the intra-operative image data based on an alignment of the pre-operative image data and corresponding portions of the 3D output image. As described in detail above with reference to Figures 3-4C, for example, the registration transformation may be a global transformation that defines a mapping between a coordinate system of the pre-operative image data and a coordinate system of the intra-operative image data.

[0058] At block 558, the method 550 may include locally registering the displayed pre-operative image data to a corresponding portion of the 3D output image of the surgical scene. As described in detail above with reference to FIGS. 3-4C and as shown in FIG. 6E, the local registration may tighten / improve the alignment of the pre-operative image data 642 to the intra-operative image data (e.g., the second vertebra 659b) (blocks 555-557), for example, using an ICP algorithm, a CPD algorithm, a GMM algorithm, and / or another algorithm initialized with a general alignment / transformation provided by the user's manual registration of the pre-operative image data 642. In some embodiments, the local registration may "snap" the pre-operative image data 642 into alignment. In some embodiments, the system 100 may prompt the user to repeat the manual alignment (blocks 555 and 556) if the accuracy of the local registration is not within a threshold tolerance.

[0059] In some aspects of the present technology, the method 550 allows a user to visualize a surgical scene and drag (block 555) and drop (block 556) the preoperative image data to align with the corresponding portion of the scene before automatically snapping (block 558) the preoperative image data to further align it. Furthermore, the registration is based on a large number of points that make up the preoperative image data and the corresponding portion of the scene, and can be performed simply and easily by the user. In contrast, conventional registration techniques typically require a user (e.g., a surgeon) to repeatedly tap corresponding points in the CT scan and corresponding points on the patient to register the CT scan to the patient. Thus, the registration is based on a relatively small number of tapped points and is time-consuming for the user. For example, the user must repeatedly move his or her head to tap points on the CT scan and the patient, whereas the method 550 of the present technology provides an integrated registration that is simple and intuitive.

[0060] In some embodiments, the system 100 can attempt to locally register the pre-operative image data to the scene 108 (block 557) while the user attempts to manually align the pre-operative image data (blocks 555 and 556). Based on the simultaneous local registration, the system 100 can help guide the user to manually place the pre-operative image data in the correct position. For example, as the user approaches the correct position, the local registration algorithm can provide an indication to the user that the pre-operative image data is approximately aligned. For example, with reference to FIGS. 6A-6E, the system 100 can create a "gravity well" effect around the second vertebra 659b that pulls / weights the pre-operative image data 642 out of the user's field of view and toward the second vertebra 659b. Alternatively or additionally, if the user manually moves the pre-operative image data 642 close enough for local registration to be successful, the system 100 can simply "snap" the pre-operative image data 642 into alignment with the second vertebra 659b while the user is still guiding the pre-operative image data 642 into position.

[0061] In some embodiments, after registering the portion of the pre-operative image data displayed to the user (e.g., the segmented portion of the CT scan), the remaining portion of the pre-operative image data (e.g., the unsegmented or remaining portion of the CT scan) can be registered to the patient. For example, with reference to FIGS. 6A-6E, the system 100 can utilize the registration of the second vertebra 659b as an initialization for registering (e.g., locally registering) the pre-operative image data of one or more of the first vertebra 659a, the third vertebra 659c, the fourth vertebra 659d, and / or other anatomical features.

[0062] 7 is a flow diagram of a process or method 760 for registering pre-operative image data to intra-operative image data in accordance with additional embodiments of the present technique. In some embodiments, method 760 may be used to globally and / or locally register pre-operative image data to intra-operative image data in blocks 333 and 334 of method 330 described in detail with reference to FIGS. 3-4C. Although certain features of method 760 are described in the context of system 100 shown in FIGS. 1 and 2 for illustrative purposes, one of ordinary skill in the art will readily appreciate that method 760 may be implemented using other suitable systems and / or devices described herein.

[0063] At block 761, the method 760 includes receiving pre-operative image data. As described in detail above, the pre-operative image data may include medical scan data representing a three-dimensional volume of a patient, such as computed tomography CT scan data. At block 762, the method 760 includes receiving intra-operative image data of the surgical scene 108, for example, from the camera array 110. As described in detail above, the intra-operative image data may include real-time or near real-time images from the camera 112 and / or the depth camera 118 of the depth sensor 114, such as images of the spine of a patient undergoing spinal surgery. In some embodiments, the intra-operative image data may include light field data from the camera 112.

[0064] At block 763, the method 760 includes analyzing the intraoperative image data to determine one or more characteristics / metrics corresponding to different types of anatomical structures of the patient in the surgical scene. For example, the registration processing device 105 can analyze light field data (e.g., hyperspectral light field data) from the cameras 112, such as color (e.g., hue, saturation, and / or value), angle information, and / or specularity information, to classify different parts of the patient's anatomical structures as tissue, bone, ligament, tendon, nerve, etc. FIG. 8 is an image of a patient's spine 868 captured by one or more cameras 112, for example, according to an embodiment of the present technology. The spine 868 is formed of bone (e.g., a first type of anatomical feature) and is interspersed with and surrounded by other anatomical features, such as flesh 869 (e.g., a second type of anatomical feature). In the illustrated embodiment, the intraoperative image data of the spine 868 is less saturated and brighter than the flesh 869. In some embodiments, one or more of the types of anatomical structures can correspond to the preoperative image data. That is, the pre-operative image data can be of one or more of the anatomical structure types of the intra-operative image data. For example, image data of the spine 868 can correspond to pre-operative image data, including a CT scan or other medical scan of the spine 868.

[0065] At block 764, the method 760 includes registering the pre-operative image data to the intra-operative image data based at least in part on one or more characteristics corresponding to different types of anatomical structures. For example, some registration algorithms (e.g., an iterative closest point (ICP) algorithm) optionally include weights that can be applied point-by-point to each correspondence used to calculate the registration transformation, such as each correspondence between (i) a point cloud or mesh generated from the depth sensor 114 and (ii) a point cloud or mesh representing the pre-operative image data. That is, the registration algorithm can apply individual weights to a correspondence between a first point in the intra-operative image data and a second point in the pre-operative image data. In some embodiments, the weights of the registration algorithm can be adjusted based on determined characteristics in the intra-operative image data that correspond to the patient's anatomical structures (block 763). For example, in the case of a spine procedure, it is often desirable to register CT data of the spine to intra-operative image data of the patient's exposed spine during the procedure. Thus, with reference to FIG. 8, if a particular point (e.g., a point in a point cloud from depth sensor 114) maps to a pixel captured by camera 112 that has characteristics indicating a high likelihood of being part of spine 868, such as relatively low saturation, high brightness, and / or the like, the weight of the correspondence of that point in the registration algorithm can be increased. Conversely, if the image data from camera 112 indicates that a point is likely to be part of flesh 869 or other anatomical structure, the weight for that point can be decreased. In some embodiments, the weight assigned to the correspondence between points can be a learned and / or tuned function of light field characteristics for the points, such as a combination of hue, saturation, color, angle, and / or specularity information. In contrast, typical approaches determine the weights of the registration algorithm from a scene-agnostic metric that is derived solely from the structure (e.g., local structure) of the point cloud or mesh used for registration.

[0066] In some aspects of the present technology, using light field image data from camera 112 to create weights for the registration transformation still allows flesh, blood, and / or other anatomical features close to the surface of spine 868 to be included in the registration and provide a positive input to the registration. In some embodiments, the weight of a particular point may be binary (e.g., fully weighted or not included) based on the light field characteristics of that point. For example, a point indicated to be along spine 868 may be weighted with a "1" and a point indicated to be along flesh 869 may be weighted with a "0". Thus, in some embodiments, method 760 operates to segment a portion of the intraoperative image data (e.g., a portion of a bone) for registration, thereby increasing the accuracy of the registration.

[0067] 9 is a flow diagram of a process or method 970 for registering pre-operative image data to intra-operative image data to generate a mediated reality view of a surgical scene, in accordance with an additional embodiment of the present technique. Although certain features of the method 970 are described in the context of the system 100 shown in Figures 1 and 2 for illustrative purposes, one of ordinary skill in the art will readily appreciate that the method 970 can be implemented using other suitable systems and / or devices described herein.

[0068] At compound block 971, the method 970 includes receiving intraoperative image data of the scene 108 and processing the intraoperative image data to generate depth information. More specifically, at block 972, the method includes capturing an image of the scene 108 with the depth camera 118 of the depth sensor 114. In some embodiments, the image is a stereo image of the scene 108 including depth information, for example, from a pattern projected / projected into the scene by the projector 116. In some embodiments, the depth sensor 114 has the same or approximately the same resolution as the preoperative image data.

[0069] At block 973, the method 970 includes processing the image to generate a point cloud depth map. For example, the processing device 102 (e.g., the image processing device 103 and / or the registration processing device 105) may process the image data from the depth sensor 114 to estimate the depth of each surface point of the scene 108 relative to a common origin and generate a point cloud representing the surface geometry of the scene 108. In some embodiments, the processing device 102 may utilize semi-global matching (SGM), semi-global block matching (SGBM), and / or other computer vision or stereo vision algorithms to process the image data to generate the point cloud. In some embodiments, the point cloud may be a depth map of 1 point per 0.11 square millimeter (9 pt / mm 2 ) to 1 point per 9 square millimeters (0.11pt / mm 2 ) range density.

[0070] At block 974, the method 970 may optionally include filtering the point cloud depth map (e.g., using a median or weighting analysis), for example, to remove outliers. At block 975, the method includes generating a 3D mesh from the point cloud depth map. In some embodiments, the processing device 102 may generate the 3D mesh using marching cubes or other suitable algorithms. In some embodiments, generating the 3D mesh may require about 25% or more of the total time to execute the combined blocks 971.

[0071] At block 976, the method 970 includes globally and / or locally registering the point cloud to the pre-operative image data. In some embodiments, the global and / or local registration may utilize any of the registration methods / techniques described in detail above with reference to FIGS. 3-8. In some embodiments, utilizing a low density / resolution point cloud instead of a high density 3D mesh is sufficient to achieve accurate registration. Thus, in the illustrated embodiment, the global and / or local registration proceeds from block 974 and utilizes a filtered point cloud for registration to the pre-operative image data. In some aspects of the present technology, utilizing a point cloud rather than a 3D mesh requires less data analysis and results in a faster registration process. For example, a 9pt / mm 2 instead of a 3D mesh with a density of 0.11pt / mm 2 By utilizing a point cloud with a density of 100x, data usage can be reduced by 81x.

[0072] At block 977, method 970 includes processing the 3D mesh and image data from cameras 112 of camera array 110 to generate / synthesize a virtual view of scene 108, as described in detail above with reference to FIG. 1. In some aspects of the present technology, because the registration process (block 976) utilizes a point cloud rather than a 3D mesh, the registration process may be initialized and begin running before and during the virtual synthesis of the view of the scene (block 977). That is, these processes may be performed in parallel, improving the processing speed of method 970.

[0073] At block 978, the method 970 includes displaying (e.g., on the display device 104) the virtual view and the registered pre-operative image data together to provide a user with a mediated reality view of the scene 108. In some embodiments, blocks 976-978 of the method 970 may operate generally similarly or identically to blocks 332-337 of the method 330, e.g., described in detail with reference to FIGS.

[0074] FIG. 10 is a flow diagram of a process or method 1080 for registering a point cloud depth map of a scene to pre-operative image data of a portion of the scene, according to an embodiment of the present technology. In some embodiments, the method 1080 can be used to locally and / or globally register a point cloud to pre-operative image data in block 976 of the method 970 described in detail with reference to FIG. 9. FIG. 11 is a graph of the accuracy of a registration algorithm over time for processing different numbers / densities of points in the point cloud, according to an embodiment of the present technology. In the illustrated embodiment, a first curve 1185 represents processing a first number of points (e.g., 10% of all points in the point cloud), a second curve 1186 represents processing a second number of points greater than the first number of points (e.g., 50% of all points in the point cloud), and a third curve 1187 represents processing a third number of points greater than the first number of points (e.g., 100% of all points in the point cloud).

[0075] 10 and 11 together, at block 1081, the method 1080 includes beginning registration of the point cloud to the pre-operative image data based on a selected number of points in the point cloud. For example, registration may begin by executing a selected registration algorithm (e.g., an ICP algorithm) based on a first number of points represented by a first curve 1185. As shown in FIG. 11, processing of the first number of points may reach a first selected accuracy A1 at an earlier time T1 than processing of a second or third number of points represented by second and third curves 1186 and 1187, respectively, reaches accuracy A1. That is, processing a smaller number of points may achieve a first accuracy level more quickly than processing a larger number of points.

[0076] However, the first curve 1185 flattens out quickly at a relatively low accuracy. Thus, in block 1082, the method 1080 may include continuing the registration of the point cloud to the preoperative image data based on a larger number of points in the point cloud after a predefined registration accuracy level (and / or a predefined processing time) is reached. For example, after the initial processing of the first number of points represented by the first curve 1185 reaches a first selected accuracy A1, the registration may continue by executing a registration algorithm selected based on a second number of points represented by the second curve 1186. Thus, the processing of the second number of points may be substantially started (e.g., initialized) at the first selected accuracy level A1 at time T1. Thus, by initially processing a smaller first number of points before switching to processing a larger second number of points at accuracy level A1, the processing time of the registration algorithm may be reduced by the difference between times T1 and T2 (i.e., time T2-T1), improving the overall processing speed.

[0077] At decision block 1083, the method 1080 includes determining whether sufficient registration accuracy has been reached. If yes, the method 1080 may proceed to end at block 1084 with the registration completed. If no, the method 1080 may return to block 1082 and continue the registration of the point cloud with the pre-operative image data based on more points in the point cloud after another predefined registration accuracy level (and / or predefined processing time) is reached. For example, after the processing of the second number of points represented by the second curve 1186 reaches a second selected accuracy A2 at time T3, the registration may continue by executing a registration algorithm selected based on a third number of points represented by the third curve 1187. Thus, the processing of the third number of points is initialized at time T3 at the second selected accuracy level A2 (an accuracy level not reached by the processing of the third number of points alone until time T4). Thus, by first processing a smaller second number of points before switching to processing a larger third number of points at accuracy level A2, the processing time of the registration algorithm may be reduced by the difference between times T3 and T4 (i.e., time T4-T2), improving overall processing speed. Method 1080 may return to block 1082 any number of times to differentially process different numbers of points in the point cloud, for example in stages.

[0078] 12 is a flow diagram of a process or method 1290 for determining the accuracy of registration, global and / or local registration, etc., between intraoperative image data and preoperative image data, in accordance with an embodiment of the present technology. Although certain features of the method 1290 are described in the context of the system 100 shown in Figures 1 and 2 for illustrative purposes, one of ordinary skill in the art will readily appreciate that the method 1290 can be implemented using other suitable systems and / or devices described herein.

[0079] At block 1291, the method 1290 includes recording and / or receiving historical registration data. The historical registration data may include, for example, (i) pre-operative image data (e.g., a 3D data set such as CT scan data), (ii) intra-operative image data (e.g., a 3D point cloud or mesh derived from the depth sensor 114), and (iii) an exemplary data set including a registration transform for mapping the pre-operative image data to the intra-operative image data. In some embodiments, the exemplary data set may be recorded / compiled from a previous surgical procedure and / or may be generated as a test case. In some embodiments, the registration transformer may be calculated using any of the methods described in detail above with reference to FIGS. 3-11.

[0080] At block 1292, the method 1290 includes defining a spatial neighborhood around the historical registration transformation. The spatial neighborhood can include slight variations / deviations in the values ​​of the historical registration transformation, such as small translational, rotational, and / or reflection variations. In some embodiments, the spatial neighborhood can be a feature vector (e.g., a 729x1 feature vector) generated by transforming the past pre-operative image data (e.g., source data) and / or the past intra-operative image data to nearby poses in a special Euclidean group space (SE(n)). In some embodiments, the nearby poses can be within a threshold rotational and translatable variance, such as within about ±5 degrees in the rotational direction and within about ±0.3 millimeters in the translational direction.

[0081] At block 1293, the method 1290 includes classifying / labeling the historical registration transformations. For example, each of the historical registration transformations and corresponding spatial neighborhoods can be classified with a binary label as a "good" or "accurate" transformation or a "bad" or "inaccurate" transformation based on a predefined criterion. In some embodiments, the predefined criterion can select an acceptable deviation from the true registration (e.g., 100% accurate registration). For example, a "good" transformation can be defined to be within a selected rotational variance (e.g., ±1 degree) and translational variance (e.g., ±0.5 millimeters) from the true registration. In some embodiments, translational and rotational random noise can be introduced into some or all of the "good" historical registration transformations to generate "bad" transformations.

[0082] At block 1294, the method 1290 includes training a machine learning model based on (i) spatial neighborhoods around the historical registration transformations and (ii) classifications for these transformations. More specifically, for each example of historical registration data, a machine learning algorithm can be trained with a feature vector and associated binary label that represent the neighborhood around the historical registration transformation. In some embodiments, the machine learning algorithm can be a singular value decomposition (SVD) or a neural network. In other embodiments, other machine learning techniques can be employed. Such machine learning techniques include support vector machines, Bayesian networks, learning regression, and / or neural networks when generating weights. The support vector machine can be trained using examples of good and bad registration transformations as training data. The support vector machine operates by finding a hypersurface in the space of possible inputs. The hypersurface attempts to split the positive examples (i.e., good registration transformations) from the negative examples (i.e., bad registration transformations) by maximizing the distance between the closest positive and negative examples and the hypersurface. A support vector machine simultaneously minimizes the empirical classification error and maximizes the geometric margin, which allows it to correctly classify data that is similar but not identical to the training data. A variety of techniques can be used to train a support vector machine. Some techniques use a sequential minimal optimization algorithm that decomposes a large quadratic programming problem into a series of smaller quadratic programming problems that can be solved analytically.

[0083] At block 1295, the method 1290 includes receiving intraoperative registration data including an intraoperative registration transformation. Similar to the historical registration data, the intraoperative registration data may include, for example, (i) preoperative image data (e.g., a 3D data set such as CT scan data), (ii) intraoperative image data (e.g., a 3D point cloud or mesh derived from the depth sensor 114), and (iii) a data set including a registration transformation for mapping the preoperative image data to the intraoperative image data. Such intraoperative registration data may be obtained using any of the techniques described in detail above with reference to FIGS. 3-11.

[0084] At block 1296, the method 1290 includes defining a spatial neighborhood around the intra-operative registration transformer. Similar to the neighborhood around the historical registration transform, the spatial neighborhood around the intra-operative registration transform may be a feature vector that defines a set of nearby poses or transformations around the determined intra-operative registration transform.

[0085] At block 1297, the method 1290 includes inputting a spatial neighborhood around the intraoperative registration transformation into the trained machine learning model. Based on the input, at block 1298, the method 1290 includes determining a fitness score for the accuracy of the intraoperative registration transformation. The fitness score can be a binary "good" or "bad" decision, or can be a score along a continuous spectrum or two or more discrete spectra. In some embodiments, if the fitness score falls below a predetermined threshold, the system 100 can attempt to re-register the preoperative image data to the intraoperative image data. In some aspects of the present technology, evaluating a neighborhood of values ​​around a given registration transformation, rather than the transformation alone, increases the reliability of the assessment of registration accuracy.

[0086] The methods 330, 550, 760, 970, 1080, and 1290 described in detail above with reference to Figures 3-12 may include certain features that are generally similar to one another and / or may operate in generally similar ways. For example, various steps of the methods may be combined with one another, omitted, and / or performed in a different order. Additionally, although reference has been made throughout to pre-operative and intra-operative image data, the sources of these data may be of other types without departing from the scope of the teachings of the present invention. For example, the pre-operative image data and / or intra-operative image data may include depth data from sources other than a camera imaging the scene.

[0087] III. Additional Examples The following examples illustrate some embodiments of the present techniques.

[0088] 1. A method of registering medical scan data of a patient to intraoperative image data of a scene including said patient, comprising: generating a three-dimensional (3D) image of the scene based on the intraoperative image data, the medical scan data corresponding to a portion of the patient that is at least partially visible in the 3D image; overlaying said medical scan data onto said 3D image; and receiving user input to move the medical scan data into alignment with a portion of the patient in the 3D image; registering the medical scan data to the portion of the patient in the 3D image based on the alignment; A method comprising:

[0089] 2. The method of Example 1, wherein the method further includes continuously receiving the intraoperative image data, and generating the 3D image includes continuously updating a virtual image based on the intraoperative image data.

[0090] 3. The method of any one of Examples 1 to 2, wherein the method further comprises displaying the 3D image and the medical scan data on the display device in substantially real time as perceived by a user of the display device.

[0091] 4. The method of any one of Examples 1-3, wherein the medical scan data is a segmented portion of a computed tomography (CT) scan.

[0092] 5. The method of any one of Examples 1-4, wherein the medical scan data is of the patient's vertebrae and the portion of the patient in the 3D image is the patient's spine.

[0093] 6. Registering the medical scan data includes globally registering the medical scan data to the portion of the patient in the 3D image; The method of any one of Examples 1-5, further comprising locally registering the medical scan data to a portion of the patient within the 3D image based at least in part on the global registration.

[0094] 7. The method of Example 6, wherein the method further includes, after locally registering the medical scan data to the portion of the patient in the 3D image, automatically moving the medical scan data to further align with the portion of the patient in the 3D image.

[0095] 8. The method of any one of Examples 1-7, wherein the user input is from a tool movable across the scene.

[0096] 9. The method of Example 8, wherein overlaying the medical scan data on the 3D image includes displaying the medical scan data at the tip of the tool within the scene.

[0097] 10. The method of any one of Examples 1-9, wherein the user input is dragging the medical scan data toward a portion of the patient within the 3D image.

[0098] 11. The method of any one of Examples 1-10, wherein the user input is to rotate the medical scan data toward a portion of a patient within the 3D image.

[0099] 12. A mediated reality system comprising: a camera array including a plurality of cameras configured to capture intraoperative image data of a scene including a patient; an input controller configured to control a position and orientation of a virtual view of the scene; a processing device communicatively coupled to the camera array and the input controller; Equipped with The processing device includes: synthesizing a virtual image corresponding to the virtual field of view based on the intraoperative image data; receiving medical scan data of the patient corresponding to a portion of the patient that is at least partially visible in the virtual image; overlaying said medical scan data onto said virtual image; receiving user input to move the medical scan data into alignment with a portion of the patient in the virtual image; registering the medical scan data to the portion of the patient in the virtual image based on the alignment; A mediated reality system configured as follows:

[0100] 13. The mediated reality system of Example 12, further comprising a display device communicatively coupled to the processing device, the display device configured to display a three-dimensional (3D) graphical representation of the medical scan data over the virtual image.

[0101] 14. The mediated reality system of Example 12 or 13, further comprising a tool communicatively coupled to the processing device, wherein the user input is based on a position of the tool relative to the scene.

[0102] 15. The mediated reality system of Example 14, wherein the user input is a physical translation, a physical rotation, or both a physical translation and a physical rotation of the tool relative to the scene.

[0103] 16. The mediated reality system of any one of Examples 12-15, wherein the scene is a surgical scene, the portion of the patient in the virtual image includes the patient's spine, and the medical scan data is computed tomography (CT) scan data.

[0104] 17. A method of registering previously captured image data to real-time image data of a scene, comprising: generating a three-dimensional (3D) virtual view of the scene based on the real-time image data, the scene including an object of interest and the previously captured image data corresponding to the object of interest; displaying said 3D virtual view on a display device visible to a user; displaying the previously captured image data on the display device over the 3D virtual view; and receiving a user input to move the previously captured image data relative to the 3D virtual view such that the previously captured image data is at least partially aligned with the object of interest in the 3D virtual view; generating a registration transformation between the previously captured image data and the object of interest in the 3D virtual view based on an alignment of the previously captured image data and the object of interest in the 3D virtual view; A method comprising:

[0105] 18. The method of example 17, wherein displaying the 3D virtual view on the display device includes displaying the 3D virtual view substantially in real time as perceived by the user.

[0106] 19. The method according to claim 1, locally registering the previously captured image data to the object of interest within the 3D virtual view based at least in part on the registration transformation; automatically moving the previously captured image data to be further aligned with the object of interest in the 3D virtual view based on the local registration; The method of Example 17 or 18, further comprising:

[0107] 20. The method of any one of Examples 17-19, wherein the user input is based on a position of the tool relative to the scene, and displaying the previously captured image data on the 3D virtual view includes displaying a 3D representation of the previously captured image data in the 3D virtual view at a position corresponding to the tip of the tool in the scene, and the user input is a physical movement of the tool across the scene.

[0108] 21. A method of registering medical scan data of a patient to intraoperative image data of a scene including said patient, comprising: determining one or more characteristics of the intraoperative image data; and determining, based on the one or more determined characteristics, (a) a first portion of the intraoperative image data corresponds to a first type of anatomical structure of the patient, and (b) a second portion of the intraoperative image data corresponds to a second type of anatomical structure of the patient, the first type of anatomical structure corresponding to the medical scan data; registering the preoperative image data to the first portion of the intraoperative image data; A method comprising:

[0109] 22. The method of example 21, wherein the preoperative image data is computed tomography (CT) scan data.

[0110] 23. The method of example 21 or 22, wherein the first type of anatomical structure is a bone.

[0111] 24. Registering the preoperative image data to the first portion of the intraoperative image data, utilizing a registration algorithm to calculate a registration transformation between the pre-operative image data and the first portion of the intra-operative image data; adjusting the registration algorithm based on the determined one or more characteristics; and The method according to any one of Examples 21 to 23, comprising:

[0112] 25. The method of Example 24, wherein utilizing the registration algorithm includes calculating a plurality of point-to-point correspondences between a first point in the intraoperative image data and a second point in the preoperative image data, and adjusting the registration algorithm includes adjusting a weight of the point-to-point correspondence based on a determination that the first point in the point-to-point correspondence corresponds to a first type of the anatomical structure or a second type of the anatomical structure.

[0113] 26. The method of Example 25, wherein adjusting the weights of the point-to-point correspondences includes (a) increasing the weights of the point-to-point correspondences involving the first points corresponding to the first type of anatomical structure, and (b) decreasing the weights of the point-to-point correspondences involving the first points corresponding to the second type of anatomical structure.

[0114] 27. The method of any one of Examples 21 to 26, wherein the one or more characteristics include at least one of color information, angle information, and specular information.

[0115] 28. The method of any one of Examples 21-27, wherein the one or more characteristics include at least one of hue, saturation, and value information.

[0116] 29. The method of any one of examples 21 to 28, wherein the intraoperative image data includes light field image data of the scene.

[0117] 30. The method of Example 29, wherein the intraoperative image data further includes image data from a depth camera including depth data of the scene, determining one or more characteristics of the image data includes determining the one or more characteristics based on the light field image data, and registering the preoperative image data to a first portion of the intraoperative image data includes registering the depth data to the preoperative image data.

[0118] 31. The method according to claim 1, generating a three-dimensional (3D) image of the scene based on the intraoperative image data; and displaying the medical scan data over the first type of anatomical structure in the 3D image of the scene; The method of any one of Examples 21 to 30, further comprising:

[0119] 32. A mediated reality system comprising: a camera array including a plurality of cameras configured to capture intraoperative image data of a scene including a patient; an input controller configured to control a position and orientation of a virtual view of the scene; a processing device communicatively coupled to the camera array and the input controller; Equipped with The processing device includes: synthesizing a virtual image corresponding to the virtual field of view based on the intraoperative image data; receiving medical scan data of said patient; determining one or more characteristics of the intraoperative image data; determining, based on the one or more determined characteristics, (a) a first portion of the intraoperative image data corresponds to a first type of anatomical structure of the patient; and (b) a second portion of the intraoperative image data corresponds to a second type of anatomical structure of the patient, the first type of anatomical structure corresponding to the medical scan data; registering the preoperative image data to a first portion of the intraoperative image data; overlaying said medical scan data over said first type of anatomical structure in said virtual image; A mediated reality system configured as follows:

[0120] 33. The mediated reality system of Example 32, wherein the scene is a surgical scene, the first type of anatomical structure is a patient's spine, and the medical scan data is computed tomography (CT) scan data.

[0121] 34. The mediated reality system of Example 32 or 33, wherein the one or more characteristics include at least one of hue, saturation, and value information, and the intraoperative image data includes light field image data of the scene.

[0122] 35. The mediated reality system of any one of Examples 32-34, wherein the processor is configured to register the preoperative image data to a first portion of the intraoperative image data by utilizing a registration algorithm to calculate a plurality of point-to-point correspondences between a first point of the intraoperative image data and a second point of the preoperative image data, and adjusting weights of the point-to-point correspondences based on a determination that the first point in the point-to-point correspondences corresponds to a first type of the anatomical structure or a second type of the anatomical structure.

[0123] 36. The mediated reality system of Examples 32-35, wherein the processor is further configured to adjust weights of the point-to-point correspondences by (a) increasing a weight of the point-to-point correspondence involving the first point corresponding to the first type of the anatomical structure, and (b) decreasing a weight of the point-to-point correspondence involving the first point corresponding to the second type of the anatomical structure.

[0124] 37. A method of registering previously captured image data to real-time image data of a scene, comprising: receiving real-time image data including light field image data of the scene; generating a three-dimensional (3D) virtual view of the scene based on the real-time image data, the scene including an object of interest, and the previously captured image data corresponding to the object of interest; determining one or more characteristics of the light field image data; determining, based on the one or more determined characteristics, (a) that a first portion of the real-time image data is likely to correspond to the object of interest, and (b) that a second portion of the real-time image data is likely to not correspond to the object of interest; registering the previously captured image data to a first portion of the real-time image data; displaying the previously captured image data over the object of interest in a 3D virtual view of the scene; A method comprising:

[0125] 38. The method of example 37, wherein the one or more characteristics include at least one of color information, angle information, and specular information.

[0126] 39. Registering the previously captured image data to a first portion of the real-time image data includes: utilizing a registration algorithm to calculate a plurality of point-to-point correspondences between first points in the real-time image data and second points in the previously captured image data; adjusting a weight of the point correspondence based on a determination that the first point in the point correspondence is likely to correspond to the object of interest; The method of Example 37 or 38, comprising:

[0127] 40. A method according to any one of examples 37 to 39, wherein determining that a first portion of the real-time image data is likely to correspond to the object of interest includes determining that light field image data corresponding to the first portion of the real-time image data has a lower saturation than other portions of the light field image data.

[0128] IV. conclusion The above detailed description of the embodiments of the technology of the present invention is not intended to be exhaustive or to limit the present invention to the above disclosed form. Although specific embodiments and examples of the technology of the present invention are described above for illustrative purposes, those skilled in the relevant art will recognize that various equivalent modifications are possible within the scope of the technology of the present invention. For example, while steps are shown in a given order, alternative embodiments may perform steps in a different order. Also, various embodiments described herein may be combined to provide further embodiments.

[0129] From the foregoing, it will be understood that, although specific embodiments of the inventive technology are described herein for purposes of illustration, well-known structures and functions have not been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments of the inventive technology. Where the context permits, singular or plural terms may include the plural or singular terms, respectively.

[0130] Furthermore, unless the word "or" is expressly limited in reference to a list of two or more items to mean only a single item exclusively from the other items, the use of "or" in such a list is to be interpreted as including (a) any single item in the list, (b) all items in the list, or (c) any combination of items in the list. Furthermore, the term "comprising" is used throughout to mean including at least the recited feature(s), so as not to exclude any number of the same features and / or additional varieties of other features. Also, while certain embodiments have been described herein for illustrative purposes, it will be understood that various changes may be made without departing from the technology of the present invention. Furthermore, although advantages associated with some embodiments of the technology of the present invention have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily exhibit such advantages to fall within the scope of the technology of the present invention. Thus, the present disclosure and related technology can encompass other embodiments not expressly shown or described herein.

Claims

1. A method of registering medical scan data of a patient to intraoperative image data of a scene including said patient, comprising: determining one or more characteristics of the intraoperative image data; determining, based on the one or more determined characteristics, (a) a first portion of the intraoperative image data corresponds to a first type of anatomical structure of the patient, and (b) a second portion of the intraoperative image data corresponds to a second type of anatomical structure of the patient, the first type of anatomical structure corresponding to the medical scan data; registering the pre-operative image data to the first portion of the intra-operative image data; Including, registering the pre-operative image data to the first portion of the intra-operative image data; utilizing a registration algorithm to compute a registration transformation between the pre-operative image data and the first portion of the intra-operative image data, the utilizing the registration algorithm comprising computing a plurality of point-to-point correspondences between first points in the intra-operative image data and second points in the pre-operative image data; adjusting the registration algorithm based on the determined one or more characteristics, where adjusting the registration algorithm comprises adjusting a weight of the point-to-point correspondence based on a determination that the first point in the point-to-point correspondence corresponds to a first type of the anatomical structure or a second type of the anatomical structure; A method comprising:

2. The preoperative image data is computed tomography (CT) scan data. The method of claim 1.

3. The first type of anatomical structure is a bone. The method of claim 1.

4. Adjusting the weights of the point-to-point correspondences includes: (a) increasing a weight of the point-to-point correspondence including the first point corresponding to a first type of the anatomical structure; and (b) decreasing a weight of the point-to-point correspondence including the first point corresponding to a second type of the anatomical structure. The method of claim 1.

5. The one or more characteristics include at least one of color information, angle information, and specular information. The method of claim 1.

6. The one or more characteristics include at least one of hue, saturation, and value information. The method of claim 1.

7. The intraoperative image data comprising light field image data of the scene. The method of claim 1.

8. The intraoperative image data further includes image data from a depth camera including depth data of the scene, determining one or more characteristics of the image data includes determining the one or more characteristics based on the light field image data, and registering the preoperative image data to a first portion of the intraoperative image data includes registering the depth data to the preoperative image data. The method according to claim 7.

9. The method according to claim 1, generating a three-dimensional (3D) image of the scene based on the intraoperative image data; and displaying the medical scan data over the first type of anatomical structure in the 3D image of the scene; The method of claim 1 further comprising:

10. A mediated reality system comprising: a camera array including a plurality of cameras configured to capture intraoperative image data of a scene including a patient; an input controller configured to control a position and orientation of a virtual view of the scene; a processing device communicatively coupled to the camera array and the input controller; Equipped with The processing device comprises: synthesizing a virtual image corresponding to the virtual field of view based on the intraoperative image data; receiving medical scan data of the patient; determining one or more characteristics of the intraoperative image data; determining, based on the one or more determined characteristics, (a) a first portion of the intraoperative image data corresponds to a first type of anatomical structure of the patient; and (b) a second portion of the intraoperative image data corresponds to a second type of anatomical structure of the patient, the first type of anatomical structure corresponding to the medical scan data; registering the preoperative image data to a first portion of the intraoperative image data by utilizing a registration algorithm to calculate a plurality of point-to-point correspondences between first points of the intraoperative image data and second points of the preoperative image data, and adjusting weights of the point-to-point correspondences based on a determination that the first points in the point-to-point correspondences correspond to the first type of anatomical structure or the second type of anatomical structure; overlaying the medical scan data over the first type of anatomical structure in the virtual image; A mediated reality system configured as follows:

11. The method of claim 1, wherein the scene is a surgical scene, the first type of anatomical structure is a spine of the patient, and the medical scan data is computed tomography (CT) scan data. The mediated reality system of claim 10.

12. The method of claim 11, wherein the one or more characteristics include at least one of hue, saturation, and value information, and the intraoperative image data includes light field image data of the scene. The mediated reality system of claim 10.

13. The processor is further configured to adjust a weight of the point-to-point correspondence by: (a) increasing a weight of the point-to-point correspondence including the first point corresponding to a first type of the anatomical structure; and (b) decreasing a weight of the point-to-point correspondence including the first point corresponding to a second type of the anatomical structure. The mediated reality system of claim 10.

14. A method of registering previously captured image data to real-time image data of a scene, comprising: receiving real-time image data including light field image data of the scene; generating a three-dimensional (3D) virtual view of the scene based on the real-time image data, the scene including an object of interest, and the previously captured image data corresponding to the object of interest; determining one or more characteristics of the light field image data; determining, based on the one or more determined characteristics, (a) that a first portion of the real-time image data is likely to correspond to the object of interest, and (b) that a second portion of the real-time image data is likely to not correspond to the object of interest; registering the previously captured image data to a first portion of the real-time image data, where registering the previously captured image data to the first portion of the real-time image data includes utilizing a registration algorithm to calculate a plurality of point-to-point correspondences between a first point in the real-time image data and a second point in the previously captured image data, and adjusting weights of the point-to-point correspondences based on a determination that the first point in the point-to-point correspondences is likely to correspond to the object of interest; displaying the previously captured image data over the object of interest in a 3D virtual view of the scene; A method comprising:

15. The one or more characteristics include at least one of color information, angle information, and specular information. The method of claim 14.

16. Determining that a first portion of the real-time image data is likely to correspond to the object of interest includes determining that light field image data corresponding to the first portion of the real-time image data has a lower saturation than other portions of the light field image data. The method of claim 14.

17. A mediated reality system comprising: a camera array including a plurality of cameras configured to capture intraoperative image data of a scene including a patient; a processing device communicatively coupled to the camera array and the input controller; Equipped with The processing device comprises: receiving medical scan data of the patient; determining one or more characteristics of the intraoperative image data; determining, based on the one or more determined characteristics, (a) a first portion of the intraoperative image data corresponds to a first type of anatomical structure of the patient; and (b) a second portion of the intraoperative image data corresponds to a second type of anatomical structure of the patient, the first type of anatomical structure corresponding to the medical scan data; registering the preoperative image data to a first portion of the intraoperative image data by utilizing a registration algorithm to calculate a plurality of point-to-point correspondences between first points of the intraoperative image data and second points of the preoperative image data, and adjusting weights of the point-to-point correspondences based on a determination that the first points in the point-to-point correspondences correspond to the first type of anatomical structure or the second type of anatomical structure; overlaying the medical scan data over the first type of anatomical structure in the virtual image; A mediated reality system configured as follows:

18. The method of claim 17, wherein the scene is a surgical scene, the first type of anatomical structure is a spine of the patient, and the medical scan data is computed tomography (CT) scan data.

20. The mediated reality system of claim 17.

19. The method of claim 18, wherein the one or more characteristics include at least one of hue, saturation, and value information, and the intraoperative image data includes light field image data of the scene.

20. The mediated reality system of claim 17.

20. The processor is further configured to adjust a weight of the point-to-point correspondence by: (a) increasing a weight of the point-to-point correspondence including the first point corresponding to a first type of the anatomical structure; and (b) decreasing a weight of the point-to-point correspondence including the first point corresponding to a second type of the anatomical structure.

20. The mediated reality system of claim 17.

Images