Surgical data fusion system, surgical imaging system, and corresponding methods and computer programs

Abstract

An example relates to a surgical data fusion system, a surgical imaging system comprising a surgical data fusion system, a surgical data fusion method and a corresponding computer program. The surgical data fusion system is configured to: obtain three-dimensional anatomical information of a patient, the three-dimensional anatomical information being based on a pre-operative scan or an intra-operative scan of the patient; obtain one or more intra-operative measurements of the patient; and determine position information of a visual representation (40) of the one or more intra-operative measurements relative to the three-dimensional anatomical information, wherein the position information is indicative of a position of the visual representation (40) of the one or more intra-operative measurements relative to at least one planar visual representation (30) of at least one slice of the three-dimensional anatomical information.

Description

Technical Field

[0001] The examples involve a surgical data fusion system, a surgical imaging system including the surgical data fusion system, a surgical data fusion method, and a corresponding computer program. Background Technology

[0002] Surgeons, particularly neurosurgeons and radiologists, are trained to interpret and understand three-dimensional (3D) anatomical data through radiographic cross-sectional images in the axial, coronal, and sagittal planes. These views allow for a precise spatial understanding of anatomical and pathological structures.

[0003] Current image-guided surgery (IGS) systems in surgical microscopes primarily overlay preoperative 3D data onto real-time optical images from the microscope, presenting a hybrid optical-3D visualization based on optical perspective. While this provides a realistic view of the surgical field of view, it may be less effective in conveying precise 3D spatial relationships between structures relative to the field of view.

[0004] Several methods have been explored to enhance surgical visualization, but each may have limitations. One approach involves optical perspective overlay. In this method, preoperative 3D imaging data is overlaid onto a real-time optical feed from a surgical microscope, aligning it with the microscope's optical perspective. However, optical perspective may not convey depth and spatial relationships as effectively as radiographic cross-sections. Furthermore, surgeons may find interpreting the overlaid images challenging because they differ from familiar radiographic visualization formats. Finally, the lack of precise spatial cues can reduce accuracy when measuring distances and understanding the location of critical structures.

[0005] Another approach utilizes a separate monitor for radiographic imaging. This involves viewing preoperative radiographic images, such as axial, coronal, and sagittal planes, on a separate monitor during surgery. However, this approach creates a disjointed workflow because surgeons must shift their attention between the surgical field and the monitor, interrupting their concentration. This approach also requires surgeons to mentally map the 2D radiographic images to the 3D surgical field, increasing cognitive load. Finally, using this approach can make it difficult to share precise spatial information with the surgical team in real time.

[0006] While some methods have solved the problem of creating 3D surface maps of surgical cavities using stereo imaging, these methods do not focus on integrating radiographic cross-section visualization with real-time optical imaging in a manner that simulates familiar radiographic formats.

[0007] It may be desirable to provide an improved concept for visualizing preoperative 3D data during surgery. Summary of the Invention

[0008] This expectation is addressed through the subject matter of the independent claims.

[0009] The various examples disclosed herein are based on the finding that in many IGS systems, surgeons may lack an intuitive and precise method to visualize and measure 3D quantitative spatial relationships during surgery in a format familiar to radiology. While commonly employed visualizations are based on projecting visualizations of 3D data onto real-time optical images to integrate them with real-time optical views, this approach does not provide a spatial understanding of anatomical and pathological structures as well as views through radiographic cross-sectional images in the axial, coronal, and sagittal planes. In the proposed concept, a joint visualization of 3D preoperative or intraoperative scans with intraoperative measurements is prepared or provided, wherein the visualization of the intraoperative measurements is placed relative to at least one planar visual representation of the 3D preoperative or intraoperative scan, for example, as known from radiographic cross-sectional images in the axial, coronal, and sagittal planes. This allows for improved visualization, thereby enhancing the surgeon's understanding of 3D anatomical data.

[0010] Some aspects of this disclosure relate to a surgical data fusion system. The surgical data fusion system includes one or more processors and one or more storage devices. The surgical data fusion system is configured to acquire three-dimensional anatomical information of a patient. The three-dimensional anatomical information is based on preoperative or intraoperative scans of the patient. The surgical data fusion system is configured to acquire one or more intraoperative measurements of the patient. The surgical data fusion system is configured to determine positional information of a visual representation of one or more intraoperative measurements relative to the three-dimensional anatomical information. The positional information indicates the position of the visual representation of one or more intraoperative measurements relative to at least one planar visual representation of at least one slice of the three-dimensional anatomical information. This allows for improved visualization, thereby improving the surgeon's understanding of the 3D anatomical data.

[0011] The proposed concept is based on simulating the view provided by a radiographic cross-sectional image, which is familiar to surgeons. In other words, at least one planar visual representation can simulate at least one cross-sectional radiographic image. For example, at least one planar visual representation can represent at least one of the axial, coronal, or sagittal planes of three-dimensional anatomical information. This provides the surgeon with a view that he or she is trained to understand.

[0012] Typically, actual surgical visualization can be provided by another system or platform. In this case, the surgical data fusion system can be configured to provide location information to the system or platform providing the surgical visualization. Alternatively, the surgical data fusion system can be a surgical visualization system configured to provide surgical visualization. In other words, the surgical data fusion system (or surgical visualization system) can be configured to generate a surgical visualization of the surgical site. The surgical visualization may include at least one planar visual representation of at least one slice of three-dimensional anatomical information, overlaid with visual representations of one or more intraoperative measurements. The overlay can be placed on the at least one planar visual representation based on location information. The surgical data fusion system can be configured to provide the surgical visualization to a display device. This provides the surgeon with improved visualization, thereby enhancing the surgeon's understanding of the 3D anatomical data.

[0013] Based on one of three predefined axes, each radiographic cross-sectional image represents a slice of a 3D representation of the patient's anatomy. If the corresponding radiographic cross-sectional images of the axial, coronal, or sagittal planes are stacked one above the other or adjacent to each other, these images represent a 3D representation of the anatomy, minus the portions located between the slices. To reach a portion of the anatomy of interest to the surgeon, a mechanism can be provided to the user (surgeon) to select one or more slices, effectively "rolling" the slice stack to reach the slice of interest. In other words, the surgical data fusion system can be configured to receive (e.g., from) user input from the user of the surgical data fusion system and select at least one slice of 3D anatomical information for surgical visualization based on the user input. In this way, at least one slice of interest can be selected for visualization, allowing the surgeon to tailor the visualization to the needs of the current surgery.

[0014] Various types of intraoperative measurements can be used to generate combined surgical visualizations along with slices containing three-dimensional anatomical information. For example, one or more intraoperative measurements of a patient may include image data (e.g., from) a surgical imaging device. The image data may, for example, indicate the surgical site. The visual representation of one or more intraoperative measurements can be based on the image data. In this way, real-time image data from a surgical imaging device (e.g., a surgical microscope) can be overlaid on slices containing three-dimensional anatomical information, providing surgeons with a valuable mechanism for planning the next steps during surgical procedures.

[0015] Typically, the visualization of a real-time view of a surgical site can be improved by ensuring that surgeons can spatially interpret the surgical site. For this purpose, an intermediate representation of the surgical site based on a 3D depth map can be created. For example, a surgical data fusion system can be configured to map image data onto a 3D depth map of the surgical site using image projection. A visual representation of one or more intraoperative measurements can be based on image data mapped onto the 3D depth map of the surgical site. By providing a visual representation based on a 3D depth map of the surgical site, the surgeon's spatial understanding can be improved. Furthermore, image data from multiple perspectives can be combined to provide a more comprehensive view of the surgical site.

[0016] More generally, surgical data fusion systems can be configured to determine a 3D depth map of the surgical site and generate surgical visualizations based on that 3D depth map. By providing surgical visualizations based on 3D depth maps of the surgical site, surgeons' spatial understanding can be improved.

[0017] The fusion of a planar visual representation of at least one slice and a visual representation of one or more intraoperative measurements represented by a 3D depth map may not be direct, as this involves placing a 3D object relative to a plane that overlaps with a 3D object at the surgical site. For example, to perform such a combined 2D-on-3D visualization, the shape of the surgical site at the location of the respective slice (e.g., depth within the stack) can be considered (as represented by a 3D depth map). In other words, a surgical data fusion system can be configured to generate a visual representation of one or more intraoperative measurements based on the shape of the surgical site at the location of at least one slice with 3D anatomical information. Specifically, a portion of the respective slice can be removed to make room for the visual representation of the surgical site, such as that represented by a 3D depth map. For example, the surgical site may appear to extend above and / or below a plane defined by the respective slice, where the slice has an incision representing the current shape of the surgical site, such as that represented by a 3D depth map. In this way, a 3D representation of one or more intraoperative measurements can be integrated with a planar view of the 3D anatomical information.

[0018] In many surgical procedures, such as when removing a tumor or to make room during the procedure, tissue or bone is permanently or temporarily removed from the patient. These removals can be reflected in surgical visualization. For example, a surgical data fusion system can be configured to determine a portion of the surgical site that has been removed compared to a preoperative scan, based on three-dimensional anatomical information and a three-dimensional depth map of the surgical site. The surgical data fusion system can be configured to generate at least one planar visual representation of at least one slice of the three-dimensional anatomical information, excluding that portion of the removed surgical site. In this way, the surgical visualization more accurately represents the current progress of the surgical procedure, providing surgeons with more valuable real-time information and context.

[0019] Discrepancies may arise between three-dimensional anatomical information and the current location of tissues, which are not caused by the surgery. For example, brain displacement occurs during neurosurgical procedures. Brain displacement refers to the movement or deformation of brain tissue that occurs during neurosurgical procedures. This can occur due to several factors, such as changes in cerebrospinal fluid pressure, gravity, and manipulation of brain tissue during surgery. Brain displacement poses a challenge to surgeons because it can alter the position of anatomical landmarks, potentially affecting the accuracy of three-dimensional anatomical information. In the proposed concept, three-dimensional depth mapping can be used to mitigate brain displacement and similar phenomena. In other words, a surgical data fusion system can be configured to determine displacement or deformation of at least a portion of the surgical site based on three-dimensional anatomical information and a three-dimensional depth map. The surgical data fusion system can be configured to adjust at least one planar visual representation of at least one slice of the three-dimensional anatomical information based on the displacement or deformation of at least that portion of the surgical site. In this way, the three-dimensional anatomical information is adjusted to the current state of the patient's anatomy, resulting in a more accurate visualization.

[0020] Various methods exist for determining 3D depth maps. For example, a surgical data fusion system can use sensor data (e.g., from) depth sensors (such as time-of-flight sensors or structured light-based sensors). In other words, a surgical data fusion system can be configured to determine a 3D depth map using at least one of sensor data (e.g., from) a time-of-flight sensor or image data showing a structured light projection (e.g., from a surgical imaging device or another imaging device / from a surgical imaging device or another imaging device). This is one way to determine a 3D depth map.

[0021] Alternatively or additionally, the surgical data fusion system can be configured to determine a three-dimensional depth map of the surgical site using stereo parallax analysis based on image frames contained in (e.g., from) image data from a surgical imaging device showing the surgical site from different perspectives. For example, stereo parallax analysis can be enhanced by projecting structured light onto the surgical site and using information from the structured light projection to determine the three-dimensional depth map. This allows for the creation of a three-dimensional depth map without the need for additional sensors.

[0022] Stereo parallax analysis refers to the process of determining the differences between images taken from two slightly different perspectives (similar to human binocular vision) to calculate depth information about a scene. These perspectives can exist in stereo image data typically generated by surgical microscopes. Therefore, image data can include stereo image data. Surgical data fusion systems can be configured to use stereo parallax analysis based on stereo image frames contained in stereo image data to determine a three-dimensional depth map of the surgical site. In this way, stereo image data generated from existing imaging modalities can be used to determine a three-dimensional depth map in real time, without requiring the surgical imaging equipment to sequentially present different perspectives.

[0023] If stereoscopic image data is unavailable, or to further enhance the accuracy of the 3D depth map, the viewing angle of the surgical imaging device can be altered to capture the surgical site from different perspectives. For example, a surgical data fusion system can be configured to control the surgical imaging system to change the viewing angle of the surgical imaging device to obtain image frames showing the surgical site from different perspectives. This is another method for generating image data suitable for stereo parallax analysis.

[0024] In some instances, surgical data fusion systems can retrieve (e.g., from) 3D anatomical information from an IGS system. During surgery, the IGS system typically tracks the location and configuration of the surgical site to provide appropriate output. This process is called registration. Registration is the process of aligning preoperative images (such as CT, MRI, or ultrasound scans) with the patient's actual anatomy during surgery. Surgical data fusion systems can leverage this functionality to determine location information. For example, a surgical data fusion system can be configured to acquire (e.g., from) 3D anatomical information from an image-guided surgical system. The surgical data fusion system can be configured to determine location information based on the registration functionality of the image-guided surgical system. In this way, the capabilities of existing systems can be reused to perform surgical data fusion.

[0025] During surgery, the primary aspect is the simultaneous visualization of three-dimensional anatomical information and, for example, real-time images from surgical imaging equipment. However, other measurements may also be of interest during surgery. For instance, one or more intraoperative measurements of the patient may include functional imaging data representing the patient's brain activity. Three-dimensional anatomical information, for example, may include three-dimensional anatomical information about the patient's brain. Visualizing the patient's brain activity by referencing three-dimensional anatomical information can provide useful information to the surgeon during surgery.

[0026] Another aspect of this disclosure relates to a surgical imaging system comprising a surgical imaging device and a surgical data fusion system. For example, the surgical data fusion system can be used to provide or assist surgical visualization during surgical procedures. This allows for improved visualization, thereby enhancing the surgeon's understanding of 3D anatomical data.

[0027] The proposed concept is particularly useful in neurosurgery because neurosurgeons use it to interpret radiographic cross-sectional images of the brain. Therefore, surgical imaging systems can be neurosurgical imaging systems. Three-dimensional anatomical information can include three-dimensional anatomical information of the patient's brain. One or more intraoperative measurements can be correlated with the patient's brain.

[0028] Some aspects of this disclosure relate to a surgical data fusion method. The method includes acquiring three-dimensional anatomical information of a patient. The three-dimensional anatomical information is based on preoperative or intraoperative scans of the patient. The method includes acquiring one or more intraoperative measurements of the patient. The method includes determining positional information of a visual representation of one or more intraoperative measurements relative to the three-dimensional anatomical information. The positional information indicates the position of the visual representation of one or more intraoperative measurements relative to at least one planar visual representation of at least one slice of the three-dimensional anatomical information.

[0029] Another aspect of this disclosure relates to a computer program having program code for performing the method when the program is executed on a processor. Attached Figure Description

[0030] The following will describe some examples of the apparatus and / or methods by way of example and with reference to the accompanying drawings, in which:

[0031] Figure 1a A block diagram of an example of a surgical data fusion system or surgical visualization system is shown;

[0032] Figure 1b A schematic diagram of a surgical imaging system including a surgical data fusion system is shown.

[0033] Figure 1c A diagram illustrating a planar visual representation of a slice including three-dimensional anatomical information, overlaid with a visualization of a real-time image from a surgical microscope;

[0034] Figure 2 An example of a radiographic cross section enhanced using real-time optical imaging is shown;

[0035] Figure 3a The plane that defines the radiographic cross-sectional view is shown;

[0036] Figure 3b A combined view including a radiometric cross-sectional view and a real-time optical feed is shown;

[0037] Figure 4 A flowchart illustrating an example of a surgical data fusion method is shown;

[0038] Figure 5A diagram illustrating an example of a system including surgical imaging equipment and a computer system is shown. Detailed Implementation

[0039] Various examples will now be described more fully with reference to the accompanying drawings, some of which illustrate certain instances. In the figures, the thickness of lines, layers, and / or regions may be exaggerated for clarity.

[0040] Figure 1a This illustrates a surgical imaging system 100 ( Figure 1b This is a schematic diagram of an example of a surgical data fusion system 110 (also referred to as a surgical visualization system or control system). For example, the surgical data fusion system 110 may be a component of the surgical imaging system 100, or it may be separate from the surgical imaging system 100, for example, as part of an image-guided surgery (IGS) system 50. Figure 1b (As shown). If the surgical data fusion system is part of the surgical imaging system 100, the surgical data fusion system can be implemented by a computer-based overall control system of the surgical imaging system and used to control various aspects of the surgical imaging system 100. Specifically, the surgical data fusion system can be used to acquire and / or process imaging data by at least one optical imaging sensor of the imaging device 120 of the surgical imaging system through various means, which will be described in more detail below. Additionally, the surgical data fusion system 110 can be configured to control additional aspects of the surgical imaging system 100, such as providing display signals to one or more display devices 130a, 130b of the surgical imaging system, and / or controlling and / or performing automated operations of the surgical imaging system, such as using an automated imaging device positioning system such as a robotic arm.

[0041] Typically, the surgical data fusion system 110 can be a computer system. The surgical data fusion system 110 includes one or more processors 114 and one or more storage devices 116. Optionally, the surgical data fusion system 110 further includes one or more interfaces 112. The one or more processors 114 are coupled to one or more storage devices 116 and one or more interfaces 112. Typically, the functionality of the surgical data fusion system 110 can be provided by one or more processors 114 in conjunction with one or more interfaces 112 (for exchanging data / information with one or more other components of the surgical imaging system 100 and external to the surgical imaging system 100, such as the optical imaging sensors of the imaging device 120 or the IGS system 50) and one or more storage devices 116 (for storing information, such as machine-readable instructions of a computer program executed by one or more processors). Typically, the functionality of the one or more processors 114 can be implemented by one or more processors 114 executing machine-readable instructions. Therefore, any feature attributable to one or more processors 114 can be defined by one or more instructions among a plurality of machine-readable instructions. The surgical data fusion system 110 may include, for example, machine-readable instructions within one or more storage devices 116.

[0042] As outlined above, the surgical data fusion system 110 may be part of the surgical imaging system 100, which includes various components in addition to the surgical data fusion system 110. For example, the surgical imaging system 100 includes an imaging device 120 and may include one or more additional components, such as one or more display devices 130a-130b. Figure 1b A schematic diagram of an example of such a surgical imaging system 100, particularly a surgical microscope system 100, is shown. Hereinafter, the surgical imaging system 100 may also be referred to as a surgical microscope system 100. A surgical microscope system is a surgical imaging system 100 that includes a (surgical) microscope as an imaging device 120. However, the proposed concept is not limited to such embodiments. The surgical imaging system 100 can be based on various (single or multiple) imaging devices, such as one or more microscopes, one or more endoscopes, and / or one or more exoscopes (sometimes also called external telescopes). Exoscopes are camera-based imaging systems, particularly camera-based 3D imaging systems, which are suitable for providing images of surgical sites with high magnification and large depth of field. Compared to microscopes, which can be used via eyepieces, exoscopes are used only via display modes, such as monitors or head-mounted displays. Therefore, the surgical imaging system 100 may alternatively be a surgical endoscope system or a surgical exoscope system. The following illustrations assume that the surgical imaging device 120 is a surgical microscope and the surgical imaging system 100 is a surgical microscope system 100.

[0043] Therefore, the surgical imaging system or surgical microscope system 100 may include imaging devices, such as microscope 120. Typically, a microscope, such as microscope 120, is an optical instrument suitable for examining objects too small to be examined (alone) by the human eye. For example, a microscope can provide optical magnification of a sample. In this concept, optical magnification is also provided for at least one optical imaging sensor. Thus, microscope 120 includes an optical imaging sensor coupled to system 110. Microscope 120 may further include one or more optical magnification components, such as objectives (i.e., lenses), for magnifying views on (e.g., of) a sample. For example, the surgical imaging device or microscope 120 is generally referred to as the 'optical device carrier' of an imaging system.

[0044] Various types of surgical imaging devices exist. If the imaging device is used in the medical or biological field, the object observed through the imaging device can be a sample of organic tissue, for example, arranged in a petri dish or present in a part of the patient's body. In the various examples presented here, imaging device 120 can be a microscope of a surgical microscope system, i.e., a microscope that will be used during surgical procedures (such as neurosurgical procedures (i.e., brain surgery)). Therefore, the sample observed through the surgical imaging device can be a sample of the patient's organic tissue, and in particular, can be the surgical site operated on by the surgeon during the surgical procedure, such as the brain. However, the proposed concept is also applicable to other types of surgery, such as ophthalmic surgery or cardiac surgery.

[0045] Figure 1b A schematic diagram of an example of a surgical imaging system 100, particularly a surgical microscope system 100, is shown, including a surgical data fusion system 110 and a microscope 120. Figure 1b The illustrated surgical microscope system 100 includes several optional components, such as a base unit 105 with a (rolling) support (including a surgical data fusion system 110), an auxiliary display 130b that can be disposed at the base unit, an eyepiece display 130a disposed at the surgical microscope 120, and an arm that holds the microscope 120 in place and couples to both the base unit and the microscope 120. Typically, these optional and non-optional components can be coupled to the surgical data fusion system 110, which can be configured to control and / or interact with the respective components.

[0046] In the proposed concept, the surgical data fusion system 110 is used to align slices (similar to cross-sections in radiographic images) of three-dimensional preoperative or intraoperative anatomical data with intraoperative measurements to visualize them in a combined view. This visualization does not necessarily have to be performed by the surgical data fusion system 110. Instead, in some instances, existing standalone platforms can be used.

[0047] The proposed concept begins with visualizing two sets of data together: preoperative or intraoperative 3D anatomical data and intraoperative measurements. Therefore, the surgical data fusion system 110 is configured to acquire the patient's three-dimensional anatomical information (based on the patient's preoperative or intraoperative scans) and one or more intraoperative measurements.

[0048] For example, three-dimensional anatomical information can be based on preoperative or intraoperative MRI (magnetic resonance imaging), CT (computed tomography) scans, or other 3D scans of the patient's anatomy. For example, three-dimensional anatomical information may not be limited to the surface of the surgical site and may also include a 3D representation of subsurface anatomy. For example, three-dimensional anatomical information may include a three-dimensional model of the patient's anatomy (e.g., at the surgical site). In many cases, the surgical data fusion system 110 can be configured to obtain three-dimensional anatomical information from the image-guided surgery (IGS) system 50. IGS is a technique that provides real-time anatomical guidance during surgical procedures using preoperative and intraoperative imaging. IGS systems typically utilize (3D) data from modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and sometimes ultrasound to create a patient-specific anatomical model. This patient-specific anatomical model can then be used as three-dimensional anatomical information.

[0049] Various types of intraoperative measurements can be visualized along with three-dimensional anatomical information. For example, in surgical imaging systems (such as...) Figure 1b In the surgical imaging system shown, one or more intraoperative measurements may include (for example,) image data (in real time, i.e., periodically updated at a frame rate of more than 1 frame per second) from the surgical imaging device, wherein the image data shows the surgical site.

[0050] Because the proposed concept is particularly well-suited to neurosurgery, where cross-sectional radiographic images are widely used, one or more intraoperative measurements can involve the patient's brain. While such measurements can be provided using surgical imaging equipment 120 (which is a neurosurgical imaging device within a neurosurgical imaging system), functional measurements can also be used for combined visualization. In other words, one or more intraoperative measurements of a patient can include functional imaging data representing the patient's brain activity, along with three-dimensional anatomical information. Examples of functional imaging data include data from fMRI (functional magnetic resonance imaging), which shows brain activity by detecting changes associated with blood flow, and EEG (electroencephalography), which measures electrical activity in the brain using electrodes placed on the scalp. PET (positron emission tomography) scans can also represent brain activity by using radioactive tracers to show metabolic activity. These data can reveal areas of the brain that are active during a specific task or at rest. For example, one or more intraoperative measurements can include fMRI (functional MRI) scans, including a series of measurements representing blood oxygen level-dependent (BOLD) contrast, indicating active brain regions during a task. Alternatively or additionally, one or more intraoperative measurements can include diffusion tensor imaging scans, a variation of MRI scans. DTI scans measure the diffusion of water molecules within the brain to create white matter fasciculations. Additionally or alternatively, one or more intraoperative measurements may include positron emission tomography (PET) scans, including measurements representing the distribution of radioactive tracers to indicate metabolic activity or neurotransmitter binding. Additionally or alternatively, one or more intraoperative measurements may include magnetoencephalography (MEG) measurements, representing magnetic fields generated by electrical activity in the brain. To align these measurements with three-dimensional anatomical information, the three-dimensional anatomical information may include three-dimensional anatomical information of the patient's brain.

[0051] The surgical data fusion system 110 is designed to align three-dimensional anatomical information with intraoperative measurements. For this purpose, the surgical data fusion system 110 is configured to determine positional information of a visual representation 40 of one or more intraoperative measurements relative to the three-dimensional anatomical information. While different types of alignment between preoperative and intraoperative measurements can also be performed in other systems, in the proposed concept, the alignment focuses on a specific type of representation of the three-dimensional anatomical information, analogous to a radiographic cross-sectional view familiar to surgeons. Specifically, the positional information indicates the location of a visual representation 40 of one or more intraoperative measurements relative to at least one planar visual representation 30 of at least one slice of the three-dimensional anatomical information. In other words, the positional information is specific to how the intraoperative measurements are positioned relative to one or more planar slices of the three-dimensional anatomical information. For example, the positional information may represent the spatial relationship between one or more intraoperative measurements (or their visual representations) and a corresponding slice of the three-dimensional anatomical information (or its planar visual representation). Focusing on visualization performed by the surgical data fusion system 110 or by a downstream visualization system, the positional information may represent the spatial relationship between the visual representations of one or more intraoperative measurements and the planar visual representation of at least one slice of the three-dimensional anatomical information. Optionally, the location information may further indicate the extent of the corresponding visual representation of at least one slice of one or more intraoperative measurements and / or three-dimensional anatomical information (e.g., in 2D or 3D space).

[0052] An example of this type of alignment and visualization is in Figure 1c As shown in (and in Figure 2 and Figures 3a to 3b (Expanded in the middle). Figure 1c An illustration shows a planar visual representation 30 of a slice including three-dimensional anatomical information, overlaid with a visual representation 40 of a real-time image from a surgical microscope. Figure 1c In the diagram, the slices shown simulate a cross-sectional view of the brain in the sagittal plane (dividing the brain into left and right sides). At the top of visualization 20, a real-time image 40 (i.e., a visual representation of intraoperative measurements) is overlaid on a slice of three-dimensional anatomical information (i.e., a cross-sectional view 30 of the brain in the sagittal plane). In the proposed concept, at least one planar visual representation of the three-dimensional anatomical information simulates at least one cross-sectional radiographic image, for example, as known from cross-sectional radiographic images representing the axial plane, coronal plane, or sagittal plane. In other words, at least one planar visual representation can represent at least one of the axial plane, coronal plane, or sagittal plane of the three-dimensional anatomical information. The axial plane (also called the transverse plane) divides the body into upper and lower segments, i.e., top and bottom. The coronal plane (also called the frontal plane) divides the body into anterior (frontal) and posterior (back) segments. The sagittal plane divides the body into left and right segments. Figure 2 and Figure 3bThe example is given, showing instances of all three planes. For example, in... Figure 2 In the image, planar visual representation 30a represents the coronal plane, planar visual representation 30b represents the sagittal plane, and planar visual representation 30c represents the axial plane. The corresponding planar visual representations are superimposed with the corresponding visual representations of the real-time images from the surgical imaging device 120. For example, multiple planar visual representations (representing different or the same type of planes, such as the axial plane, coronal plane, and sagittal plane) can be visualized simultaneously, or a single planar visual representation can be visualized (and, for example, selected by the user).

[0053] Each planar visual representation of the three-dimensional anatomical information represents a slice of the three-dimensional anatomical information. In other words, a 3D model of the patient's anatomy is subdivided into slices, where the planar visual representation represents a cross-section of the anatomical structure at one slice. To obtain a meaningful representation, the user can select one or more slices for surgical visualization. In other words, the surgical data fusion system can be configured to receive user input from the user of the surgical data fusion system and select at least one slice of the three-dimensional anatomical information for surgical visualization (and for determining location information) based on the user input. For example, the user can use a slider control to select at least one slice of the three-dimensional anatomical information, where the slider control is part of the surgical visualization or provided via a tactile user interface element of the surgical imaging system. Additionally or alternatively, the surgical visualization may have a user interface element to select which of the coronal, sagittal, and / or axial planes will be represented as the planar visual representation in the surgical visualization. Initially, one or more default slices can be automatically selected, for example, based on the surgical site represented in one or more intraoperative measurements. For example, the surgical data fusion system 110 can be configured to select one or more slices based on the surgical site represented in one or more intraoperative measurements (e.g., based on one or more planes aligned with the center point of the surgical site as represented in one or more intraoperative measurements).

[0054] The determination of location information is central to the proposed concept. As outlined above, location information indicates the position of a visual representation of one or more intraoperative measurements relative to at least one planar visual representation of at least one slice of three-dimensional anatomical information. Therefore, the surgical data fusion system 110 can be configured to determine a) the overall spatial relationship between one or more intraoperative measurements and three-dimensional anatomical information; and b) how and where the one or more intraoperative measurements are visualized relative to at least one planar visual representation of at least one slice of three-dimensional anatomical information. Operation a) can be performed with the assistance of the IGS system 50. For example, the surgical data fusion system 110 can be configured to determine location information (e.g., the spatial relationship between one or more intraoperative measurements and three-dimensional anatomical information) based on the registration functionality of an image-guided surgical system.

[0055] Alternatively, the surgical data fusion system 110 can be configured to use three-dimensional anatomical information and one or more intraoperative measurements to determine the spatial relationship between one or more intraoperative measurements and the three-dimensional anatomical information. For example, the surgical data fusion system 110 can be configured to determine a three-dimensional depth map of the surgical site and determine the spatial relationship between the three-dimensional depth map of the surgical site and the three-dimensional anatomical information (e.g., using anatomical feature matching). For example, the surgical data fusion system 110 can be configured to determine the matching between anatomical features included in the three-dimensional depth map and the three-dimensional anatomical information to determine the spatial relationship. The surgical data fusion system 110 can be further configured to map one or more intraoperative measurements onto a three-dimensional depth map and determine the spatial relationship between one or more intraoperative measurements and the three-dimensional anatomical information based on the spatial relationship between the three-dimensional depth map of the surgical site and the three-dimensional anatomical information, and based on the mapping of one or more intraoperative measurements onto the three-dimensional depth map of the surgical site.

[0056] Various methods exist for determining a suitable depth map, such as optical or ultrasonic time-of-flight sensors. Therefore, a surgical data fusion system can be configured to use sensor data from a time-of-flight sensor to determine a three-dimensional depth map. Alternatively, image data from a surgical imaging device can be used to generate a depth map. In other words, the surgical data fusion system 110 can be configured to generate a three-dimensional depth map based on image data using stereo disparity analysis. Specifically, the surgical data fusion system 110 can be configured to determine a three-dimensional depth map of the surgical site using stereo disparity analysis based on image frames contained in image data from a surgical imaging device showing the surgical site from different viewpoints. Stereo disparity analysis, also known as stereo vision, is a technique for determining depth information based on two or more images taken from slightly different viewpoints. This process involves identifying corresponding points between images and calculating the difference in their positions, known as disparity. The disparity values ​​are then used to estimate the depth map of the surgical site.

[0057] In many cases, surgical imaging devices, such as surgical microscopes, are stereoscopic imaging devices because a stereoscopic view of the surgical site provides surgeons with improved depth perception. Therefore, image data can include stereoscopic image data, showing the surgical site from two slightly different perspectives. Surgical data fusion system 110 can use these different views to generate a three-dimensional depth map. In other words, surgical data fusion system 110 can be configured to determine a three-dimensional depth map of the surgical site using stereoscopic image frames contained in the stereoscopic image data, using stereo parallax analysis. Alternatively or additionally, the surgical imaging device can be moved to obtain different perspectives. In other words, surgical data fusion system 110 can be configured to control surgical imaging system 100 to change the perspective of surgical imaging device 120 (e.g., using a robotic arm) to obtain image frames showing the surgical site from different perspectives.

[0058] Structured light projection can be used to perform or improve the generation of three-dimensional depth maps. Therefore, a surgical data fusion system can be configured to determine a three-dimensional depth map using image data illustrating a structured light projection. Using structured light projection, a three-dimensional depth map can be determined by projecting a known light pattern (typically lines or a grid) onto the surgical site. Therefore, generating a three-dimensional map of a sample using structured light can include projecting a series of known patterns (such as stripes or a grid) onto the surgical site using a projector from surgical data fusion system 110 or surgical imaging system 100. Therefore, surgical imaging system 100 or surgical data fusion system 110 can include a structured light emitter configured to emit structured light toward the surgical site. A camera, such as surgical imaging device 120 or an overview camera, observes how this pattern deforms as it falls on the surface. Surgical data fusion system 110 can use triangulation based on the amount of distortion to calculate the depth at each point. Surgical data fusion system 110 can be configured to measure the displacement of the projected pattern to determine the distance of different points at the surgical site relative to the camera. This knowledge can be used to generate a three-dimensional depth map of the sample.

[0059] Operation b (how and where to visualize one or more intraoperative measurements relative to at least one planar visual representation of at least one slice of the three-dimensional anatomical information) depends on one or more slices of the three-dimensional anatomical information, and on the extent of the corresponding visual representation of the intraoperative measurements. For example, the surgical data fusion system 110 can be configured to determine, for example, the spatial relationship between at least one slice of the three-dimensional anatomical information and one or more intraoperative measurements based on the determined overall spatial relationship between the one or more intraoperative measurements and the three-dimensional anatomical information. Therefore, location information may include information about the spatial relationship between at least one slice of the three-dimensional anatomical information and one or more intraoperative measurements (e.g., including relative position and / or relative rotational offset). Additionally, the surgical data fusion system 110 can be configured to determine the extent of the visual representation of one or more intraoperative measurements (e.g., in two dimensions in the same plane as at least one slice, or in three dimensions relative to the plane of at least one slice) (which is inherent to one or more intraoperative measurements) to determine how the visual representation of one or more intraoperative measurements intersects with the planar visual representation of at least one slice. Therefore, location information may include information about how the visual representation of one or more intraoperative measurements intersects with the planar visual representation of at least one slice.

[0060] In some instances, the surgical data fusion system 110 acts as a surgical visualization system and provides surgical visualization. In other words, the surgical data fusion system 110 can be configured to generate a surgical visualization 20 of the surgical site 10. This surgical visualization is based on location information, taking into account a planar visual representation of at least one slice and a visual representation of one or more intraoperative measurements. Specifically, the surgical visualization may include at least one planar visual representation 30 of at least one slice of three-dimensional anatomical information, overlaid with a visual representation 40 of one or more intraoperative measurements. For example, the surgical data fusion system 110 can be configured to place the overlay on at least one planar visual representation based on location information.

[0061] The primary application of the proposed concept is the combined visualization of planar visual representations of real-time image data from surgical imaging devices and corresponding slices of three-dimensional anatomical information. Therefore, a visual representation of one or more intraoperative measurements can be based on image data from a surgical imaging device that shows the surgical site. For example, in Figure 1b , Figure 2 and Figure 3bAn example of this visualization is illustrated. To improve the fidelity of this visualization, the aforementioned 3D depth map can be used to generate a visual representation of the image data. For example, a surgical data fusion system can be configured to map image data onto a 3D depth map of the surgical site using image projection, where a visual representation of one or more intraoperative measurements is based on the image data mapped onto the 3D depth map of the surgical site. Typically, image projection is the process of mapping points from one space to another. In many cases, image projection is used to project 3D objects onto a 2D image plane, simulating how a 2D camera captures a 3D scene. In the proposed concept, 2D-to-3D (from image data to a depth map) to 2D (from a depth map to a 2D version of the visual representation of the image data to be included in the surgical visualization) projection can be used. For example, mathematical operations can be used to project image information contained in the image data onto the depth map, for example, based on a disparity map used to generate the 3D depth map, or using feature mapping. The image information contained in the 3D depth map of the surgical site can then be projected using mathematical operations into a corresponding 2D visual representation of the image data based on positional information, since the positional information indicates the viewpoint at the surgical site used for surgical visualization. These mathematical operations are based on the perspective of the corresponding two-dimensional image relative to a three-dimensional object. Specifically, the pixels of the image data can first be projected onto a three-dimensional depth map, and then reprojected from the three-dimensional depth map onto an image plane used for surgical visualization. On the other hand, the image plane used for surgical visualization is a plane based on a planar visual representation of the three-dimensional depth map.

[0062] The same technique can be used for other types of intraoperative measurements. First, the corresponding measurements can be mapped onto a three-dimensional depth map of the surgical site, and then onto an image plane used for surgical visualization. Therefore, the surgical data fusion system 110 can be configured to generate surgical visualizations based on the three-dimensional depth map of the surgical site.

[0063] Typically, the fusion of visual representations of one or more intraoperative measurements and planar visual representations of at least one slice can prioritize the visual representation of the one or more intraoperative measurements, as these measurements can be real-time and thus reflect the current state of the surgical site. Therefore, the shape of the surgical site as represented in one or more intraoperative measurements can be used to replace the corresponding portion of the planar visual representation of the slice based on the three-dimensional anatomical information. For example, surgical data fusion system 110 can be configured to generate visual representations of one or more intraoperative measurements based on the shape of the surgical site at the location of at least one slice of three-dimensional anatomical information. For example, a portion of the planar visual representation of a corresponding slice can be replaced by the visual representation of one or more intraoperative measurements (e.g., a superimposed visual representation of one or more intraoperative measurements) based on the shape of the surgical site at the location of at least one slice of three-dimensional anatomical information.

[0064] As outlined above, one or more intraoperative measurements can be more timely than preoperative or intraoperative scans of the patient included in the three-dimensional anatomical information. Specifically, the patient's anatomy may have been altered to fit into the surgical site during surgery, for example, when a portion of the skull has been removed or when tissue has been moved aside. Since these changes are represented in one or more intraoperative measurements, they can also be incorporated into a planar visual representation of the corresponding slice. In other words, the surgical data fusion system can be configured to determine the portion of the surgical site that has been removed compared to the preoperative scan, based on the three-dimensional anatomical information and a three-dimensional depth map of the surgical site. This can also be accomplished by comparing anatomical features included in the three-dimensional anatomical information with anatomical features represented in one or more intraoperative measurements (e.g., in a three-dimensional depth map generated based on one or more intraoperative measurements). The surgical data fusion system 110 can be configured to generate at least one planar visual representation of at least one slice of the three-dimensional anatomical information that does not contain that portion of the surgical site that could be removed. Similarly, changes not attributable to the removal of a portion of the surgical site can also be tracked. Specifically, the surgical data fusion system can be configured to determine at least a portion of the surgical site based on three-dimensional anatomical information and a three-dimensional depth map of the surgical site, and to adjust at least one planar visual representation of at least one slice of the three-dimensional anatomical information based on the displacement or deformation of at least that portion of the surgical site.

[0065] The surgical data fusion system 110 can be configured to provide surgical visualization to display devices, such as eyepiece display 130a or auxiliary display 130b. For example, the surgical data fusion system 110 can be configured to output display signals that include surgical visualization for the display devices. For example, the display signals can be signals used to drive (e.g., control) the corresponding display devices. For example, the display signals can include video data and / or control commands for driving the displays. For example, the display signals can be provided via one of one or more interfaces 112 of the system. Therefore, the surgical data fusion system 110 can include a video interface 112 adapted to provide display signals to the display devices 130a, 130b of the microscope system 100.

[0066] In the proposed surgical imaging system, the optical imaging sensor of the surgical imaging device 120 can be used to provide the aforementioned image data. Therefore, an optical imaging sensor, which may be part of the imaging device 120 (e.g., a microscope), can be configured to generate image data. For example, the optical imaging sensor of the imaging device 120 may include or be an APS (Active Pixel Sensor) based imaging sensor or a CCD (Charge-Coupled Device) based imaging sensor. For example, in an APS-based imaging sensor, light is recorded at each pixel using a photodetector and an active amplifier. APS-based imaging sensors are typically based on CMOS (Complementary Metal-Oxide-Semiconductor) or S-CMOS (Scientific CMOS) technology. In a CCD-based imaging sensor, incident photons are converted into electronic charges at the semiconductor-oxide interface, which then move between capacitors within the imaging sensor via circuitry to perform imaging.

[0067] One or more interfaces 112 of system 110 may correspond to one or more inputs and / or outputs for receiving and / or sending information within a module, between modules, or between modules of different entities. This information may be a digital (bit) value based on a specified code. For example, one or more interfaces 112 may include interface circuitry configured to receive and / or send information. One or more processors 114 of system 110 may be implemented using one or more processing units, one or more processing devices, or any means for processing, such as a processor, computer, or programmable hardware component that can operate with appropriately adapted software. In other words, the described functionality of one or more processors 114 may also be implemented in software, which then executes on one or more programmable hardware components. Such hardware components may include general-purpose processors, digital signal processors (DSPs), microcontrollers, etc. One or more storage devices 116 of system 110 may include at least one element from the group of computer-readable storage media, such as magnetic or optical storage media, such as hard disk drives, flash memory, floppy disks, random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or network storage devices.

[0068] Combined with the proposed concept or one or more instances described above or below (e.g., Figures 2 to 5 Further details and aspects of the surgical data fusion system 110 and the surgical imaging system 100 are mentioned. The surgical data fusion system 110 and the surgical imaging system 100 may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more instances described above or below.

[0069] Various examples of this disclosure relate to the integration of radiographic cross-sectional visualization with real-time surgical microscopy (radiological 3D visualization) or other intraoperative measurements. The proposed concept details a novel visualization method that combines preoperative 3D imaging data with real-time optical imaging (or other measurements) obtained from a surgical microscope. This data is presented in the form of radiographic cross-sections, which may include axial, coronal, and sagittal views. The system can capture the 3D surface of the surgical cavity intraoperatively using methods such as stereoscopic optical imaging, structured light projection (SLP), or time-of-flight (TOF) sensors. This allows the system to align and integrate real-time optical data or other measurements with preoperative 3D data. The resulting hybrid visualization provides surgeons with an augmented reality view that combines familiar and precise radiographic cross-sections, enabling intuitive understanding and measurement of 3D spatial relationships during surgery.

[0070] Figure 2 An example of radiographic cross-section enhancement using real-time optical imaging is shown. This figure illustrates how radiographic cross-section views 30a-30c—axial view 30c, coronal view 30a, and sagittal view 30b—are integrated with real-time optical imaging data 40a, 40b, and 40c. The different selected planes (coronal plane 200a, sagittal plane 200b, and axial plane 200c) are shown in the corresponding lower right. Figure 2 In the example shown, the system aligns real-time optical data with preoperative 3D scans, maintaining a familiar radiological perspective while providing updated and accurate views during surgery. The left side shows a standard visualization of medical 3D scan data, while the right side shows the proposed hybrid optics-3D medical imaging visualization.

[0071] The proposed concept provides a mechanism for enhancing intraoperative visualization by integrating radiographic cross-sectional imaging, for example, with real-time optical imaging obtained from surgical microscopes or other intraoperative measurements. This approach leverages the advantages of both visualization methods, providing surgeons with detailed spatial information in a familiar format while maintaining a true view of the surgical field.

[0072] The system is based on one or more key components. For example, a surgical imaging device with 3D surface capture capabilities, such as a surgical microscope, can be utilized. This surgical imaging device can be equipped with technologies such as stereoscopic optical imaging, structured light projection (SLP), time-of-flight (ToF) sensors, or other 3D scanning technologies, enabling it to capture the real-time 3D surface of the surgical cavity during surgery (e.g., as part of one or more intraoperative measurements).

[0073] Preoperative or intraoperative 3D surgical imaging data (e.g., three-dimensional anatomical information) obtained from modalities such as MRI (magnetic resonance imaging) or CT (computed tomography) scans can provide detailed anatomical and pathological structures relevant to the surgery. Image-guided surgery (IGS) systems can be used for spatial alignment (registration) of intraoperative 3D surface data with preoperative 3D imaging data. This system can also provide real-time updates of surgical instrument tracking and combined data.

[0074] Finally, visualization modules (such as the surgical data fusion system 110) can be used to process and combine aligned data to generate radiographic cross-sectional views in the axial, coronal, and sagittal planes, and integrate these views with real-time optical imaging to create augmented reality displays.

[0075] The system's workflow and implementation can be as follows: Surgical imaging equipment can be used to continuously capture the 3D surface of the surgical cavity during the procedure, generating real-time surface maps using methods such as stereoscopic imaging. The IGS system can be used to align intraoperative 3D surface data with preoperative 3D imaging data, achieving spatial alignment through the use of reference markers, anatomical landmarks, or surface matching algorithms. The aligned 3D data can be processed to extract cross-sectional images in the axial, coronal, and sagittal planes, combining both preoperative and intraoperative optical surface data. These cross-sectional images can be overlaid onto a real-time optical feed from the surgical microscope, allowing the surgeon to directly view the combined data in a familiar radiographic format within the surgical field of view.

[0076] In some instances, interactive features may be included, allowing surgeons to adjust the position of the cross-sectional planes in real time to focus on the region of interest, quantify distances and spatial relationships using measurement tools, and display all three planes simultaneously or switch between them.

[0077] Figure 3a and Figure 3b A visualization of the radiometric cross-section superimposed on a real-time optical image is shown. Figure 3a Planes 200a, 200b, and 200c, which define the radiometric cross-sectional views, are shown. Figure 3b A combined view including a radiographic cross-sectional view and a real-time optical feed is shown. Slider 300a, 300b, and 300c can be used to shift the coronal plane 200a (slider 300a), the sagittal plane 200b (slider 300b), and / or the axial plane 200c (slider 300c). Figure 3a and Figure 3bThis demonstrates the overlay (or vice versa) of radiographic cross-sectional views 30 (axial, coronal, and sagittal views) from preoperative 3D scans onto a real-time optical feed 40 (optical image data) from a stereoscopic surgical microscope. The combined view provides the surgeon with a true optical view of the surgical field and precise spatial information derived from the preoperative 3D data. The system's interactive features allow the surgeon to explore different planes and focus on areas of interest during surgery.

[0078] The proposed mechanism provides a familiar visualization format, presenting surgeons with radiographic cross-sectional views they are accustomed to interpreting, thereby enhancing understanding. It enhances 3D spatial understanding by combining real-time optical data with preoperative imaging, providing precise spatial relationships between structures. Furthermore, it improves surgical planning and execution, allowing for accurate navigation and manipulation of instruments relative to key anatomical features. Additionally, it simplifies workflow by reducing cognitive burden by eliminating the need for mental mapping between separate displays. Finally, it offers versatility, as it can be integrated into existing IGS systems and surgical microscopes with 3D imaging capabilities.

[0079] Several alternative implementation methods are possible. Different 3D surface capture methods can be employed; while stereoscopic optical imaging is one option, structured light projection (SLP) or time-of-flight (TOF) sensors can also be used to capture 3D surfaces. Advanced visualization techniques, such as combining volumetric rendering or multiplanar reconstruction, can enhance visualization. User interface customization can be provided, allowing surgeons to tailor display settings such as opacity levels, color schemes, and planar orientation. Finally, integration with other modalities, including functional imaging data (e.g., fMRI, DTI), can provide additional information about tissue functionality.

[0080] The proposed concept integrates radiographic cross-sectional visualization with real-time optical imaging in surgical microscopy. It provides augmented reality views combining axial, coronal, and / or sagittal planes with real-time optical imaging. It offers radiographically inspired visualization, enhanced 3D spatial understanding, and / or hybrid optical and radiographic imaging during surgery.

[0081] Combined with the proposed concept or one or more instances described above or below (e.g., Figures 1a to 1c , Figures 4 to 5 Further details and aspects of the integration of radiographic cross-sectional visualization with real-time surgical microscopy are mentioned. The integration of radiographic cross-sectional visualization with real-time surgical microscopy may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more instances described above or below.

[0082] Figure 4A flowchart illustrating an example of a surgical data fusion method is shown. The method includes obtaining three-dimensional anatomical information of 410 patients, based on preoperative or intraoperative scans of the patients. The method includes obtaining one or more intraoperative measurements of 420 patients. The method includes determining positional information of 430 visual representations of the one or more intraoperative measurements relative to the three-dimensional anatomical information. The positional information indicates the position of the visual representation 40 of the one or more intraoperative measurements relative to at least one planar visual representation 30 of at least one slice of the three-dimensional anatomical information. Optionally, the method may further include generating a surgical visualization 20 of 440 surgical sites and providing 450 surgical visualization to a display device. For example, the method may be... Figure 1a and Figure 1b The surgical data fusion system 110 is executed. Combined with... Figure 1a and Figure 1b The surgical data fusion system 110, or combined with Figure 2 and Figures 3a to 3b The features introduced by the provided instances can also be included in Figure 4 The corresponding surgical data fusion method.

[0083] Combined with the proposed concept or one or more instances described above or below (e.g., Figures 1a to 3b , Figure 5 The text mentions further details and aspects of surgical data fusion methods. These methods may include one or more additional optional features corresponding to one or more aspects of the proposed concept or one or more instances described above or below.

[0084] As used herein, the term “and / or” includes any and all combinations of one or more of the associated entries listed, and may be abbreviated to “ / ”.

[0085] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent descriptions of the corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent descriptions of corresponding blocks, items, or features of the corresponding apparatus.

[0086] Some implementations relate to an imaging device that includes, for example, a combination of Figures 1a to 4 The system described in one or more of these. Alternatively, imaging devices such as microscopes, exoscopes, or endoscopes can be combined as follows. Figures 1a to 4 One or more of the ones described are part of or connected to the system. Figure 5A schematic diagram of a system 500 configured to perform the methods described herein is shown. System 500 includes an imaging device 510 and a computer system 520. The imaging device 510 is configured to capture images and is connected to the computer system 520. The computer system 520 is configured to perform at least a portion of the methods described herein. The computer system 520 may be configured to execute machine learning algorithms. The computer system 520 and the imaging device 510 may be separate entities, but may also be integrated together in a common enclosure. The computer system 520 may be part of the central processing system of the imaging device 510, and / or the computer system 520 may be part of a sub-component of the imaging device 510, such as a sensor, actuator, camera, or illumination unit of the imaging device 510.

[0087] Computer system 520 may be a local computer device (e.g., a personal computer, laptop computer, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations, such as local clients and / or one or more remote server farms and / or data centers). Computer system 520 may include any circuitry or combination of circuitry. In one embodiment, computer system 520 may include one or more processors of any type. As used herein, a processor may mean any type of computing circuitry, such as, but not limited to, an imaging device or imaging device component (e.g., a camera), such as, but not limited to, a microprocessor, microcontroller, complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, graphics processor, digital signal processor (DSP), multi-core processor, field-programmable gate array (FPGA), or any other type of processor or processing circuitry. Other types of circuitry that may be included in computer system 520 may be custom circuitry, application-specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuitry) used in wireless devices like mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. Computer system 520 may include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives that process removable media (such as optical discs (CDs), flash memory cards, digital video discs (DVDs), etc.). Computer system 520 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touchscreen, voice recognition device, or any other device that allows a system user to input information into and receive information from computer system 520.

[0088] Some or all of the method steps may be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers, or electronic circuits. In some embodiments, one or more of the most important method steps may be performed by such devices.

[0089] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementations can be performed using a non-transitory storage medium (such as a digital storage medium, e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory) that stores electronically readable control signals thereon, which cooperate with (or are capable of cooperating with) a programmable computer system to cause the corresponding methods to be executed. Therefore, the digital storage medium can be computer-readable.

[0090] Some embodiments of the invention include a data carrier having an electronically readable control signal that is capable of cooperating with a programmable computer system to perform one of the methods described herein.

[0091] Typically, embodiments of the present invention can be implemented as a computer program product having program code that, when run on a computer, is operable to perform one of the methods. The program code may, for example, be stored on a machine-readable medium.

[0092] Other implementations include a computer program stored on a machine-readable medium for performing one of the methods described herein.

[0093] In other words, embodiments of the present invention are therefore computer programs having program code that, when run on a computer, performs one of the methods described herein.

[0094] Therefore, another embodiment of the invention is a storage medium (or data carrier, or computer-readable medium) comprising a computer program stored thereon for performing one of the methods described herein when executed by a processor. Data carriers, digital storage media, or recording media are generally tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0095] Therefore, another embodiment of the invention represents a data stream or signal sequence for a computer program used to perform one of the methods described herein. The data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0096] Another embodiment includes a processing device, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0097] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0098] Another embodiment of the invention includes an apparatus or system configured to transmit a computer program to a receiver (e.g., electronically or optically), the computer program being used to perform one of the methods described herein. The receiver may be, for example, a computer, mobile device, memory device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0099] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functionality of the methods described herein. In some embodiments, the field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0100] List of reference numerals

[0101] 10: Surgical site

[0102] 20: Visualization

[0103] 30, 30a, 30b, 30c: Planar visual representation of slices containing 3D anatomical information

[0104] 40, 40a, 40b, 40c: Visual representation of one or more intraoperative measurements.

[0105] 50: IGS system

[0106] 100: Surgical Imaging System

[0107] 110: Surgical Data Fusion System

[0108] 112: One or more interfaces

[0109] 114: One or more processors

[0110] 116: One or more storage devices

[0111] 120: Imaging equipment

[0112] 130a: Eyepiece display

[0113] 130b: Auxiliary display

[0114] 200a: Coronal plane

[0115] 200b: Sagittal plane

[0116] 200c: Axial plane

[0117] 300a: A slider used to change the coronal plane.

[0118] 300b: A slider used to change the sagittal plane.

[0119] 300c: A slider used to change the axis plane.

[0120] 410: Obtain three-dimensional anatomical information

[0121] 420: Obtain one or more intraoperative measurements.

[0122] 430: Determine location information

[0123] 440: Generating Surgical Visualizations

[0124] 450: Provides surgical visualization

[0125] 500: System

[0126] 510: Imaging equipment

[0127] 520: Computer System

Claims

1. A surgical data fusion system (110), the surgical data fusion system comprising one or more processors and one or more storage devices, wherein the surgical data fusion system is configured to: Obtain the patient's three-dimensional anatomical information, which is based on the patient's preoperative or intraoperative scans; Obtain one or more intraoperative measurements of the patient; and Determine the positional information of the visual representation (40) of the one or more intraoperative measurements relative to the three-dimensional anatomical information, wherein the positional information indicates the position of the visual representation (40) of the one or more intraoperative measurements relative to at least one planar visual representation (30) of at least one slice of the three-dimensional anatomical information.

2. The surgical data fusion system according to claim 1, wherein the at least one planar visual representation simulates at least one cross-sectional radiographic image.

3. The surgical data fusion system according to any one of claims 1 or 2, wherein the at least one planar visual representation represents at least one of the axial plane, coronal plane, or sagittal plane of the three-dimensional anatomical information.

4. The surgical data fusion system of claim 1, wherein the surgical data fusion system is configured to generate a surgical visualization (20) of a surgical site (10), the surgical visualization including at least one planar visual representation (30) of at least one slice of the three-dimensional anatomical information, the at least one planar visual representation superimposed with a visual representation (40) of the one or more intraoperative measurements, wherein the superposition is placed on the at least one planar visual representation based on the position information, and the surgical data fusion system is configured to provide the surgical visualization to a display device (130a, 130b).

5. The surgical data fusion system of claim 4, wherein the surgical data fusion system is configured to obtain user input from a user of the surgical data fusion system and select at least one slice of the three-dimensional anatomical information for surgical visualization based on the user input.

6. The surgical data fusion system according to any one of claims 4 or 5, wherein the one or more intraoperative measurements of the patient include image data from a surgical imaging device showing the surgical site, and wherein the visual representation of the one or more intraoperative measurements is based on the image data.

7. The surgical data fusion system according to any one of claims 4 to 6, wherein the surgical data fusion system is configured to determine a three-dimensional depth map of the surgical site and generate the surgical visualization based on the three-dimensional depth map of the surgical site.

8. The surgical data fusion system of claim 7, wherein the surgical data fusion system is configured to generate the visual representation of the one or more intraoperative measurements based on the shape of the surgical site at the location of the at least one slice of the three-dimensional anatomical information.

9. The surgical data fusion system according to claim 7 or 8, wherein the surgical data fusion system is configured to determine a portion of the surgical site that has been removed compared to the preoperative scan based on the three-dimensional anatomical information of the surgical site and the three-dimensional depth map, and to generate the at least one planar visual representation of the at least one slice of the three-dimensional anatomical information, wherein the portion of the removed surgical site is not present. And / or wherein the surgical data fusion system is configured to determine at least a portion of the surgical site’s displacement or deformation based on the three-dimensional anatomical information and the three-dimensional depth map of the surgical site, and to adjust the at least one planar visual representation of the at least one slice of the three-dimensional anatomical information based on the displacement or deformation of at least the portion of the surgical site.

10. The surgical data fusion system according to any one of claims 7 to 9, wherein the surgical data fusion system is configured to determine the three-dimensional depth map of the surgical site using stereo parallax analysis based on image frames contained in image data of a surgical imaging device showing the surgical site from different perspectives. Alternatively, the surgical data fusion system is configured to determine the three-dimensional depth map using at least one of sensor data from a time-of-flight sensor or image data showing a structured light projection.

11. The surgical data fusion system according to any one of claims 1 to 10, wherein the surgical data fusion system is configured to obtain the three-dimensional anatomical information of the image-guided surgical system (50) and determine the location information based on the registration functionality of the image-guided surgical system.

12. The surgical data fusion system according to any one of claims 1 to 11, wherein the one or more intraoperative measurements of the patient include functional imaging data representing the patient's brain activity, and the three-dimensional anatomical information includes three-dimensional anatomical information of the patient's brain.

13. A surgical imaging system (100) comprising a surgical imaging device (120) and a surgical data fusion system (110) according to any one of claims 1 to 12.

14. A surgical data fusion method, the surgical data fusion method comprising: Obtain (210) three-dimensional anatomical information of the patient, the three-dimensional anatomical information being based on preoperative or intraoperative scans of the patient; Obtain one or more intraoperative measurements of the patient described in (220); Determine (230) the positional information of the visual representation of the one or more intraoperative measurements relative to the three-dimensional anatomical information, wherein the positional information indicates the position of the visual representation (40) of the one or more intraoperative measurements relative to at least one planar visual representation (30) of at least one slice of the three-dimensional anatomical information.

15. A computer program having program code for performing the method according to claim 14 when the program is executed on a processor.

Images