Augmented reality head-mounted viewer and probe for medical imaging

Abstract

An augmented reality (AR) system and method for obtaining measurements from a patient during a medical procedure and displaying the measurements is provided. The system comprises: an AR head-mounted viewer comprising a near-eye display for displaying a measurement superimposed in a field of view of a surgeon to a patient's tissue at a location relative to the patient from which the measurement was collected; at least one sensor for tracking a location of the patient; a memory for storing the measurement and a location associated with the measurement; wherein the system is configured to perform the following steps during the medical procedure: (a) receiving a measurement collected from the patient's tissue, wherein the measurement is collected by a probe positioned proximate to a different region of the patient's tissue; (b) tracking a location of the probe while the probe collects the measurement to determine a location at which the measurement was collected; (c) simultaneously performing steps (a) and (b): tracking the location of the biomarker on the patient using the at least one sensor so that a relationship between the location of the biomarker and the location at which the measurement was collected can be determined; (d) displaying the measurement on the near-eye display at a location so that throughout the medical procedure the measurement is aligned in the field of view of the surgeon to the patient's tissue at the location relative to the patient from which the measurement was collected: continuously tracking the location of the biomarker throughout the medical procedure to obtain a current location of the biomarker; updating the location of the measurement on the display based on the current location of the biomarker.

Description

Technical Field

[0001] This invention relates to an augmented reality system for medical procedures. Background Technology

[0002] Cancer surgery consists of removing the tumor. To ensure complete removal, surgeons remove a portion of the healthy tissue surrounding the tumor. Complete removal is crucial to preventing tumor recurrence, which is associated with increased surgical costs, morbidity, and mortality. To ensure complete removal, the tumor needs to be correctly identified and accurately mapped.

[0003] Current technologies used for tumor identification and mapping include:

[0004] • Biopsy, that is, obtaining a tissue sample using a needle for post-pathological analysis of the tissue sample in a laboratory;

[0005] • Patient scan, i.e., assessing tumor margins on a patient's medical images;

[0006] • Thermal imaging, namely, using infrared thermal imaging for cancer detection;

[0007] Raman spectroscopy, which allows the analysis of the chemical composition of biological tissues, has been widely used in the past decade for cancer screening, diagnosis, and intraoperative surgical guidance.

[0008] Raman spectroscopy, used for tumor identification and mapping, is a powerful technique because it allows surgeons to distinguish between healthy and cancerous tissue during surgery, while being less invasive than biopsy. However, known Raman techniques have a key limitation: surgeons need to perform several measurements with the probe to obtain a complete map of the cancerous tissue, as the tissue area analyzed in a single session is very small (approximately 1 mm). This means that surgeons must either recall multiple locations of cancerous tissue identified by probe measurements during surgery or use physical markers (such as labels) to mark the areas of cancerous tissue being detected. The former leads to reduced accuracy, while the latter results in longer surgical times because the surgeon must stop and place the physical markers in the correct positions.

[0009] A further solution to this problem is to observe the location of the cancerous tissue on an image of the patient's body displayed on a monitor in the operating room. However, surgeons must match this information to the actual patient's body beforehand. This can lead to reduced surgical accuracy because the matching relies on the surgeon's ability to apply the location shown on the monitor to the patient, making it susceptible to human error. Furthermore, this results in longer surgical times because surgeons must constantly leave the patient to look at the monitor.

[0010] The present invention aims to solve the aforementioned problems associated with the current program. Summary of the Invention

[0011] According to a first aspect, an augmented reality (AR) system is provided for obtaining and displaying measurements from a patient during a medical procedure. The system includes: an AR headset, at least one sensor, a memory, and a processor. The AR headset includes a near-eye display for displaying the measurements superimposed on a surgeon's field of vision of the patient's tissue at a location relative to the patient from which the measurements are collected; at least one sensor for tracking the patient's position; and a memory for storing the measurements and the positions associated with the measurements. The system is configured to perform the following steps during the medical procedure: (a) receiving measurements collected from the patient's tissue, wherein the measurements… (a) collecting measurements from probes located in different regions of adjacent patient tissue; (b) tracking the position of the probes while they are collecting measurements to determine the location where the measurements are collected; (c) performing steps (a) and (b) simultaneously: tracking the position of the biomarker on the patient using at least one sensor such that the relationship between the position of the biomarker and the location where the measurements are collected can be determined; (d) displaying the measurements at a location on a near-eye display such that, throughout the medical procedure, the measurements are aligned at that location relative to the patient in the surgeon’s field of vision of the patient tissue by: continuously tracking the position of the biomarker throughout the medical procedure to obtain the current position of the biomarker; and updating the position of the measurements on the display based on the current position of the biomarker.

[0012] In this way, since the location of the biomarker is known at the same time as the probe measurement, the relationship between their relative positions can be determined. Therefore, by continuously tracking the location of the biomarker, the measurement can be superimposed on the display relative to its correct location obtained from the patient's body when displayed at a later time point.

[0013] As a biomarker for tracking, it eliminates the need to place physical tags on or around the patient, resulting in a tag-free procedure. This is advantageous because biomarkers do not fall off the patient like physically attached tags. Furthermore, surgeons do not need to stop and reattach tags to the patient, potentially reducing the duration of surgery or the harm and risks to the patient.

[0014] When tracking a patient's body in real time, surgeons can see measurements directly on an augmented reality display within their line of sight. In other words, these measurements are superimposed on the patient's real-world view at their corresponding location. This means surgeons don't have to constantly leave the patient's body to look at external monitors. This also reduces the errors associated with surgeons subjectively translating what they see on the monitor into the patient's actual body.

[0015] The step of updating the position of a measurement on a display based on the current position of a biomarker involves the relationship between the location of the biomarker and the location where the measurement was collected.

[0016] The location where measurements are collected is the location on the patient where each measurement is collected. In other words, it is the point on the patient from which the probe has taken the measurement. This relationship can be between the location of the biomarker and the location where each measurement is taken, determined by tracking the location where each measurement is taken. Depending on the nature of the probe and the measurement to be performed, the term "adjacent to patient tissue" can be interpreted as meaning immediately adjacent to, on, or within the patient tissue.

[0017] Preferably, a biomarker is one or more features of a biomarker. In this way, one or more features of the biomarker are tracked. For example, this may involve using at least one sensor to track the location of one or more features of the biomarker on the patient, such that the relationship between the location of the one or more features of the biomarker and the location where the measurement is collected can be determined. It may also include continuously tracking the location of one or more features of the biomarker throughout the medical procedure to obtain its current location; and updating the location of the measurement on the display based on the current location of the one or more features of the biomarker.

[0018] One or more features of a biomarker may include the edges or intersections of the biomarker. These edges or intersections can be identifiable elements of the biomarker. Advantageously, these intersections or edges can provide easily identifiable reference points on the biomarker. These features can be extracted from raw data obtained by the sensor.

[0019] Sensors can track the patient's position (i.e., location and orientation). The current position is the location of the biometrics at the current moment, as the patient's position may change throughout the procedure due to the patient's movement and the movement of the AR head-mounted viewer relative to each other.

[0020] The method may also include creating a virtual map of the measurements, wherein displaying the measurements on a near-eye monitor includes displaying the virtual map. The virtual map can provide surgeons with visual guidance regarding cancerous areas in a patient's tissue. The virtual map can also be a spatial map of the measurements that provides surgeons with an intuitive and easily interpretable view.

[0021] A virtual map can be a color-coded data map showing regions with different measurements represented by different colors. For example, cancerous tissue can be represented by a different color than areas of non-cancer tissue. Alternatively or additionally, the amount of cancerous tissue can be represented by the colors of the virtual map. Regions with a higher percentage of cancerous tissue can have a different color than regions with a lower percentage of cancerous tissue.

[0022] Preferably, tracking the location of biomarkers on a patient may further include illuminating an area of ​​the patient with light of a specific wavelength and detecting, by a sensor, the fluorescent signal emitted by the biomarker in response to the illumination. Prior to surgery, a fluorescent dye can be injected into the patient's venous system. The dye can then accumulate in the biomarker of interest. This could be, for example, a blood vessel in the patient. Advantageously, by using a fluorescent dye, it is possible to track internal areas of the patient located within the patient's body. This is advantageous because it allows the tracking point to be as close as possible to the point of interest. Furthermore, it eliminates the need for invasive surgical procedures involving the insertion of physical markers near the detection location. Since the patient's body is not fixed in place and will move throughout the procedure (e.g., during breathing), tracking the biomarker allows the marker to move with the patient's body, thereby improving tracking and thus superimposing the measurements on a display.

[0023] The fluorescent marker can be any known fluorescent dye or contrast agent. For example, it can be indocyanine green (ICG). Irradiation of the patient may involve the use of a near-infrared (NIR) light source. The light source excites the molecular ligands in the biomarker, which respond by emitting light at a specific wavelength. A camera sensitive to this range of the spectrum can be used to detect the light. The detected light can be in the form of raw data, which can be used to form an image that clearly represents the biomarker. The wavelength of the NIR can be selected depending on the dye used. The fluorescent signal is used for location tracking purposes, rather than representing a specific medical measurement itself. The NIR camera and NIR light source can be located on an AR head-mounted viewer. In other arrangements, the NIR camera and / or NIR light source can be positioned separately from the AR head-mounted viewer. Alternatively, any other type of fluorescent marker can be used, such as methylene blue, alanine blue, and other fluorescent markers that are sensitive to light in the visible spectrum. The camera and light source can be selected based on the type of contrast agent used in the visible, NIR, or IR wavelengths.

[0024] The AR system may also include a probe. Preferably, the probe is a Raman probe. Advantageously, Raman spectroscopy provides a non-invasive procedure for identifying and mapping cancerous tissue. By presenting these measurements on a near-eye display, the surgeon can observe the measurements in the correct position relative to the patient within their direct line of sight. The probe can be a handheld probe. For example, it can be a handheld tool for collecting measurements from the patient. The Raman probe may include a diode laser for exciting the tissue and a sensor for detecting the Raman scattering measurements from the tissue. However, other arrangements of the Raman probe can be used.

[0025] Alternatively, the probe can be a temperature probe capable of acquiring temperature measurements from the patient. These temperature measurements can be displayed as a virtual map on a monitor, showing their correct location relative to the site from which they were collected in the surgeon's view. In other arrangements, the probe can be an electromagnetic probe. For example, an electromagnetic probe could be an electromagnetic pulsating probe. Electromagnetic pulsating probes can be used to take measurements from or deliver pulses to the patient's brain during invasive procedures. Alternatively, any other type of probe, such as a pH probe, can be used.

[0026] When an AR system includes a probe, the AR system may also include the step of using the probe to perform data collection measurements. However, it should also be understood that the probe can alternatively be considered independent of the AR system.

[0027] The medical procedure can be a surgical procedure. Specifically, the medical procedure can be cancer surgery. This can be a surgical procedure to remove cancerous tissue (such as a tumor) from a patient. This invention allows for the collection of measurements during the surgical procedure, while enabling the surgeon to visualize the measurements throughout the procedure. Alternatively, in other arrangements, measurements can be collected prior to surgery for later display during the surgical procedure. The measurements can be used as guidance for the surgeon throughout the surgical procedure.

[0028] Preferably, at least one sensor may include an RGB camera. For example, an RGB camera can acquire images of the biomarker throughout the procedure. In some arrangements, a single RGB camera may be provided. In other arrangements, multiple RGB cameras may be present. In other arrangements, the camera may be one or more near-infrared (NIR) or infrared (IR) cameras. Alternatively, multiple cameras may be combined, or multispectral and hyperspectral sensors may be used. The choice of camera type will depend on the nature of the biomarker being tracked. Having a single camera enables the generation of 2D images (unless the movement is for registration and creation of 3D volumes), while multiple cameras enable the generation of 3D images / models to be produced.

[0029] In addition, the head-mounted visual device may include a depth sensor. For example, this could be a time-of-flight sensor. Alternatively, it could be a stereo camera, a combination of LiDAR, RADAR, hyperspectral cameras, or any other known type of sensor capable of measuring distance and scale. The depth sensor is able to measure the distance from the head-mounted visual device to the patient. In this way, the position of the head-mounted visual device relative to the patient can be determined throughout the procedure, thereby improving the accuracy of positioning the virtual mapping on the display. For example, the depth sensor can estimate the position and / or orientation (pose) of the head-mounted visual device relative to the patient throughout the procedure. This can be achieved by generating a point cloud or depth image of the surgical site. This can help provide a transformation between patient coordinates and head-mounted visual device coordinates. The depth sensor can generate a depth map or point cloud of biomarkers. Because the position measured by the probe can be 3D data, the 3D position (e.g., feature) of the biomarker needs to be determined in order to determine its relationship to the position of the biomarker (e.g., feature). This can be achieved by obtaining a depth map or point cloud.

[0030] Continuously tracking the location of biomarkers throughout the medical procedure to obtain their current location may also include: capturing images of the patient via sensors; extracting one or more features from the images of the biomarkers; determining correspondences between one or more features across consecutive frames of the images; and using a transformation matrix to estimate the motion of one or more features across consecutive frames of the images. The transformation matrix has preferably been calculated and is based on sensor calibration parameters. The extraction of one or more features may be performed by applying an algorithm. This may be derived from the raw image data.

[0031] Patient images can be raw data, such as from fluorescence emission. Successive images over time can form a series of image frames. Depending on the nature of the raw image data obtained, the features can be 2D or 3D. Robust algorithms can be used to identify 2D / 3D features. For example, such identification can be based on the size, shape, and color intensity of biomarkers. The correspondence between features on consecutive frames of an image can be used to track features.

[0032] The transformation is calculated continuously throughout the procedure to allow for real-time tracking during surgery.

[0033] The method may also include applying an optimization procedure to remove noisy or incomplete data before calculating the transformation matrix. This may involve using global optimization. Alternatively or additionally, outlier removal algorithms or optimization filters may be used. For example, optimizing Random Sample Consensus (RANSAC) or the Iterative Closest Point algorithm. Advantageously, these techniques can help remove noise and speckle, thereby reducing false matches that may be caused by tracking bias due to noisy data, such as due to ambient lighting.

[0034] Estimating the motion of features across consecutive frames of an image using a transformation matrix may also include applying rigid or non-rigid registration. Since the patient's body is a dynamic object (e.g., due to respiratory movements or soft tissue deformation), displacement of 2D / 3D features may occur over several frames. By calculating and then using the transformation matrix, the motion of 2D / 3D features can be estimated across consecutive frames. Non-rigid registration and elastic deformation correction can be used to estimate the position of 2D / 3D features in real time. When performing non-rigid estimation, the region of interest may shrink or expand, resulting in dynamic raw data. For this, elastic deformation correction may be necessary. This allows for precise matching between image frames, even with changes and motion of the patient's body.

[0035] Due to the dynamic nature of the system, the shape of features can change with each frame, potentially leading to non-uniformity in the image. In some arrangements, image gradient operations on the 3D data can be used to determine the displacement between feature locations between frames. For this purpose, acceptance criteria and associated thresholds can be defined. Once the threshold is reached, the probe's coordinate system is registered with the patient's coordinate system.

[0036] As described above, one or more features of a biomarker may include the edges or intersections of the biomarker. Advantageously, these intersections or edges can provide easily identifiable reference points on the biomarker. This improves the accuracy of tracking the biomarker between frames. Alternatively, the entire structure of the biomarker can be tracked, rather than the features of the biomarker used. For example, if it is a skin marker (such as a birth marker), these features may be the shape of the biomarker.

[0037] Preferably, the biomarker may include the patient's blood vessels. Because the patient's blood vessels are distributed throughout the patient's body, they can be tracked at any location throughout the patient's body. Subsurface detection of blood vessels can be achieved by irradiating them with NIR light. Alternatively, the biomarker may be at least one of the patient's lymph nodes, the patient's nervous system, or the surface of the patient's organs.

[0038] Alternatively, other biomarkers may be used. These include skin markings, moles, birthmarks, joints, muscles, organs, or any other area of ​​the patient's tissue. Alternatively, biomarkers may include anatomical landmarks such as the corner of the eye or a notch above the sternum.

[0039] In some setups, multiple biomarkers can be used for tracking. For example, multiple blood vessels can be used, and each of the vascular features can be tracked. In other setups, various different types of biomarkers can be used. For example, vascular and skin markers. Increasing the number of biomarkers used can increase the accuracy of patient tracking.

[0040] The step of tracking the probe's position may include detecting a marker on the probe via a sensor and calculating the position of the probe tip representing the location where measurements are collected. In this way, the probe itself can be tracked using the position of the probe tip indicating the location where a measurement is being performed. Therefore, the location where a measurement is being performed can be determined with high precision. The relationship between the position of the probe tip and the marker is calculated based on the known probe geometry. A single marker or multiple markers may be present.

[0041] Probe markings can be visible references, such as the geometry on the probe. Alternatively, probe markings can be markings that actively emit electromagnetic radiation.

[0042] The probe markers can be detected by a tracking system on a head-mounted viewer. For example, one or more sensors can track the probe markers. This could be an electromagnetic sensor, for example, an infrared or RGB camera used to detect images of the markers. The sensors can detect images of manually marked probes, enabling image pattern recognition to identify the marker's location. This can be done using, for example, Aruco, April tags, or known patterns. In other arrangements, machine learning algorithms can be used to track the probe's position. Machine learning algorithms (e.g., AI) can be trained to track the probe's position.

[0043] Alternatively, the tracking system can be located outside the head-mounted viewer, thereby performing external tracking. In this arrangement, the position of the head-mounted viewer will be tracked in addition to the probe position.

[0044] In other arrangements, the probe's geometry can be tracked using multiple cameras in a stereo configuration. This is achieved by comparing images of the probe from each camera and by knowing the probe's geometry, enabling the tracking of the probe's position. In a further arrangement, a sensor located on the probe can be responsible for tracking its position. This can be estimated, for example, using a range estimation method. For instance, the sensor located on the probe could be an inertial measurement unit (IMU). Advantageously, using an IMU can improve tracking accuracy and also anchor the virtual mapping to a fixed position relative to the real world.

[0045] Alternatively, the probe can be tracked using an electromagnetic generator (e.g., a mat) placed on a table beneath the patient. In this arrangement, the probe and head-mounted viewer include magnetic markers placed thereon, and the AR system includes a device configured to generate an electromagnetic field. In this way, the magnetic field generated by the device detects the movement of these small magnetic units in the electromagnetic field, thereby tracking their movement to obtain their spatial positioning. In this arrangement, an RGB camera for tracking the probe may not be necessary. Preferably, the step of tracking the probe's position may include detecting changes in the magnetic field caused by the movement of the magnetic markers on the probe, wherein the probe's position is identified by the changes in the magnetic field.

[0046] Measurements can be used to indicate areas of cancerous tissue. Each measurement can include intensity, where the probe is positioned at the point where the associated measurement is acquired. Intensity can indicate whether the tissue being measured is cancerous. Intensity can be displayed in a virtual map to indicate the area of ​​cancerous tissue.

[0047] The relationship between the location of a biomarker (e.g., one or more features) and the location where measurements are collected can be determined by transforming the two locations in the same coordinate system. Specifically, the location of the biomarker's features and the location where measurements are collected are transformed into a common coordinate system. By registering the coordinate system of the measurements with the coordinate system of the biomarker at an example location during measurement, the relative position between the biomarker and the measurements can be easily determined in the same reference system, thereby establishing the relationship. In this way, each point where a feature or measurement is located is represented by a coordinate vector. Advantageously, through this registration, the measurements (or virtual mappings) are correctly aligned with the patient's body when viewed by the surgeon. This registration continues throughout the surgical procedure. This can involve transforming the coordinate system to a common coordinate system to ensure that the images of the virtual mappings and biomarkers can be compared with each other and thus correctly aligned on the display based on the determined relationship. This can be done by transforming the coordinates of the measurements in the head-mounted viewer's reference system to the coordinates in the biomarker's reference system, making the relative positions possible.

[0048] Preferably, the measurement is displayed at a location on a near-eye monitor such that, throughout the medical procedure, the measurement is aligned at that location relative to the patient in the surgeon's field of vision of the patient's tissues. This may further include tracking the position of the surgeon's eye and adjusting the position of the measurement on the near-eye monitor based on the current position of the surgeon's eye. Advantageously, this means that by taking the position of the surgeon's eye into account, the virtual mapping on the monitor can be more precise. The position can be tracked throughout the medical procedure to ensure real-time adjustments.

[0049] Throughout the medical procedure, the wearer's eye position will not be fixed, as their field of vision and gaze will vary. By continuously determining the wearer's eye position and their position relative to the head-mounted viewer, the accuracy of the generated image's localization within the wearer's field of vision can be improved. The eye-tracking sensor can also determine the wearer's eye focus, i.e., where they are focused at any given time. In this way, the virtual map can be displayed in a way that ensures it is always focused within the wearer's patient field of vision.

[0050] An eye-tracking sensor can consist of multiple eye-tracking sensors. For example, there might be a single sensor tracking each eye. Alternatively, a single tracking sensor can track the position of both eyes. An eye-tracking sensor may include an IR light source and a detector that scans the position of each eye to determine its location. The light source may be in the form of an LED or a laser. Alternatively or additionally, an eye-tracking sensor may be a potential-based eye-tracking sensor. A potential-based eye-tracking sensor uses electrodes placed around the eye to measure eye movement.

[0051] The processor can receive the position of the wearer's eyes. Preferably, the processor is further configured to obtain the position of the wearer's eyes by obtaining the interpupillary distance of the wearer's eyes.

[0052] The near-eye display can be a single display. Alternatively, the near-eye display can be two displays, one for displaying an image of one eye. The near-eye display can be a waveguide. Alternatively, the display can be a beam splitter display or a laser-reflective display. The display can project an image into the wearer's field of vision using mirrors. The display can be made of glass and / or plastic. In this way, it is transparent. Alternatively, the near-eye display can be a lens. Or, the near-eye display can be a beam of light that projects an image into the wearer's eye so that the image is displayed on the retina. In this way, the near-eye display can be a virtual retinal display.

[0053] The device can be further configured to perform a calibration process to calibrate the AR head-mounted viewer. The calibration procedure may involve obtaining the relative positions of the sensors on the head-mounted viewer. The display steps can be updated based on the calibration. The processor can determine the relative positions of the sensors and the display relative to each other. Sensor parameters can also be determined. For example, parameters of the pinhole camera model used by the algorithm, or parameters to correct distortions introduced by optics in front of the sensors, can be determined. Further calibration is performed to determine the surgeon's interpupillary distance (IPD) (i.e., the distance between their eyes). This can be obtained by measuring using an eye-tracking sensor. Alternatively, this can be manually entered into the software by the surgeon. The position of the surgeon's eyes relative to the display can also be determined. This calibration ensures accurate overlap between the probe measurements and the correct patient position from which the probe measurements are collected. The transformation matrix may include details of the obtained calibration parameters.

[0054] According to another aspect, a method is provided for obtaining measurements from a patient and displaying the measurements to a surgeon using an augmented reality (AR) system during a medical procedure, the method comprising the steps of: (a) receiving measurements collected from patient tissue, wherein the measurements are collected by probes located in different regions adjacent to the patient tissue; (b) tracking the position of the probes while the probes collect the measurements to determine the location where the measurements are collected; (c) simultaneously performing steps (a) and (b): tracking the position of a biomarker on the patient using at least one sensor such that a relationship between the position of the biomarker and the location where the measurements are collected can be determined; and (d) displaying the measurements at a location on a near-eye display of an AR head-mounted viewer worn by the surgeon such that, throughout the medical procedure, the measurements are aligned at that location relative to the patient in the surgeon's field of vision of the patient tissue: continuously tracking the position of the biomarker throughout the medical procedure to obtain the current position of the biomarker; and updating the position of the measurements on the near-eye display based on the current position of the biomarker.

[0055] Augmented reality systems in this regard can be augmented reality systems as described above.

[0056] According to another aspect, a computer program product is provided, the computer program product including instructions that, when executed by a computer, cause the computer to perform the following steps: (a) receiving measurements collected from patient tissue, wherein the measurements are collected by probes located in different regions adjacent to the patient tissue; (b) tracking the position of the probes while the probes collect the measurements to determine the location where the measurements are collected; (c) simultaneously performing steps (a) and (b): tracking the position of a biomarker on the patient using at least one sensor, such that a relationship between the position of the biomarker and the location where the measurements are collected can be determined; and (d) displaying the measurements at a location on a near-eye display of an AR head-mounted viewer worn by a surgeon, such that, throughout the medical procedure, the measurements are aligned at that location relative to the patient in the surgeon's field of vision of the patient tissue, by continuously tracking the position of the biomarker throughout the medical procedure to obtain the current position of the biomarker; and updating the position of the measurements on the near-eye display based on the current position of the biomarker.

[0057] According to another aspect, a non-transitory computer-readable medium is provided, which is configured to perform the above methods when executed on a processor. Attached Figure Description

[0058] Figure 1 A schematic diagram of an existing Raman system for analyzing the chemical composition of biological tissues is shown;

[0059] Figure 2 A schematic diagram of an example augmented reality (AR) head-mounted viewer according to an embodiment of the present invention is shown;

[0060] Figure 3 A schematic diagram of an example augmented reality (AR) system according to an embodiment of the present invention is shown for obtaining and visualizing measurement values;

[0061] Figure 4 It shows how to use Figure 3 The flowchart shown illustrates the steps performed by the example AR system to obtain measurements from the patient and accurately display those measurements in the user's field of vision; and

[0062] Figure 5 As shown in the example of use Figure 3 The AR system obtained in Figure 2 The example virtual map is shown in the AR head-mounted viewer. Detailed Implementation

[0063] Figure 1A prior Raman system 200 for analyzing the chemical composition of biological tissues, such as detecting and mapping tumors, is shown. System 200 includes a diode laser 204 attached to a Raman probe 202 via an optical fiber cable 206. The Raman probe 202 is also attached to a spectrometer 216 via an optical fiber cable 212. The spectrometer 216 may have a volume phase technology (VPT) grating coupled to a CCD camera 220. The CCD camera 220 communicates with a computer processing unit 222.

[0064] During the Raman spectroscopy procedure, probe 202 is positioned adjacent to the tissue 210 to be analyzed on patient 208, such that a low-power laser beam from laser 204 is incident on the area of ​​tissue 210 to be analyzed, as shown in... Figure 1 As can be seen, the light incident on the tissue causes tissue luminescence through Raman scattering, and the light is then detected by probe 202 and sent to spectrometer 216 (i.e., an instrument that detects and analyzes the incident light according to its wavelength and records the resulting spectrum).

[0065] The output data from the spectrometer 216 is processed by the CCD camera 220 and the computer processing unit 222. When cancerous tissue emits light within a certain spectrum, those values ​​compatible with the cancerous tissue can be transformed into a graphical representation (i.e., a virtual map). This virtual map is aligned with an image of the patient's tissue 208, thereby highlighting the area of ​​cancerous tissue. The image and the virtual map are then fused and displayed on a monitor set up in the operating room. The surgeon uses this visualization as a guide to locate the area of ​​cancerous tissue during surgery.

[0066] However, displaying images on a patient's monitor, far from the operating room, has drawbacks. The surgeon must match this information displayed on the monitor to the actual patient's body beforehand. This can lead to reduced surgical accuracy, as the matching relies on the surgeon's ability to apply the position shown on the monitor to the patient, making it susceptible to human error. Furthermore, this results in longer surgical times, as the surgeon must constantly leave the patient to look at the monitor.

[0067] Figure 2 An augmented reality (AR) system 100 according to an embodiment of the present invention is shown. The AR system 100 includes an AR head-mounted viewer 2 and a processor 12.

[0068] The augmented reality head-mounted viewer 2 has two displays: a first display 4a and a second display 4b. The first display 4a is used to display an image to the right eye of the wearer of the head-mounted viewer, while the second display 4b is used to display an image to the left eye of the wearer of the head-mounted viewer 2. The displays 4a and 4b are attached to the housing 16 of the head-mounted viewer 2.

[0069] Two cameras, 6a and 6b, are located on the housing 16 of the head-mounted viewer. Camera 6a is positioned above the first display 4a, while camera 6b is positioned above the second display 4b. Cameras 6a and 6b are capable of detecting near-infrared (NIR) light and RGB light (i.e., multispectral cameras). These are shown as two cameras; however, any number of cameras can be used. Additionally, a single camera can detect visible light and NIR light.

[0070] Light source 8 is also located on housing 16. Light source 8 is an NIR light source configured to emit NIR light. Light source 8 is located between cameras 6a and 6b, although the light source can be located anywhere on the AR headset 2. Alternatively, light source 8 can be located outside the AR headset 2 (or, for some types of setups, it may not be necessary at all).

[0071] Two depth sensors 10a and 10b are located on the housing 16 of the head-mounted viewer 2. The depth sensors are time-of-flight sensors configured to determine the depth from the head-mounted viewer 2 to an object, but any type of depth sensor can be used.

[0072] The head-mounted viewer 2 also includes an eye-tracking sensor 18. The eye-tracking sensor is located on the side of the head-mounted viewer facing the wearer's head. The eye-tracking sensor is configured to determine the position of the wearer's eyes on the head-mounted viewer 2. The eye-tracking sensor 18 can also determine which direction the wearer's eyes are looking.

[0073] The processor 12 is located externally to the AR head-mounted viewer 2. The processor can be a computer or the processor of another data processing device. The AR head-mounted viewer 2 is connected to the processor 12 via a cable 14. The cable 14 is used to transmit signals between the head-mounted viewer and the processor 12. For example, data obtained from cameras 6a, 6b, the eye-tracking sensor 18, and the depth sensors 10a, 10b can be transmitted to the processor 12 via the cable 14. The cable 14 is also used to transmit communication signals between the processor 12 and the head-mounted viewer 2 to control the cameras 6a, 6b, the depth sensors 10a, 10b, the light source 8, and the eye-tracking sensor 18 to perform their functions.

[0074] AR system 100 can be used in conjunction with a Raman probe (not shown) during surgical procedures, such as procedures to remove cancerous tissue. The AR system can be used to obtain measurements of the cancerous area in the patient's tissue and display these measurements on displays 4a, 4b within the surgeon's line of sight, thereby providing guidance during the surgeon's surgical procedure.

[0075] Figure 3 Another example augmented reality (AR) system 101 according to an embodiment of the present invention is shown. Figure 3The AR system 101 has an AR head-mounted viewer 3. The AR head-mounted viewer 3 has... Figure 2 The AR head-mounted viewer 2 has the same features as the AR head-mounted viewer 2, with a sensor suite 20 containing multiple cameras 6, light sources 8, and multiple distance sensors 10 (although...). Figure 3 (Multiple cameras and distance sensors are not explicitly shown in the image).

[0076] AR system 101 also includes a Raman probe 22. The Raman probe 22 has the features described above. Figure 1 The structure described is connected to the processor via a spectrometer (not shown). The probe consists of a measuring tip 26, which is the area of ​​the probe that collects measurements. The probe 22 also includes markers 24 that can be used to track its position relative to the patient and the AR head-mounted viewer 3.

[0077] although Figure 3 The processor 12 is not shown. The AR system 101 has a processor that communicates with the AR head-mounted viewer 3 and the probe 22 to perform processing steps.

[0078] Figure 3 The image shows a probe 22 in a region adjacent to patient tissue 28. Patient tissue 28 is the region from which measurements will be collected by probe 22, such as a suspected cancer site. Biomarkers 30 are located on patient tissue 28 and are used to extract features for tracking patient location, as will be described in detail below.

[0079] The use will now be described. Figure 3 This is a specific example method for an AR system. The example procedure involves obtaining measurements from the cerebral cortex, where biomarker 30 is a blood vessel on the brain surface that has accumulated fluorescent labels. However, it should be understood that this method can be applied to any part of the patient's body. In this example procedure, the patient is initially injected with a fluorescent dye, such as through an injection into the patient's venous system. The dye accumulates in the patient's blood vessels or tissues and can be used as biomarker 30 by means of fluorescence emission, such as... Figure 3 As shown.

[0080] During the surgical procedure, the surgeon wears an AR head-mounted viewer 3 and places a Raman probe 22 near the patient tissue 28 to be analyzed. When the probe 22 is in the line of sight of an RGB camera 6, the RGB camera 6 detects probe markers 24. The processor then implements a pattern recognition algorithm to estimate the pose (i.e., its position and orientation) of the markers 24. This can then be used to estimate the position of the probe tip 26, i.e., the location where Raman measurements are collected. Because the processor knows the geometry of the probe 22, a transformation considering the probe's geometry can be used to estimate the position of the probe tip 26 based on the position of the tracked probe markers. The head-mounted viewer coordinate system is then set. The designated position of the probe tip, matching the location of the cancerous tissue region, is within the head-mounted viewer coordinate system.

[0081] The surgeon places a Raman probe 22 next to the patient to acquire measurements from the patient's tissue. The collected measurements are analyzed, and those representing cancerous tissue can be stored. The measurements are stored along with the corresponding locations on the probe 22 via markers 24. These stored measurements can have associated intensity values ​​and location coordinates stored as (x, y, z). This allows for the generation of a 3D virtual map.

[0082] Simultaneously, the NIR light source 10 illuminates the patient tissue, causing the fluorescent dye injected into the patient to fluoresce. The injection site can be selected so that the dye accumulates in specific areas of the patient, such as specific blood vessels near the area measured by the probe of interest. These blood vessels can be used as biomarkers 30 for extracting features used to track the patient. The NIR camera 6 detects the fluorescence emitted from the blood vessels as a series of images. 2D or 3D features of the biomarkers are extracted from these raw data images. A patient coordinate system is set up and the 2D / 3D features are stored within it.

[0083] The relative positions of the feature and the probe measurement location can be calculated to determine the relationship between their positions. This relationship can be used throughout the procedure to update the position of a virtual map of the measurement results on the display by tracking the position of the feature. This can be achieved through a transformation between the patient's coordinate system and the head-mounted viewer's coordinate system.

[0084] During the tracking process, the location (i.e., motion of features) of features (biomarkers) on consecutive frames is estimated (as will be described in further detail below).

[0085] As outlined above, a virtual map is generated from probe measurements. This virtual map represents the cancerous area of ​​the tissue. Registration is then performed so that the patient coordinate system is mapped to the head-mounted viewer coordinate system. In this way, the virtual map can be displayed on the AR head-mounted viewer display at the correct position relative to the patient. This allows the surgeon to perform procedures using, for example, a virtual map representing cancerous tissue that is clearly aligned with the patient, thus serving as a guide superimposed on the surgeon's field of vision at the correct position. This is achieved throughout the procedure by continuously tracking the position of features and by continuously detecting NIR fluorescence signals from biomarkers (e.g., blood vessels), thereby forming a series of image frames.

[0086] Furthermore, eye tracking can be used to further orient the virtual mapping on the monitors to the correct field of vision for the surgeon. One or more eye-tracking sensors can track the coordinates of the patient's eyes. By transforming the coordinates of the surgeon's eyes into the coordinate system of the head-mounted viewer, the position of the virtual mapping can be updated on each monitor.

[0087] Figure 5 A virtual mapping 500, as shown in the example discussed above, is displayed when using... Figure 3 The AR system obtains an AR head-mounted viewer. This virtual mapping can be superimposed on displays 6a, 6b at the correct position relative to the surgeon's field of vision at the location where measurements are being taken. Figure 5 The virtual map 500 shown has three regions, each with a different intensity indicated by the color or shading of the mapped region. Intensity represents the intensity of a recorded measurement. For example, in the example described herein, higher intensity (i.e., darker areas) could represent a higher concentration of cancerous tissue. Region 501 has the lowest intensity and represents an area without cancerous tissue. Region 503, located within region 501, has a higher intensity than region 501 and represents a small amount of cancerous tissue. Region 505, located within region 503, has a higher intensity than both regions 503 and 501 and represents a large amount of cancerous tissue. For example, region 505 could represent a tumor site.

[0088] Figure 5 The virtual map shown is illustrated as a 2D map, which is how the surgeon would view the virtual map when displayed on monitors 4a and 4b. However, the virtual map can actually include a 3D map of measurements obtained from the tissue site, with a 2D projection of that 3D map displayed on the monitor. The virtual map displayed on each monitor 4a and 4b can be a different projection of that 3D map. This takes into account the different positions of the surgeon's eye.

[0089] Figure 4A flowchart of the above method 400 is shown, which is performed to obtain measurement values ​​from the patient and, as... Figure 3 The AR system shown accurately displays the measurement values ​​within the user's field of vision.

[0090] In step 401, a calibration procedure is performed to calibrate the AR head-mounted viewer. This may include calibrating the sensors on the head-mounted viewer to take into account their relationships with each other. The processor determines the relative positions of sensors 6, 8, 10 and display 4 relative to each other. Sensor parameters may also be determined. For example, parameters of the pinhole camera model used by the algorithm may be determined, or parameters to correct distortions introduced by optics in front of the sensors. Further calibration is performed to determine the surgeon's interpupillary distance (IPD) (i.e., the distance between the surgeon's eyes). This can be obtained by measuring using an eye-tracking sensor. Alternatively, this can be manually entered into the software by the surgeon. The position of the surgeon's eyes relative to the display may also be determined. This calibration ensures accurate superposition of probe measurements with the correct patient position collected. Eye-tracking related calibrations can be performed continuously throughout the use of the device.

[0091] In step 403, optional probe calibration is performed. This allows the probe geometry to be determined, enabling the position of the tip of probe 26 to be determined relative to the position of probe mark 24.

[0092] In step 405, the Raman probe is positioned adjacent to the patient tissue and measurements are acquired. The probe can determine measurements from areas of interest, such as areas detecting cancerous tissue. Measurements can be taken continuously by the surgeon simply placing the probe near the patient tissue, or by the surgeon activating measurement acquisition by pressing a button on the probe while pointing it at the area of ​​interest.

[0093] In step 407, the position of the probe is tracked to determine the location for measurement. As outlined in step 403, the position of the probe tip or measuring component 26 may be known relative to the probe mark 24. The RGB camera 6 on the head-mounted viewer 3 can capture an image of the probe mark 24. Coordinates can be assigned to the probe tip 26 within a coordinate system (head-mounted viewer coordinate system). This can be achieved through detection algorithms such as image pattern recognition. The coordinates are sent to a processor that creates a virtual map of the area highlighting the cancerous tissue.

[0094] In step 409, one or more biomarkers are tracked to determine their location at each time point during the surgical procedure. The use of biomarkers provides a method for tracking a patient throughout the surgical procedure without the need for physical markers.

[0095] Tracking step 409 may involve several sub-steps. As outlined above, biomarkers (such as blood vessels) can be tracked via biomarkers that emit fluorescent signals after contrast agent injection during or prior to the surgical procedure. An NIR light source 8 on the head-mounted viewer 3 illuminates the area of ​​patient tissue of interest at one or more wavelengths known to cause fluorescence. An NIR camera 6 captures the fluorescence emitted from the patient. The signal can then be filtered to remove unwanted noise. The raw data image formed by the captured NIR light is corrected or adjusted using calibration parameters obtained during calibration step 401.

[0096] The features of the biomarker are then detected (i.e., identified) so that they provide easily identifiable reference points for tracking across image frames. For example, features may include the edges or intersections of blood vessels. Robust algorithms are used to identify these features based on, for example, the size, shape, and color intensity of the biomarker. Features can be identified as 2D or 3D features. For example, if multiple NIR cameras are used, this allows for the determination of a 3D model for each frame. However, if only a single camera is used (and there is no depth sensor), 2D images can be used instead. Features are extracted from each frame. The location of the features is set within coordinate system 2 (CS2).

[0097] The relative position between the biomarker's features and the probe measurement location can be calculated to determine the relationship between their positions. This relationship can be used throughout the procedure to update the position of the virtual map on the display by tracking the position of the biomarker's features. This can be achieved by transforming between the patient coordinate system and the head-mounted viewer coordinate system (CS1), where the relative position between the patient and head-mounted viewer coordinate systems can be easily determined.

[0098] Then, features are tracked between consecutive frames to estimate motion. This is done by finding the correspondence between 2D / 3D features on consecutive frames taken at consecutive time points.

[0099] To estimate the motion of a biomarker feature across consecutive frames, a transformation matrix is ​​computed. This transformation can be either 2D or 3D, depending on whether the tracked feature is 2D or 3D. Individual transformations can be computed to estimate motion between each consecutive frame, thus providing frame-to-frame transformations. Because the patient's body is dynamic and exhibits constant motion due to respiratory movements and soft tissue deformation, biomarker features can shift across consecutive frames. To correct for this, non-rigid registration and elastic deformation correction can be used to estimate the biomarker's position (in 2D or 3D) in real time. (These are preferred when compared to rigid registration, as rigid registration presents problems associated with sensitivity, repeatability, robustness, or multiple views for dynamic objects.) Elastically deformable models can be generated to help maximize the accuracy of registration. Furthermore, due to this patient motion, the shape of the feature can also change across frames, which can lead to non-uniformity when comparing features. The displacement values ​​of the feature can be confirmed by image gradient manipulation of the 3D data. For this purpose, a threshold level is assigned based on an acceptance criterion. Once this threshold is reached, a virtual mapping is recorded in segment 411 as described below. For example, this could be when there is approximately 80% to 90% feature matching. The transformation matrix can be continuously estimated throughout the surgical procedure to provide ongoing tracking of the biomarker.

[0100] Before calculating the transformation matrix, optimization procedures can be applied to remove noisy or incomplete data. This may involve using global optimization. Alternatively or additionally, outlier removal algorithms or optimization filters can be used. For example, optimizing Random Sample Consensus (RANSAC) or the Iterative Closest Point algorithm. Advantageously, these techniques can help remove noise and speckle, thereby reducing false matches that may be caused by tracking biases originating from noisy data.

[0101] Steps 405 and 407 are performed simultaneously, thus correlating the measurements taken with the probe with the collected data coordinates. Steps 405 and 407 are also performed simultaneously with step 409, so that the position of the probe measurements can be tracked relative to the characteristics of the biomarker while measurements are being taken and throughout the remainder of the surgical procedure.

[0102] In step 411, registration is performed so that the coordinates measured by the probe can be mapped to the coordinates of the biomarker's features. This ensures that the virtual mapping is correctly aligned with the area of ​​the patient's body when viewed on a monitor.

[0103] In step 412, the position of the surgeon's eyes is tracked to determine their gaze relative to the display. The head-mounted viewer 3 may have one or more eye-tracking sensors located thereon capable of tracking the position of the surgeon's eyes. This is achieved by tracking a point on the surgeon's retina using the eye-tracking sensors. The user's IPD (interpupillary distance) can also be used in conjunction with the position of the surgeon's eyes to ensure accurate placement of the measurement on the display.

[0104] One or more sensors can be used to track the surgeon's eyes. Alternatively, a separate tracking sensor can track the position of each eye. The eye-tracking sensor may include an IR light source and a detector that scans the position of each eye to determine its location. The light source may be in the form of an LED or a laser. Alternatively or additionally, the eye-tracking sensor may be a potential-based eye-tracking sensor. A potential-based eye-tracking sensor uses electrodes placed around the eye to measure eye movement.

[0105] In step 414, probe measurements are displayed on a monitor, superimposed relative to the patient's captured position. The processor displays a virtual map on displays 4a and 4b. Each display 4a and 4b will show a virtual map correctly positioned relative to its associated surgeon's eye and the patient's field of vision. The position, orientation, and scaling of the virtual map are dynamically adjusted in real time based on the position of the AR head-mounted viewer 3 relative to the continuously tracked patient. The surgeon wearing the AR head-mounted viewer perceives the virtual map as overlaid on the patient's body, making it usable as visual guidance during surgery.

[0106] By repeatedly performing steps 409, 411, 412, and 414 through this procedure, the patient's movements throughout the procedure are taken into account, thereby ensuring that the measurements are superimposed on the display at the correct location.

[0107] Steps 405 and 407 can optionally be repeated to obtain additional measurements throughout the procedure.

[0108] To ensure correct orientation and real-time scaling of the virtual map, the position of the head-mounted viewer relative to the target must be continuously tracked during the procedure. As outlined above, this can be achieved by tracking the position of the patient's biomarkers throughout the procedure. However, tracking the distance between the head-mounted viewer and the patient can also be done throughout the surgical procedure. This can be achieved using one or more depth sensors (such as time-of-flight sensors) located on the head-mounted viewer. For example, depth sensors can enable the determination of the head-mounted viewer's position and / or orientation (pose) relative to the patient throughout the procedure. This can be achieved by generating point clouds or depth images of the surgical site. This allows the scale of the image on the display to be adjusted accordingly to account for the distance to the patient. This can help provide a transformation between patient coordinates and head-mounted viewer coordinates. For example, depth sensors can enable the determination of the head-mounted viewer's position and / or orientation (pose) relative to the patient throughout the procedure. This can add additional parameters to biomarker tracking by knowing the distance between the biomarker and the head-mounted viewer. Since probe data is available as 3D data, the use of point clouds or depth images allows for the acquisition of 3D data of biomarker features, which can then be compared to determine the relationship between the two positions.

[0109] A processor can be a device that executes instructions within a data processing apparatus. For example, instructions may be stored in memory. A processor may include one or more processing units for executing instructions (e.g., in a multi-core configuration). Instructions may execute within various operating systems on the data processing apparatus, such as UNIX, LINUX, Microsoft... More specifically, instructions can cause different data manipulations on data stored in memory (e.g., creating, reading, updating, and deleting programs). It should also be understood that different instructions can be executed during initialization when a computer-implemented method starts. Some operations may be required to perform one or more methods described herein, while others may be more general and / or specific to a particular programming language (e.g., C, C#, C++, Java, Python, or other suitable programming languages).

[0110] Many aspects of this disclosure have been described in detail, and it will be apparent that modifications and variations are possible without departing from the scope of the various aspects of this disclosure as defined in the appended claims. Since various changes can be made to the above constructions, products, and methods without departing from the scope of the various aspects of this disclosure, all content contained in the foregoing description and shown in the accompanying drawings should be interpreted as illustrative rather than restrictive.

[0111] In addition to the procedure described above, which uses the RGB camera 10 on the head-mounted viewer and a visible mark on the probe in step 407, alternative methods can be used to track the probe's position. For example, an infrared camera (or NIR) can be used instead for tracking. This can be achieved by the probe mark emitting an infrared signal detected by the probe. Alternatively, any type of electromagnetic transmitter and sensor can be used to track the probe's position. Alternatively or additionally, a custom-trained model based on machine learning can be used.

[0112] The probe tracking described above is described as being performed by sensors on the head-mounted viewer. Alternatively, sensors 6, 10 (e.g., cameras) may be positioned separately from the head-mounted viewer 3. In this arrangement, markings on the head-mounted viewer 3 are also tracked to ensure that the position of the head-mounted viewer 3 relative to an external tracking system is also known. In other arrangements, probe 22 may also or alternatively include one or more sensors that enable the probe to track its own position and transmit it to a processor. For example, probe 22 may include an inertial measurement unit (IMU) that enables the tracking of details of its position and / or orientation.

[0113] Alternatively, the probe can be tracked using an electromagnetic field generated beneath or near the patient and the operating environment. In this arrangement, the probe and head-mounted viewer include magnetic markers placed thereon. In this way, the magnetic field generated by the pad detects the movement of these small magnetic units in the electromagnetic field, thereby tracking their movement to obtain their spatial positioning. In this arrangement, an RGB camera for tracking the probe may not be necessary.

[0114] The sensors that track biomarkers can also be located outside the head-mounted viewer 3 in the same manner as described above. The light source 8 that illuminates the biomarkers to perform fluorescence can be located away from the AR head-mounted viewer 3, rather than specifically on the head-mounted viewer 3.

[0115] In some arrangements, the processor and its associated memory may be located on the AR headset. In other arrangements, the processor and memory may be positioned remotely from the AR headset. When located on the headset, the processor may be positioned at the back of the wearer's head. In other arrangements, the processor may be positioned within the housing of the headset (i.e., at the front of the wearer's head). However, any other positioning of the processor is also possible.

[0116] While the above description focuses on Raman spectroscopy, the systems and methods described above can also be used for other purposes. For example, the probe is not limited to a Raman probe. Alternatively, the probe can be a temperature probe capable of acquiring temperature measurements from the patient. The temperature measurements can be displayed as a virtual map on a monitor, located in the correct position within the surgeon's field of vision relative to the location where the temperature measurements were acquired. Probe 22 can be used to acquire measurements from any region of the patient's body. For example, this could be the cerebral cortex, where the biomarker is the pattern of blood vessels on the surface of the brain. Alternatively, the patient's liver can be observed, where the biomarker is the pattern of blood vessels near the liver. The surgical procedure described above is used for cancer surgery, i.e., for the removal of cancerous tissue. The virtual map helps guide the surgeon during this surgical procedure. However, it should be understood that the system described above can be used for any type of surgical procedure. In other arrangements, the probe can be an electromagnetic probe. For example, the electromagnetic probe can be an electromagnetic pulsating probe. Electromagnetic pulsating probes can be used to measure or assess the patient's brain function during invasive procedures. Alternatively, any other type of probe, such as a pH probe, can be used. In other arrangements, the probe can be used to place virtual markers on the user's face to create a virtual map that can be used later during surgery.

[0117] The term "surgeon" is used herein to refer to a person wearing the AR head-mounted viewer 3. However, as will be understood, the person does not have to be a surgeon and can be anyone using the device. The terms surgeon, wearer, and user are used interchangeably herein. The term "coordinate system" as used herein can be understood as implied coordinate system.

[0118] Image detection and associated action descriptions are performed by the camera. However, any type of image sensor / image sensing device can be used. The camera can be configured to detect still images or video.

[0119] The AR head-mounted viewer is shown in the accompanying drawings as having two displays attached to a housing with two arms (temples). However, it will be understood that the AR head-mounted viewer of the present invention is not limited to this arrangement, and it is conceivable to attach the head-mounted display (HMD) to any conventional device for the wearer. This could include using straps passing around and / or above the head to hold the head-mounted viewer in place. Alternatively, attachment devices, such as caps, that attach the device to the entire top of the head could be used. Furthermore, the specific arrangement of the housing shown is merely an example, and any type of housing could be used. In other arrangements, the AR system may include one or more glass displays that can be placed on the patient, instead of the AR head-mounted viewer. The methods described above are thus used to display measurements on the displays as described in the above embodiments.

[0120] The steps described herein, using eye tracking, provide increased accuracy in aligning the virtual map on the display. However, this is an optional step, and proper alignment can be achieved without using eye tracking. Furthermore, the eye-tracking sensor does not necessarily need to be located on the AR head-mounted viewer 3 and can be located within an external sensing system separate from the head-mounted viewer 3.

[0121] While this paper has described NIR cameras and NIR sources for performing fluorescence-based procedures, they are not necessarily limited to this wavelength range. Other wavelengths of light can be used. For example, any range of infrared wavelengths can be used. In other arrangements, visible light can be used. For example, IR light can be used to visualize blood vessels without injecting fluorescent dyes.

[0122] It will be understood that the type and number of cameras and sensors described above are not limiting. Single or multiple cameras and / or distance sensors can be used. Stereo camera systems and motion trackers with time-of-flight capabilities can be examples of such systems providing accurate positioning.

[0123] In the methods described above, when determining the position of probe 22 relative to biomarker 30, the characteristics of biomarker 30 can be used as the primary reference. However, it should be understood that, optionally, the probe position rather than the characteristics of biomarker 30 can be used as the primary reference point.

[0124] In other setups, artificial markers on the surgical table can be used instead of biomarkers. Besides biomarkers, this can also be used to improve tracking accuracy.

[0125] In the method described in this paper, biomarkers are described as being tracked. It is understood that this tracking can also be the tracking of target tissue (i.e., not blood vessels and other descriptive examples).

[0126] about Figure 2 and 3 The sensors described are merely examples of the types and numbers of sensors that can be utilized. In many arrangements, multiple sensors can be used to maximize the accuracy of probe and patient tracking.

[0127] As shown in the example above, there can be two displays 4a and 4b. However, in other arrangements, a single display for both eyes can be used. Alternatively, an AR head-mounted viewer may include a single near-eye display above a single eye, i.e., only 4a and / or 4b, instead of both.

[0128] While the above illustrates that data can be communicated between different components via one or more cables, it should be understood that the connection does not have to be wired. For example, any type of wireless communication device can be used to transmit signals from probes, head-mounted displays, and processors. This can include, for example, USB, WiFi, or Bluetooth.

[0129] In some setups, previously acquired medical images can also be included on the virtual map. This could be data from patient scans (such as CT or MRI scans). Furthermore, in addition to the virtual map, supplementary information from the field of view can be displayed, such as patient-specific information. This could be information from the patient's file. AR head-mounted viewers can also be configured to implement other features, such as having communication devices capable of providing audio or video signals to surgeons, enabling them to perform remote guidance procedures.

[0130] In other alternative arrangements, further applications are foreseeable and will allow for the registration of virtual mappings with patient-specific digital data (i.e., patient-specific 3D models of high-risk anatomy surrounding areas of cancerous tissue). This can be achieved in the following ways:

[0131] ■ A sensor for detecting biomarkers is integrated into an AR head-mounted viewer. Data collected by the sensor is used to create a patient-specific 3D model of the structure based on the biomarkers (e.g., biomarkers can identify high-risk anatomical structures to be displayed in addition to a virtual map). The system matches the 3D model coordinate system with the patient coordinate system for use in the procedure described above, i.e., displaying both the measurements that form the 3D model and the probe measurements.

[0132] ■ Physical markers are attached to the patient's body and the patient is scanned. A patient-specific 3D model is then obtained from the medical scan, and its position relative to the markers is input into the system. The 3D model coordinate system is matched with the patient's coordinate system. In this way, in addition to probe measurements, patient scan (such as MRI, CT, ultrasound) and / or camera measurements can be displayed.

Claims

1. An augmented reality system for obtaining and displaying measurements from a patient during a medical procedure, the system comprising: AR head-mounted viewer, the AR head-mounted viewer including a near-eye display, the near-eye display being used to display the measurements superimposed on the surgeon's field of vision of the patient's tissues at a location relative to the patient in which the measurements were collected; At least one sensor, the at least one sensor being used to track the patient's position; A memory for storing the measurement value and the location associated with the measurement value; as well as processor; The system is configured to perform the following steps during the medical procedure: (a) Receive measurements collected from the patient tissue, wherein the measurements are collected by probes located in different regions adjacent to the patient tissue; (b) Track the position of the probe while it collects the measurements to determine the location where the measurements are collected; (c) Perform steps (a) and (b) simultaneously: use the at least one sensor to track the location of the biomarker on the patient, such that the relationship between the location of the biomarker and the location where the measurement was collected can be determined; (d) The measurement is displayed on the near-eye monitor at a location such that, throughout the medical procedure, the measurement is aligned at that location relative to the patient in the surgeon's field of vision of the patient's tissues where the measurement was collected: The location of the biomarker is continuously tracked throughout the medical process to obtain its current location; The position of the measurement on the display is updated based on the current position of the biomarker.

2. The augmented reality system of claim 1 further includes creating a virtual mapping of the measured values, wherein, Displaying the measurement values ​​on the near-eye display includes displaying the virtual map.

3. The augmented reality system according to claim 1 or 2, wherein, Tracking the location of the biomarker on the patient further includes illuminating an area of ​​the patient with light of a specific wavelength, and detecting by the sensor a fluorescent signal emitted by the biomarker in response to the illumination.

4. The augmented reality system according to claim 1 further includes a probe.

5. The augmented reality system according to claim 4, wherein, The probe is a Raman probe.

6. The augmented reality system according to claim 1, wherein, The biomarker is one or more features of the biomarker.

7. The augmented reality system according to claim 6, wherein, Continuously tracking the location of the feature throughout the medical procedure to obtain the current location of the feature further includes: The sensor captures images of the patient; Extract one or more features of the biomarker from the image; Determine the correspondence between one or more features on consecutive frames of the image; Motion of one or more features on consecutive frames of the image is estimated by using a transformation matrix.

8. The augmented reality system according to claim 7, wherein, Estimating the motion of one or more features on consecutive frames of the image by using the transformation matrix further includes applying rigid or non-rigid registration.

9. The augmented reality system according to claim 6, wherein, The biomarker's one or more features include the biomarker's edges or intersections.

10. The augmented reality system according to claim 1, wherein, The biomarkers include at least one of the patient's blood vessels, the patient's lymph nodes, the patient's nervous system, and the surface of the patient's organs.

11. The augmented reality system according to claim 1, wherein, The steps for tracking the position of the probe include: The sensor detects a mark on the probe and calculates the position of the probe tip, the position of which indicates the location where the measurement value is collected; or The change in magnetic field caused by the movement of a magnetic marker on the probe is detected, wherein the change in magnetic field identifies the position of the probe.

12. The augmented reality system according to claim 1, wherein, The relationship between the location of the biomarker and the location where the measurement value was collected is determined by transforming the location of the biomarker and the location where the measurement value was collected into a common coordinate system.

13. The augmented reality system according to claim 1, wherein, Displaying the measurement value at a location on the near-eye monitor, such that throughout the medical procedure, the measurement value is aligned at that location relative to the patient in the surgeon's field of vision of the patient's tissues where the measurement value was collected, further comprising: The position of the surgeon's eye is tracked, and the position of the measurement on the near-eye display is adjusted based on the current position of the surgeon's eye.

14. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the following steps: (a) Receive measurements collected from patient tissue, wherein the measurements are collected by probes located in different regions adjacent to the patient tissue; (b) Track the position of the probe while it collects the measurements to determine the location where the measurements are collected; (c) Perform steps (a) and (b) simultaneously: use at least one sensor to track the location of the biomarker on the patient, such that the relationship between the location of the biomarker and the location where the measurement was collected can be determined; (d) The measurements are displayed at a location on a near-eye display of an AR head-mounted viewer worn by the surgeon, such that throughout the medical procedure, the measurements are aligned at that location relative to the patient in the surgeon's field of vision of the patient's tissues where the measurements were collected: The location of the biomarker is continuously tracked throughout the medical process to obtain its current location; The position of the measurement on the near-eye display is updated based on the current position of the biomarker.

Images