Augmented reality headsets and probes for medical imaging

Abstract

An augmented reality (AR) system and method are provided for obtaining and displaying measurements from a patient during a medical procedure. The system comprises an AR headset with a near-eye display for displaying measurements superimposed in a surgeon's field of view of the patient's tissue at a position relative to the patient where the measurements were collected, at least one sensor for tracking a position of the patient, a memory for storing the measurements and their associated positions, and a processor, the system being configured to perform the following steps during a medical procedure: a. receiving measurements collected from the patient's tissue, the measurements being collected by a probe positioned adjacent various regions of the patient's tissue; b. tracking the position of the probe while it is collecting the measurements to determine the positions at which the measurements were collected; c. simultaneously with performing steps a and b, tracking positions of biological landmarks on the patient using the at least one sensor so that a relationship between the positions of the biological landmarks and the positions at which the measurements were collected can be determined; and d. displaying the measurements at a position on the near-eye display such that the measurements are aligned in the surgeon's field of view of the patient's tissue at the position relative to the patient where the measurements were collected throughout the medical procedure by continually tracking the positions of the biological landmarks to obtain their current positions throughout the medical procedure and updating the positions of the measurements on the display based on the current positions of the biological landmarks.

Description

Technical Field

[0001] The present invention relates to an augmented reality system for use in medical treatment.

Background Art

[0002] Cancer surgery consists of removing tumors. To ensure complete removal, the surgeon removes a part of the healthy tissue around the tumor. Although complete removal is important to prevent tumor recurrence, it is accompanied by increased surgical costs and morbidity and mortality rates. To ensure complete removal, it is necessary to correctly identify and accurately map the tumor.

[0003] Current techniques for tumor identification and mapping include the following. · Biopsy, that is, obtaining a tissue sample using a needle for subsequent pathological analysis in a laboratory. · Patient scanning, that is, evaluating the margins of the tumor in the patient's medical images. · Thermal imaging, that is, detecting cancer using infrared thermal imaging. · Raman spectroscopy, which is a technique capable of analyzing the chemical composition of biological tissues and has been widely used for cancer screening, diagnosis, and intraoperative surgical guidance over the past decade.

[0004] Raman spectroscopy for tumor identification and mapping is a powerful technique because it allows surgeons to distinguish between healthy and cancerous tissue during surgery while being less invasive than biopsy. However, known Raman techniques have significant limitations because the area of ​​tissue analyzed at one time is very small (about 1 mm), requiring surgeons to perform several measurements with a probe to obtain a complete map of the cancerous tissue. This means that surgeons must either recall multiple locations of cancerous tissue determined by probe measurements during surgery, or mark the detected areas of cancerous tissue using physical markers such as stickers. The first method leads to reduced accuracy, while the latter increases surgical time because surgeons must stop and position physical markers correctly.

[0005] A further solution to this problem is to observe the location of cancerous tissue on an image of the patient's body displayed on a monitor in the operating room. However, the surgeon must first compare this information with the actual patient's body. This can lead to a decrease in surgical accuracy because this comparison is prone to human error, depending on the surgeon's ability to match the location shown on the monitor to the patient. In addition, this can prolong the surgical time because the surgeon has to constantly look away from the patient to look at the monitor. [Overview of the project] [Problems that the invention aims to solve]

[0006] The present invention aims to solve the aforementioned problems associated with current treatments. [Means for solving the problem]

[0007] According to the first embodiment, an augmented reality (AR) system is provided for acquiring and displaying measurements from a patient during a medical procedure, the system comprising: an AR headset having a near-eye display for displaying measurements superimposed on the surgeon's field of view of the patient's tissue at the location relative to the patient from which the measurements were collected; at least one sensor for tracking the patient's location; a memory for storing measurements and their associated locations; and a processor, the system during a medical procedure comprising the following steps: (a) receiving measurements collected from the patient's tissue, wherein the measurements are collected by a probe positioned adjacent to various areas of the patient's tissue; (b) tracking the position of the probe while it is collecting measurements to determine the location from which the measurements were collected; and (c) The system is configured to perform steps (a) and (b) simultaneously with the step of using at least one sensor to track the location of a patient's biological landmarks so that a relationship can be determined between the location of the biological landmarks and the location where measurements are collected; and (d) displaying measurements at a location on a near-eye display, wherein throughout the medical procedure, the system continuously tracks the location of the biological landmarks to obtain their current location and updates the location of measurements on the display based on the current location of the biological landmarks so that throughout the medical procedure, the measurements are aligned in the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurements were collected.

[0008] In this way, probe measurements are obtained simultaneously with the location of biological landmarks, allowing for the determination of the relationship between their relative positions. Therefore, by continuously tracking the location of biological landmarks, the measurements can be overlaid on the display at a later point in time in the correct position relative to the location obtained from the patient's body.

[0009] Because biological landmarks are used for tracking, there is no need to place physical markers on or around the patient, resulting in a marker-free procedure. This is advantageous because biological landmarks will not detach from the patient as physically attached markers do. In addition, surgeons do not need to stop to attach markers to the patient, which may shorten surgical time or reduce the harm and risk to the patient.

[0010] Because the patient's body is tracked in real time, the surgeon can keep the patient in their line of sight and directly view the measurements on the augmented reality display. In other words, the measurements are superimposed onto the patient's real-world view at their corresponding locations. This means that the surgeon does not need to constantly avert their eyes from the patient's body to look at an external monitor. This also reduces the errors that come with the subjective translation of what the surgeon sees on the monitor onto the patient's actual body.

[0011] The step of updating the location of measurements on the display based on the current location of a biological landmark involves using the relationship between the location of the biological landmark and the location where the measurements were collected.

[0012] The location where measurements are collected is the location of the patient from which each measurement is collected. In other words, it is the point on the patient where the probe performed the measurement. This relationship may also be the relationship between the location of a biological landmark and the location where each measurement is performed, which is determined by tracking the location where each measurement is performed. The term "adjacent to patient tissue" may be interpreted to mean adjacent to, on, or within patient tissue, depending on the nature of the probe and the measurement performed.

[0013] Preferably, the biological landmark is one or more features of a biological landmark. Thus, tracking is tracking one or more features of a biological landmark. For example, this tracking may include using at least one sensor to track the location of one or more features of a biological landmark on the patient so that a relationship can be determined between the location of one or more features of the biological landmark and the location where measurements are collected. The method may further include the steps of continuously tracking the location of one or more features of a biological landmark throughout the medical procedure to obtain its current location, and updating the location of measurements on a display based on the current location of one or more features of the biological landmark.

[0014] One or more features of a biological landmark may include edges or intersections of the biological landmark. These may also be elements of the biological landmark that can be identified. Advantageously, the intersections or edges can provide easily identifiable reference points on the biological landmark. These features may be extracted from raw data acquired by a sensor.

[0015] The sensor can track the patient's position (i.e., location and orientation). The current position is the position of the biological landmark at the moment, due to the fact that the patient's position may change throughout the procedure due to the patient's movement and the movement of the AR headsets relative to each other.

[0016] The method may further include a step of creating a virtual map of measurements, and the step of displaying the measurements on a near-eye display may include a step of displaying the virtual map. The virtual map can provide the surgeon with a visual guide regarding the cancerous area of ​​the patient's tissue. The virtual map may also be a spatial map of measurements that provides the surgeon with an intuitive and easily interpretable view.

[0017] The virtual map may be a color-coded data map in which areas of different measurements are indicated by different colors. For example, cancerous tissue may be shown in a different color than areas of non-cancerous tissue. Alternatively, or in addition, the amount of cancerous tissue can be indicated by the color of the virtual map. Areas with a high percentage of cancerous tissue may be colored differently from areas with a low percentage of cancerous tissue.

[0018] Preferably, the step of tracking the location of a biological landmark on a patient may further include illuminating a region of the patient using light of a specific wavelength and detecting a fluorescent signal emitted by the biological landmark in response to the illumination using a sensor. Before surgery, a fluorescent dye may be injected into the patient's venous system. The dye can then be accumulated at the biological landmark of interest. For example, the biological landmark may be a blood vessel of the patient. Advantageously, by using a fluorescent dye, it is possible to track internal regions of the patient located within the patient's body. Fluorescent dyes are advantageous because they allow the tracking point to be brought as close as possible to the point of interest. In addition, fluorescent dyes do not require the surgically invasive procedure of inserting a physical marker near the detection site. Since the patient's body is not fixed in place and moves throughout the procedure, for example when breathing, tracking a biological landmark means the marker moves with the patient's body, which improves tracking and thus improves the overlap of measurements on the display.

[0019] The fluorescent marker may be any known fluorescent dye or contrast agent. For example, the fluorescent marker may be indocyanine green (ICG). Illuminating the patient may involve using a near-infrared (NIR) light source. The light source excites molecular ligands in the biological landmark, and in response, the molecular ligands emit light of a specific wavelength. The light can be detected using a camera sensitive to this range of light spectrum. The detected light may be in the form of raw data that can be used to form an image that clearly shows the biological landmark. The wavelength of the NIR may be selected depending on the dye used. The fluorescent signal is used for location tracking purposes, rather than indicating a specific medical measurement itself. The NIR camera and NIR light source may be placed on an AR headset. In other configurations, the NIR camera and / or NIR light source may be placed separately from the AR headset. Alternatively, any other type of fluorescent marker can be used, such as methylene blue, alanine blue, and other fluorescent markers sensitive to light in the visible spectrum. The camera and light source may be selected based on the type of contrast agent used, whether it is a visible wavelength, an NIR wavelength, or an IR wavelength.

[0020] The AR system may further include a probe. Preferably, the probe is a Raman probe. Advantageously, Raman spectroscopy provides a non-invasive procedure for identifying and mapping cancerous tissue. By presenting these measurements on a near-eye display, the surgeon can view the measurements directly in their line of sight and in the correct position relative to the patient. The probe may also be a handheld probe. For example, the probe may be a handheld tool for collecting measurements from the patient. A Raman probe may comprise a diode laser for exciting the tissue and a sensor for detecting Raman scattering measurements from the tissue. However, other configurations of Raman probes may be used.

[0021] Alternatively, the probe may be a temperature probe capable of obtaining temperature measurements from the patient. The temperature measurements may be displayed on a screen as a virtual map in the correct position within the surgeon's field of view relative to where the measurements were obtained. In other configurations, the probe may be an electromagnetic probe. For example, the electromagnetic probe may be an electromagnetic pulsation probe. Electromagnetic pulsation probes may be used to take measurements from or to deliver impulses to the patient's brain during invasive procedures. Alternatively, any other type of probe, such as a PH probe, may be used.

[0022] If the AR system includes a probe, the AR system may further include the step of using the probe to collect measurements. However, it will also be understood that the probe can instead be considered as separate from the AR system.

[0023] The medical procedure may be a surgical procedure. Specifically, it may be a cancer surgery. This may be a surgery to remove cancerous tissue, such as a tumor, from a patient. The present invention makes it possible to collect measurements during a surgical procedure and at the same time to visualize the measurements throughout the procedure. Alternatively, in other configurations, the measurements may be collected before the surgical procedure and displayed later during the surgical procedure. The measurements can serve as a guide for the surgeon throughout the surgery.

[0024] Preferably, at least one sensor may include an RGB camera. For example, an RGB camera can acquire images of biological landmarks throughout the procedure. In some configurations, a single RGB camera may be provided. In other configurations, there may be multiple RGB cameras. In other configurations, the camera may be one or more near-infrared (NIR) or infrared (IR) cameras. Alternatively, multiple cameras can be combined, or multispectral and hyperspectral sensors can be used. The choice of camera properties depends on the nature of the biological landmark being tracked. A single camera can produce 2D images (unless moved to align and create a 3D volume), while multiple cameras can produce 3D images / models.

[0025] In addition, the headset may be equipped with a depth sensor. For example, the depth sensor may be a time-of-flight sensor. Alternatively, the depth sensor may be a stereo camera, LIDAR, RADAR, hyperspectral camera, or a combination of sensors, or other known types of sensors capable of measuring distance and scale. The depth sensor can measure the distance from the headset to the patient. In this way, the relative position of the headset to the patient can be determined throughout the procedure, thereby improving the accuracy of positioning on the virtual map on the display. For example, the depth sensor can estimate the relative position and / or orientation (posture) of the headset to the patient, which should be determined throughout the procedure. This may be done by generating a point cloud or depth image of the surgical site. This may be useful for performing a conversion between patient coordinates and headset coordinates. The depth sensor can generate a depth map or point cloud of biological landmarks. Since the probe's measurement position may be 3D data, it is necessary to determine the 3D position of the biological landmark (e.g., a feature) in order to determine its relationship to the position of the biological landmark (e.g., a feature). This determination can be achieved by acquiring a depth map or point cloud.

[0026] The step of continuously tracking the position of a biological landmark throughout a medical procedure to obtain its current position can further include the steps of capturing an image of the patient by a sensor, extracting one or more features from the image of the biological landmark, determining a correspondence relationship between one or more features over successive frames of the image, and estimating the movement of one or more features over successive frames of the image using a transformation matrix. The transformation matrix is preferably already calculated and is based on the calibration parameters of the sensor. The step of extracting one or more features may be by applying an algorithm for extracting said features. This extraction may be from raw image data.

[0027] The image of the patient may be raw data such as light emitted from fluorescence. Successive images over time can form a series of image frames. The features may be 2D or 3D features depending on the nature of the raw image data obtained. Robust algorithms can be used to identify 2D / 3D features. This identification may be based on, for example, the size, shape and color intensity of the biological landmark. The features can be tracked using the correspondence relationship between features over successive frames of the image.

[0028] To enable real-time tracking during surgery, continuous calculation of the transformation is performed throughout the procedure.

[0029] This method may further include applying optimization procedures to remove noisy or incomplete data before calculating the transformation matrix. This may include using global optimization. Alternatively, or in addition, outlier exclusion algorithms or optimization filters may be used, such as optimized random sample consensus (RANSAC) or iterative nearest neighbor algorithms. Advantageously, these techniques help remove noise and speckle, thus reducing mismatches that may be caused by tracking drift resulting from noisy data, such as that caused by ambient lighting.

[0030] The step of estimating feature motion across consecutive frames of an image by using a transformation matrix may further include the step of applying rigid or non-rigid registration. Because a patient's body is a dynamic object (e.g., due to respiratory movements or soft tissue deformation), displacement of 2D / 3D features may occur over several frames. By calculating and using a transformation matrix, the motion of 2D / 3D features can be estimated across consecutive frames. Using non-rigid registration and elastic deformation correction, the position of 2D / 3D features can be estimated in real time. While non-rigid estimation is being performed, the region of interest may contract or expand, resulting in dynamic raw data. Therefore, elastic deformation correction may be necessary. This correction enables accurate matching between image frames despite changes and movements of the patient's body.

[0031] Due to the dynamic nature of the system, the shape of features may also change across frames, resulting in non-uniform images. In some configurations, image gradient calculations can be used on 3D data to determine the displacement of feature positions between frames. For this purpose, tolerance criteria and associated thresholds may be defined. When the threshold is reached, the probe's coordinate system is aligned with the patient's coordinate system.

[0032] As outlined above, one or more features of a biological landmark may include edges or intersections of the biological landmark. Advantageously, intersections or edges can provide easily identifiable reference points on the biological landmark, thereby improving the accuracy of landmark tracking between frames. Alternatively, instead of using features of the biological landmark, the entire structure of the biological landmark may be tracked. For example, if the biological landmark is a skin scar such as a nevus, the features may be the shape of the biological landmark.

[0033] Preferably, the biological landmark may include the patient's blood vessels. Since the patient's blood vessels are located throughout the patient's body, tracking can be performed anywhere throughout the patient's body. The blood vessels may be illuminated by NIR light to enable subsurface detection of the vessels. Alternatively, the biological landmark may be at least one of the patient's lymph nodes, the patient's nervous system, or the surface of the patient's organs.

[0034] Alternatively, other biological landmarks may be used, such as skin scars, moles, nevi, joints, muscles, organs, or any other area of ​​the patient's tissue. Alternatively, the biological landmarks may include anatomical landmarks such as the outer corner of the eye or a suprasternal notch.

[0035] In some configurations, multiple biological landmarks may be used for tracking. For example, multiple blood vessels may be used, and each of the vascular features may be tracked. In other configurations, multiple different types of biological landmarks may be used, for example, both blood vessels and skin marks. Increasing the number of biological landmarks used can improve the accuracy of patient tracking.

[0036] The step of tracking the probe's position may include detecting a marker on the probe via a sensor and calculating the position of the probe tip representing the location where the measurement was collected. In this way, the probe itself can be tracked by the position of the probe tip indicating the location where the measurement was taken. Therefore, the location where the measurement was taken can be determined with high accuracy. The relationship between the probe tip position and the marker is calculated according to the known geometric shape of the probe. There may be a single marker or multiple markers.

[0037] The probe marker may be a visible criterion, such as a geometric shape on the probe. Alternatively, the probe marker may be a marker that actively emits electromagnetic radiation.

[0038] The probe marker may be detected by a tracking system on the headset. For example, one or more sensors may track the probe marker. For example, this may be an electromagnetic sensor. The sensor may be an infrared camera or an RGB camera for detecting images of the marker. The sensor can detect images of the probe artificial marker and thereby perform image pattern recognition to identify the marker's position. This may involve using, for example, Aruco, April tags, or known patterns. In other configurations, a machine learning algorithm may be used to track the probe's position. The machine learning algorithm (e.g., AI) may be trained to track the probe's position.

[0039] Alternatively, the tracking system may be located outside the headset, enabling external tracking. In this configuration, the headset's position is tracked in addition to the probe's position.

[0040] In other configurations, the geometric shape of the probe may be tracked by using multiple cameras in a stereo configuration. This allows the probe's position to be tracked by comparing images of the probe from each camera and by knowing the probe's geometric shape. In further configurations, a sensor placed on the probe can play a role in tracking the probe's position. For example, the probe position may be estimated using odometry. For example, the sensor placed on the probe may be an inertial measurement unit (IMU). Advantageously, using an IMU can improve the accuracy of tracking and allow the virtual map to be fixed in a fixed position relative to the real world.

[0041] Alternatively, the probe may be tracked using an electromagnetic generator (e.g., a mat) placed on a table beneath the patient. In this configuration, the probe and headset include magnetic markers placed on them, and the AR system includes a device configured to generate an electromagnetic field. In this way, the magnetic field generated by the device detects the movement of these small magnetic units within the electromagnetic field, thereby tracking their movement and obtaining their spatial position. In this configuration, an RGB camera used to track the probe may not be necessary. Preferably, the step of tracking the probe's position may include detecting a change in the magnetic field caused by the movement of magnetic markers on the probe, and the change in the magnetic field identifies the probe's position.

[0042] The measurements may be used to indicate areas of cancerous tissue. Each measurement may include an intensity, and the probe's position at the point where the measurement is obtained is associated with that intensity. The intensity can indicate that the tissue from which the measurement is obtained is cancerous. By displaying this intensity on a virtual map, the areas of cancerous tissue can be shown.

[0043] The relationship between the location of a biological landmark (e.g., one or more features) and the location where measurements are collected can be determined by transforming both locations into the same coordinate system. Specifically, this involves transforming the location of the biological landmark features and the location where measurements are collected into a common coordinate system. By aligning the coordinate system of the measurements with the coordinate system of the biological landmark at the time the measurements were acquired, the relative position between the biological landmark and the measurements can be easily determined on the same reference frame, thereby enabling the determination of the relationship. In this way, each point where a feature or measurement exists is represented by a coordinate vector. Advantageously, this alignment ensures that the measurements (or virtual map) are correctly aligned with the patient's body when viewed by the surgeon. This alignment is performed continuously throughout the surgery. This may include transforming the coordinate systems into a common coordinate system to ensure that the virtual map and the images of the biological landmarks can be compared with each other and thereby accurately aligned on the display based on the determined relationship. This transformation may be by transforming the coordinates of the measurements in the headset reference frame to the coordinates of the biological landmark reference frame, so that the relative position can be determined.

[0044] Preferably, the step of displaying a measurement taken at a certain location on a near-eye display, such that throughout the medical procedure the measurement is aligned within the surgeon's field of view of the patient's tissue at the location where the measurement was collected, can further include the steps of tracking the surgeon's eye position and adjusting the position of the measurement on the near-eye display based on the surgeon's eye position. Advantageously, this means that the positioning of the virtual map on the display can be more accurate by taking the surgeon's eye position into account. Tracking its position throughout the medical procedure ensures real-time adjustments.

[0045] Throughout the medical procedure, the wearer's eye position is not fixed, and their field of view and line of sight change throughout the procedure. The accuracy of positioning the generated image within the wearer's field of view can be improved by continuously determining the wearer's eye position and the relative position of the eyes to the headset. The eye-tracking sensor can also determine the wearer's eye focus, i.e., the position the wearer is focusing on at any given time. In this way, the virtual map can be displayed so that it is always in focus on the wearer's field of view relative to the patient.

[0046] The eye-tracking sensor may consist of multiple sensors. For example, there may be a single sensor tracking each eye. Alternatively, a single tracking sensor may track the positions of both eyes. The eye-tracking sensor may comprise an IR light source and a detector that scans the position of each eye to determine its position. The light source may be in the form of an LED or a laser. Alternatively, or in addition, the eye-tracking sensor may be an electroglomerate eye-tracking sensor. An electroglomerate eye-tracking sensor measures eye movement using electrodes placed around the eyes.

[0047] The processor can receive the wearer's eye position. Preferably, the processor is further configured to obtain the wearer's eye position by obtaining the interpupillary distance of the wearer's eyes.

[0048] The near-eye display may be a single display. Alternatively, the near-eye display may consist of two displays, one displaying an image to each eye. The near-eye display may be a waveguide. Alternatively, the display may be a beam-splitter display or a laser-reflecting display. The display may use mirrors to project an image into the wearer's field of view. The display may be made of glass and / or plastic. Thus, the display is transparent. The near-eye display may also be a lens. Alternatively, the near-eye display may be a beam that projects an image onto the wearer's eye so that the image is displayed on the retina. Thus, the near-eye display may be a virtual retinal display.

[0049] The device may be further configured to perform calibration procedures for calibrating the AR headset. Calibration procedures may include obtaining the relative positions of sensors on the headset. Display steps may be updated based on the calibration. The processor can determine the relative positions of the sensors and the display. Sensor parameters can also be determined. For example, parameters for the pinhole camera model used by the algorithm, or parameters for correcting distortions caused by the optical system in front of the sensor can be determined. Further calibration is also performed to determine the surgeon's interpupillary distance (IPD) (i.e., the distance between the surgeon's eyes). This may be obtained by taking measurements using an eye-tracking sensor to determine the IPD. Alternatively, the IPD may be manually entered into the software by the surgeon. The position of the surgeon's eyes relative to the display can also be determined. This calibration ensures accurate superposition of the obtained probe measurements with the correct position of the patient from which the probe measurements were collected. The transformation matrix may include details of the obtained calibration parameters.

[0050] In a further embodiment, a method is provided for using an augmented reality (AR) system to acquire and display measurements from a patient to a surgeon during a medical procedure, the method comprising: (a) receiving measurements collected from the patient's tissue, the measurements being collected by probes positioned adjacent to various areas of the patient's tissue; (b) tracking the position of the probes while they are collecting measurements to determine the location where the measurements are being collected; (c) simultaneously performing steps (a) and (b) using at least one sensor to track the location of a biological landmark on the patient to determine the relationship between the location of the biological landmark on the patient and the location where the measurements are being collected; and (d) displaying the measurements at a location on a near-eye display of an AR headset worn by the surgeon, the method comprising continuously tracking the location of the biological landmark throughout the medical procedure to acquire its current location and updating the position of the measurements on the near-eye display based on the current location of the biological landmark, so that throughout the medical procedure, the measurements are displayed so that they are aligned in the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurements were collected.

[0051] The augmented reality system of this embodiment may also be the augmented reality system of the above embodiment.

[0052] In a further embodiment, a computer program product is provided which, when the program is executed by a computer, includes instructions to cause the computer to perform the following steps: (a) receiving measurements collected from a patient's tissue, the measurements being collected by a probe positioned adjacent to various areas of the patient's tissue; (b) tracking the position of the probe while it is collecting measurements to determine the location where the measurements are collected; (c) simultaneously performing steps (a) and (b) using at least one sensor to track the position of a biological landmark on the patient to determine the relationship between the position of the biological landmark and the location where the measurements are collected; and (d) displaying the measurements at a certain location on a near-eye display of an AR headset worn by a surgeon, the display being such that throughout the medical procedure, the measurements are aligned in the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurements were collected, by continuously tracking the position of the biological landmark throughout the medical procedure to obtain its current position and updating the position of the measurements on the near-eye display based on the current position of the biological landmark.

[0053] In a further embodiment, a non-transient computer-readable medium is provided that, when executed on a processor, is configured to perform the method described above. [Brief explanation of the drawing]

[0054] [Figure 1] This is a schematic diagram of a conventional Raman system used to analyze the chemical composition of biological tissues. [Figure 2] This is a schematic diagram of an exemplary augmented reality (AR) headset according to an embodiment of the present invention. [Figure 3] This is a schematic diagram of an exemplary augmented reality (AR) system according to one embodiment of the present invention for acquiring and visualizing measured values. [Figure 4]Figure 3 shows a flowchart illustrating the steps taken to acquire measurements from a patient and accurately display them in the user's field of view using an exemplary AR system. [Figure 5] This is an example virtual map displayed on the AR headset in Figure 2, acquired using the AR system in Figure 3. [Modes for carrying out the invention]

[0055] Figure 1 shows a conventional Raman system 200 for analyzing the chemical composition of biological tissues, such as for detecting and mapping tumors. The system 200 comprises a diode laser 204 attached to a Raman probe 202 via an optical fiber cable 206. The Raman probe 202 is also attached to a spectrometer 216 via an optical fiber cable 212. The spectrometer 216 may have a volume phase technology (VPT) grating coupled to a CCD camera 220. The CCD camera 220 communicates with a computer processing unit 222.

[0056] During the Raman spectroscopy procedure, the probe 202 is positioned adjacent to the tissue 210 of the patient 208 being analyzed, as shown in Figure 1, so that a low-power laser beam from the laser 204 is incident on the area of ​​tissue 210 being analyzed. This light incident on the tissue causes it to emit light through Raman scattering, which is then detected by the probe 202 and sent to the spectrometer 216 (i.e., a device that detects and analyzes incident light according to its wavelength and records the resulting spectrum).

[0057] The output data from the spectrometer 216 is processed by the CCD camera 220 and the computer processing unit 222. Because cancerous tissue emits light within a specific spectrum, values ​​that match the cancerous tissue can be converted into a graphic representation (i.e., a virtual map). This virtual map can be aligned with an image of the patient's tissue 208, so that the areas of cancerous tissue are highlighted. The images are merged with the virtual map and then displayed on a monitor set up in the operating room. The surgeon uses this visualization as guidance to locate areas of cancerous tissue during surgery.

[0058] However, displaying images on a monitor far from the patient in the operating room has drawbacks. The surgeon must compare this information displayed on the monitor with the actual patient's body in front of them. This can lead to a decrease in surgical accuracy because this comparison is prone to human error, depending on the surgeon's ability to match the positions shown on the monitor to the patient. In addition, this can prolong surgical time because the surgeon has to constantly look away from the patient to look at the monitor.

[0059] Figure 2 shows an augmented reality (AR) system 100 according to one embodiment of the present invention. The AR system 100 includes an AR headset 2 and a processor 12.

[0060] The augmented reality headset 2 has two displays, a first display 4a and a second display 4b. The first display 4a is for displaying an image to the wearer's right eye, and the second display 4b is for displaying an image to the wearer's left eye. Displays 4a and 4b are mounted on the housing 16 of the headset 2.

[0061] The headset housing 16 houses two cameras 6a and 6b. Camera 6a is positioned above the first display 4a, and camera 6b is positioned above the second display 4b. Cameras 6a and 6b are capable of detecting near-infrared (NIR) and RGB light (i.e., multispectral cameras). Although shown as two cameras, any number of cameras may be used. In addition, visible light and NIR light may be detected by separate cameras.

[0062] Furthermore, a light source 8 is located in the housing 16. The light source 8 is an NIR light source that emits NIR light. While the light source 8 is located between cameras 6a and 6b, it can be placed at any position on the AR headset 2. Alternatively, the light source 8 may be located outside the AR headset 2 (or, in certain types of configurations, they may not be necessary at all).

[0063] Two depth sensors 10a and 10b are located on the housing 16 of the headset 2. The depth sensors are time-of-flight sensors configured to determine the depth from the headset 2 to an object, but any type of depth sensor may be used.

[0064] The headset 2 further includes an eye-tracking sensor 18. The eye-tracking sensor is located on the side of the headset facing the wearer's head. The eye-tracking sensor is configured to determine the position of the wearer's eyes in the headset 2. The eye-tracking sensor 18 can also determine which direction the wearer's eyes are looking.

[0065] The processor 12 is located outside the AR headset 2. The processor may be a computer or another data processing unit. The AR headset 2 is connected to the processor 12 via a cable 14. The cable 14 is for transmitting signals between the headset and the processor 12. For example, data acquired from cameras 6a, 6b, eye-tracking sensors 18, and depth sensors 10a, 10b may be transmitted to the processor 12 via the cable 14. The cable 14 also transmits communication signals between the processor 12 and the headset 2 to control the cameras 6a, 6b, depth sensors 10a, 10b, light source 8, and eye-tracking sensors 18 to perform their functions.

[0066] The AR system 100 may be used in combination with a Raman probe (not shown) during surgical procedures such as the removal of cancerous tissue. The AR system is used to acquire measurements of the cancerous area in the patient's tissue and display these measurements on displays 4a, 4b, which are visible to the surgeon in their line of sight, thereby providing the surgeon with guidance during surgery.

[0067] Figure 3 shows another exemplary augmented reality (AR) system 101 according to one embodiment of the present invention. The AR system 101 in Figure 3 has an AR headset 3. The AR headset 3 has the same features as the AR headset 2 in Figure 2, which has a sensor set 20 comprising multiple cameras 6, a light source 8, and multiple distance sensors 10 (however, the multiple cameras and distance sensors are not explicitly shown in Figure 3).

[0068] The AR system 101 further includes a Raman probe 22. The Raman probe 22 has the structure described above in relation to Figure 1 and is connected to a processor via a spectrometer (not shown). The probe consists of a measuring tip 26, which is the area of ​​the probe that collects measurements. The probe 22 also includes a marker 24 that can be used to track the relative position of the probe 22 to the patient and the AR headset 3.

[0069] Although not shown in Figure 3, the AR system 101 has a processor that communicates with the AR headset 3 and probe 22 to perform processing steps.

[0070] In Figure 3, the probe 22 is shown adjacent to a region of patient tissue 28. Patient tissue 28 is the region from which measurements are collected by the probe 22, for example, a suspected cancerous area. Biological landmarks 30 are located on the patient tissue 28, which are used to extract features for tracking the patient's location, as will be described in detail below.

[0071] Here, we describe one specific exemplary use of the AR system shown in Figure 3. This exemplary procedure targets obtaining measurements from the cerebral cortex using biological landmarks 30, which are blood vessels on the brain surface from which fluorescent markers are collected. However, it will be understood that this method may be applicable to any part of the patient's body. In this exemplary procedure, the patient is first injected with a fluorescent dye, such as into a venous system. The dye can accumulate in the patient's blood vessels or tissues and function as a biological landmark 30, as shown in Figure 3, by fluorescing.

[0072] During a surgical procedure, the surgeon wears an AR headset 3 and positions a Raman probe 22 next to the patient tissue 28 to be analyzed. An RGB camera 6 detects a probe marker 24 when the probe 22 is within the line of sight of the RGB camera 6. The processor then performs a pattern recognition algorithm to estimate the orientation (i.e., its position and orientation) of the marker 24. This orientation can then be used to estimate the position of the probe tip 26, i.e., where the Raman measurements will be collected. Because the processor knows the geometric shape of the probe 22, a transformation that takes the probe's geometric shape into account can be used to estimate the position of the probe tip 26 from tracking the position of the probe marker. The headset coordinate system is then established. Within the headset coordinate system, the position of the probe tip is specified to coincide with the location of the cancerous tissue region.

[0073] The surgeon places a Raman probe 22 next to the patient to obtain measurements from the patient's tissue. The collected measurements are analyzed and can be stored to indicate cancerous tissue. The measurements are stored along with the corresponding positions of the probe 22 through tracking via markers 24. These stored measurements may have associated intensity values ​​and stored position coordinates as (x,y,z). This allows for the generation of a 3D virtual map.

[0074] Simultaneously with the above, the NIR light source 10 illuminates the patient tissue, causing the fluorescent dye injected into the patient to fluoresce. The injection site may be selected so that the dye accumulates in a specific area of ​​the patient, such as a particular blood vessel near the region of interest for probe measurement. These blood vessels can serve as biological landmarks 30 used to extract features for tracking the patient. The fluorescent light emitted from the blood vessels is detected as a series of images by the NIR camera 6. 2D or 3D features of the biological landmarks are extracted from these raw data images. A patient coordinate system is established, and the 2D / 3D features are stored in this coordinate system.

[0075] The relative position between the feature and the probe measurement position can be calculated to determine the relationship between these positions. This relationship can be used throughout the procedure to update the position of the virtual map of measurement results on the display by tracking the position of the feature. This determination may be made by transforming between the patient coordinate system and the headset coordinate system.

[0076] The location (i.e., movement) of features (biological landmarks) across consecutive frames is estimated during the tracking process (explained in more detail below).

[0077] As outlined above, a virtual map is generated from probe measurements. The virtual map indicates the area of ​​cancerous tissue. Alignment is then performed so that the patient's coordinate system is mapped to the headset's coordinate system. In this way, the virtual map can be displayed on the AR headset display in the correct position relative to the patient. This allows the surgeon to perform surgical procedures using a virtual map that shows cancerous tissue clearly aligned with (for example) the patient, thereby acting as a guide superimposed in the correct position in the surgeon's field of view relative to the patient. This is achieved by continuously tracking the location of features by continuously detecting NIR fluorescence signals from biological landmarks (e.g., blood vessels) throughout the procedure, thereby forming a series of image frames.

[0078] In addition, eye-tracking can be used to further orient the virtual map on the display to the surgeon's correct field of view. One or more eye-tracking sensors can track the patient's eye coordinates. By converting the surgeon's eye coordinates to the headset coordinate system, the position of the virtual map can be updated on each display.

[0079] Figure 5 shows an exemplary virtual map 500, as described above, displayed on an AR headset and acquired using the AR system of Figure 3. The virtual map may be overlaid on displays 6a, 6b in the correct position relative to the surgeon's field of view of the patient at the location where the measurement was taken. The virtual map 500 shown in Figure 5 has three regions with different intensities, indicated by the color or shading of the map regions. The intensity indicates the intensity of the recorded measurement. For example, in the examples described herein, higher intensity (i.e., darker regions) can indicate a higher concentration of cancerous tissue. Region 501 has the lowest intensity and indicates an area without cancerous tissue. Region 503, located within region 501, has a higher intensity than region 501 and indicates a low level of cancerous tissue. Region 505, located within region 503, has a higher intensity than regions 503 and 501 and indicates a high level of cancerous tissue. For example, region 505 may indicate a tumor site.

[0080] The virtual map shown in Figure 5 is illustrated as a 2D map, representing how a surgeon would see it displayed on displays 4a and 4b. However, the virtual map may actually consist of a 3D map of measurements obtained from the tissue site, and a 2D projection of the 3D map is displayed on the screen. The virtual maps displayed on each of the displays 4a and 4b may be different projections of the 3D map. This projection takes into account the different eye positions of the surgeon.

[0081] Figure 4 shows a flowchart of the above-described method 400, which is performed to obtain measurements from a patient and to accurately display the measurements in the user's field of view of an AR system as shown in Figure 3.

[0082] In step 401, a calibration procedure is performed to calibrate the AR headset. This may include calibrating the sensors to take into account their relationships with each other. The processor determines the relative positions of sensors 6, 8, and 10 and the display 4. Sensor parameters can also be determined. For example, parameters for the pinhole camera model used by the algorithm, or parameters to compensate for distortions caused by the optical system in front of the sensor can be determined. Further calibration is also performed to determine the surgeon's interpupillary distance (IPD) (i.e., the distance between the surgeon's eyes). This may be obtained by taking measurements using the eye-tracking sensor to determine the IPD. Alternatively, the IPD may be manually entered into the software by the surgeon. The position of the surgeon's eyes relative to the display can also be determined. This calibration ensures accurate superposition of the acquired probe measurements with the correct position of the patient from which the probe measurements were collected. Calibrations related to eye tracking can be performed continuously during use of the device.

[0083] In step 403, an optional probe calibration is performed. This calibration allows the geometric shape of the probe to be determined, and the position of the probe tip 26 relative to the position of the probe marker 24 to be determined.

[0084] In step 405, the Raman probe is positioned adjacent to the patient tissue and measurements are taken. The probe can determine measurements from a tissue area of ​​interest, such as detecting the area of ​​cancerous tissue. Measurements can be continuously acquired by the surgeon simply by placing the probe next to the patient tissue, or by the surgeon pointing to the area of ​​interest and pressing a button on the probe to activate measurement acquisition.

[0085] In step 407, the probe's position is tracked to determine where the measurement was performed. As outlined in step 403, the position of the probe tip or measurement component 26 may be known relative to the probe marker 24. The RGB camera 6 of the headset 3 can capture an image of the probe marker 24. Coordinates can be assigned to the probe tip 26 in a coordinate system (headset coordinate system). This assignment may be achieved through a detection algorithm such as image pattern recognition. The coordinates are sent to a processor, and a virtual map is created that highlights the areas of cancerous tissue.

[0086] In step 409, one or more biological landmarks are tracked, and their locations at each point in time are determined during the surgical procedure. The use of biological landmarks provides a method for tracking the patient throughout the procedure without the need for physical markers.

[0087] The tracking step 409 may include several substeps. As outlined above, biological landmarks (such as blood vessels) can be tracked through biological landmarks that emit fluorescent signals after contrast agent is injected during or prior to a surgical procedure. The NIR light source 8 of the headset 3 illuminates an area of ​​the patient's tissue of interest at one or more wavelengths known to induce fluorescence. The fluorescent light emitted from the patient is captured by the NIR camera 6. The signal may then be filtered to remove unwanted noise. The raw data image formed from the captured NIR light is corrected or modified using calibration parameters obtained during the calibration step 401.

[0088] Next, features of biological landmarks are detected (i.e., identified), and as a result, these features can provide easily identifiable reference points for tracking across image frames. These features may include, for example, blood vessel edges or intersections. For example, a robust algorithm may be used to identify these features based on the size, shape, and color intensity of the biological landmarks. Features may be identified as 2D or 3D features. For example, when using multiple NIR cameras, a 3D model of each frame may be determined. However, when using only a single camera (without a depth sensor), a 2D image may be used instead. Features are extracted from each frame. The location of the features is set within CS2.

[0089] The relative positions between the features of biological landmarks and the probe measurement locations can be calculated to determine the relationship between these positions. This relationship can then be used throughout the procedure to update the position of a virtual map on the display by tracking the location of the features of the biological landmarks. This may also be achieved through a transformation between the patient coordinate system and the headset coordinate system (CS1), which allows for easy determination of the relative positions between the patient coordinate system and the headset coordinate system.

[0090] Next, features are tracked across consecutive frames to estimate motion. This estimation is achieved by finding correspondences between 2D / 3D features across consecutive frames acquired at consecutive points in time.

[0091] A transformation matrix is ​​calculated to estimate the movement of biological landmark features across consecutive frames. This may be a 2D or 3D transformation, depending on whether the tracked feature is 2D or 3D. Separate transformations may be calculated to estimate the movement between each consecutive frame, thus providing transformations between frames. Due to the fact that a patient's body is dynamic and constantly moving due to respiratory movements and soft tissue deformation, biological landmark features may be displaced across consecutive frames. To compensate for this displacement, non-rigid registration and elastic deformation correction can be used to estimate the position of biological features (in 2D or 3D) in real time. (For dynamic objects, non-rigid registration and elastic deformation correction are preferred over rigid registration because rigid registration poses problems associated with sensitivity, reproducibility, robustness, or multiple views.) The elastic deformation model may be generated to help maximize the accuracy of the alignment. Furthermore, the shape of the feature may change across frames due to such patient movement, potentially resulting in non-uniformity when comparing features. The displacement values ​​of the features can be verified by image gradient calculations on the 3D data. For this purpose, threshold levels are assigned based on tolerance criteria. When the threshold is reached, the virtual map is aligned as described later in Section 411. For example, this may be when the features match by approximately 80–90%. The transformation matrix may be continuously estimated throughout the surgical procedure to provide continuous tracking of the biological markers.

[0092] Before calculating the transformation matrix, optimization procedures can be applied to remove noisy or incomplete data. This may include using global optimization. Alternatively, or in addition, outlier exclusion algorithms or optimization filters may be used. Examples include optimized random sample consensus (RANSAC) or iterative nearest neighbor algorithms. Advantageously, these techniques help remove noise and speckle, thereby reducing mismatches that can be caused by tracking drift originating from noisy data.

[0093] Step 405 is performed concurrently with step 407, thereby relating the measurements obtained with the probe to the collected data coordinates. Steps 405 and 407 are also performed concurrently with step 409, thereby allowing the location of the probe measurements to be tracked against features of biological landmarks not only when the measurements are taken, but throughout the remainder of the surgical procedure.

[0094] In step 411, alignment is performed so that the coordinates of the probe measurements can be mapped to the coordinates of the features of biological landmarks. This ensures that the virtual map is correctly aligned with the patient's body region when viewed on the display.

[0095] In step 412, the surgeon's eye position is tracked and the surgeon's gaze toward the display is determined. The headset 3 may be equipped with one or more eye-tracking sensors capable of tracking the surgeon's eye position. This tracking may be achieved by the eye-tracking sensors tracking a point on the surgeon's retina. To ensure accurate placement of measurements on the display, the user's IPD (interpupillary distance) may also be used in conjunction with the surgeon's eye position.

[0096] One or more sensors may be used to track both of the surgeon's eyes. Alternatively, separate tracking sensors may track the position of each eye. The eye-tracking sensor may comprise an IR light source and a detector that scans the position of each eye to determine its location. The light source may be in the form of an LED or a laser. Alternatively, or in addition, the eye-tracking sensor may be an electroglottal eye-tracking sensor. An electroglottal eye-tracking sensor measures eye movement using electrodes placed around the eye.

[0097] In step 414, probe measurements are displayed on the display, superimposed on the patient's position from which the measurements were acquired. The processor displays virtual maps on displays 4a and 4b. Each display 4a and 4b displays a virtual map that is correctly positioned relative to the associated surgeon's eye and the patient's field of view. The position, orientation, and scale of the virtual maps are dynamically adjusted in real time based on the relative position of the AR headset 3 to the patient, which is continuously tracked. The surgeon wearing the AR headset perceives the virtual map as being superimposed on the patient's body, and can therefore use the virtual map as visual guidance during surgery.

[0098] Steps 409, 411, 412, and 414 are repeated throughout the procedure to take into account the patient's movement and to ensure that the measurements are correctly superimposed on the display.

[0099] Steps 405 and 407 may be optionally repeated to obtain additional measurements throughout the procedure.

[0100] The relative position of the headset to the target must be continuously tracked during the procedure to ensure that the virtual map is displayed in real time with the correct position, orientation, and scale. As outlined above, this tracking may be done by tracking the position of the patient's biological landmarks throughout the procedure. However, tracking the distance between the headset and the patient can also be done throughout the surgical procedure. This tracking may be done by one or more depth sensors placed on the headset, such as a time-of-flight sensor. For example, a depth sensor may make it possible to determine the relative position and / or orientation (posture) of the headset to the patient throughout the procedure. This may be done by generating a point cloud or depth image of the surgical site. This allows the scale of the image on the display to be adjusted accordingly, taking into account the distance from the patient. This may be useful for performing a conversion between patient coordinates and headset coordinates. For example, a depth sensor may make it possible to determine the relative position and / or orientation (posture) of the headset to the patient throughout the procedure. This allows additional parameters to be added to biological landmark tracking by knowing the distance between the biological landmark and the headset. Since probe data can be acquired as 3D data, it may be possible to obtain 3D data of the features of biological landmarks by using point clouds or depth images. As a result, these features can be compared to determine the relationship between two locations.

[0101] A processor may be a processor for executing instructions in a data processing device. Instructions may be stored, for example, in memory. A processor may include one or more processing units (for example, in a multi-core configuration) for executing instructions. Instructions may be UNIX®, LINUX (Registered trademark)These can be executed within various different operating systems on data processing devices, such as Microsoft Windows®. More specifically, instructions can trigger various data operations on data stored in memory (e.g., create, read, update, and delete procedures). It should also be understood that when a computer implementation is started, various instructions may be executed during initialization. Some operations may be required to perform one or more of the methods described herein, while other operations may be performed in a specific programming language (e.g., C, C#, C++, Java®, Python). (Registered trademark) It may be more general and / or specific to (or other appropriate programming languages).

[0102] While the aspects of this disclosure have been described in detail, it will be apparent that modifications and variations are possible without departing from the scope of the aspects of this disclosure as defined in the attached claims. Since various changes can be made to the above-described configurations, products, and methods without departing from the scope of the aspects of this disclosure, all matters included in the above description and shown in the attached drawings are intended to be interpreted as illustrative, not restrictive.

[0103] The probe's position can be tracked using alternative methods other than those described above in step 407, which uses the RGB camera 10 on the headset and the visible marker on the probe. For example, an infrared camera (or NIR) can be used instead for tracking. This may be done by using a probe marker that emits an infrared signal detected by the probe. Alternatively, any type of electromagnetic emitter (transmitter) and sensor may be used to track the probe's position. Alternatively, or in addition, a custom trained model based on machine learning may be used.

[0104] The probe tracking described above is assumed to be performed by sensors on the headset. Alternatively, sensors 6, 10 (e.g., cameras) may be located separately from the headset 3. In this configuration, markers on the headset 3 are also tracked so that the relative position of the headset 3 to the external tracking system is also known. In other configurations, probe 22 may also include one or more sensors that enable it to track its own position and communicate that position to the processor. For example, probe 22 may include an inertial measurement unit (IMU) that enables it to track details of its position and / or orientation.

[0105] Alternatively, the probe may be tracked using an electromagnetic field generated beneath or near the patient and surgical setting. In this configuration, the probe and headset are equipped with magnetic markers placed on them. In this way, the magnetic field generated by the mat detects the movement of these small magnetic units in the electromagnetic field, thereby tracking their movement and obtaining their spatial position. In this configuration, an RGB camera used to track the probe may not be necessary.

[0106] Sensors for tracking biological markers can also be placed outside the headset 3, as described above. Light sources 8 that illuminate biological landmarks and generate fluorescence may be located far away from the AR headset 3, rather than being specifically placed on the headset 3.

[0107] In some configurations, the processor and its associated memory may be located on the AR headset. In other configurations, the processor and memory may be located remotely from the AR headset. When located on the headset, the processor may be located at the back of the wearer's head. In other configurations, the processor may be located in the housing of the headset (i.e., the front part of the wearer's head). However, any other location for the processor may be possible.

[0108] While the above description focuses on Raman spectroscopy, the above system and method can be used for other purposes. For example, the probe is not necessarily limited to a Raman probe. Alternatively, the probe may be a temperature probe capable of obtaining temperature measurements from the patient. The temperature measurements may be displayed on a screen as a virtual map in the correct position in the surgeon's field of view relative to where the measurements were obtained. Probe 22 can be used to take measurements from any area of ​​the patient's body. For example, this area may be the cerebral cortex with biological landmarks of vascular patterns on the brain surface. Or, it may be observing the patient's liver with biological landmarks of vascular patterns near the liver. The surgical procedure described above is for cancer surgery, i.e., the removal of cancerous tissue. The virtual map helps guide the surgeon during the surgery. However, it will be understood that the above system can be used for any type of surgery. In other configurations, the probe may be an electromagnetic probe. For example, the electromagnetic probe may be an electromagnetic pulsation probe. An electromagnetic pulsation probe can be used to measure or evaluate the brain function of a patient in an invasive procedure. Alternatively, any other type of probe, such as a PH probe, may be used. In other configurations, probes can be used to place virtual markers on the user's face, creating a virtual map that can be used later during surgery.

[0109] The term "surgeon" is used herein to refer to a person wearing the AR headset 3. However, as should be understood, the person does not necessarily have to be a surgeon and could be any person using the device. The terms surgeon, wearer, and user may be used interchangeably herein. The term "coordinate system" as used herein may be understood to mean a coordinate system.

[0110] Image detection and associated actions are described as being performed by the camera. However, any type of image sensor / image sensing device can be used. The camera may be configured to detect still images or video.

[0111] The AR headset is shown in the figure as having two displays mounted on a housing having two arms (temples). However, it will be understood that the AR headset of the present invention is not limited to this configuration, and any conventional means of attaching a head-mounted display (HMD) to the wearer may be envisioned. This means may include using a strap that passes around and / or over the head to hold the headset in place. Alternatively, mounting means that attach the device over the entire top of the head, such as a hat, may be used. In addition, the particular configuration of the housing shown in the figure is merely an example, and any type of housing may be used. In other configurations, the AR system may comprise one or more glass displays that can be placed on the patient rather than an AR headset. Thus, the above method is used to display measurements on the displays, as described in the above embodiments.

[0112] The step of using eye-tracking as described herein improves the accuracy of aligning the virtual map on the display. However, this is an optional step, and proper alignment can be achieved without using eye-tracking. In addition, the eye-tracking sensor does not necessarily have to be placed on the AR headset 3, but may be placed in an external sensing system separate from the headset 3.

[0113] This specification describes NIR cameras and NIR light sources for performing fluorescence-based procedures, but is not necessarily limited to such wavelength ranges. Other wavelengths of light may be used. For example, any range of infrared wavelengths can be used. In other configurations, visible light can be used. For example, IR light can be used to visualize blood vessels without injecting fluorescent dyes.

[0114] It will be understood that the types and number of cameras and sensors mentioned above are not limited. One or more cameras and / or distance sensors may be used. Time-of-flight motion trackers and stereoscopic camera systems may be one such example for providing precise positioning.

[0115] In the method described above, the features of the biological landmark 30 can be used as the primary reference point when determining the relative position of the probe 22 to the biological landmark 30. However, it will be understood that, alternatively, the probe position, rather than the features of the biological landmark 30, may be used as the primary reference point.

[0116] In other configurations, artificial markers on the operating table may be used instead of biological landmarks. These markers can also be used in addition to biological landmarks to improve tracking accuracy.

[0117] Biological landmarks are described as being tracked in the methods herein. It will be understood that this tracking may be of target tissue (i.e., blood vessels and other examples).

[0118] The sensors described in relation to Figures 2 and 3 are merely examples of the types and numbers of sensors that can be used. In many configurations, multiple sensors can be used to maximize the accuracy of probe and patient tracking.

[0119] As shown in the example above, there may be two displays 4a and 4b. However, in other configurations, a single display for both eyes may be used. Alternatively, the AR headset may have a single near-eye display above the monocular, i.e., having only 4a and / or 4b, rather than both eyes.

[0120] As shown above, data can be communicated between different components through one or more cables, but it will be understood that the connection does not necessarily have to be wired. For example, signals can be transmitted from, for example, a probe, a headset, and a processor using any type of wireless communication means. These means may include, for example, USB, Wi-Fi, or Bluetooth®.

[0121] In some configurations, previously acquired medical images can also be included in the virtual map. For example, this may be data from patient scans such as CT or MRI scans. In addition to the virtual map, additional information within the field of view can be displayed. For example, patient-specific information. This specific information may be from the patient's files. The AR headset can also be configured to implement other functions, such as having communication means capable of providing audio or video signals to a surgeon so that the surgeon can perform a telementoring process.

[0122] Other alternative configurations may anticipate additional applications that would allow the virtual map to be aligned with patient-specific digital data (i.e., patient-specific 3D models of high-risk anatomical structures surrounding areas of cancerous tissue). This can be achieved by:

[0123] ■ Use sensors integrated into an AR headset to detect biological markers. The data collected by the sensors is used to create a patient-specific 3D model of this structure based on the biological marker (for example, the biological marker can indicate high-risk anatomical structures that are displayed in addition to the virtual map). The system matches the coordinate system of the 3D model to the patient coordinate system for use in the procedures described above, i.e., it displays both the measured and probe measurements that make up the 3D model.

[0124] ■ Physical markers are attached to the patient's body, and the patient is scanned. Then, a patient-specific 3D model is obtained from the medical scan, and their relative positions to the markers are imported into the system. The coordinate system of the 3D model is matched to the patient's coordinate system. In this way, in addition to probe measurements, measurements from the patient scan (MRI, CT, ultrasound, etc.) and / or camera can be displayed.

Claims

1. An augmented reality (AR) system for acquiring and displaying measurements from a patient during medical procedures, An AR headset equipped with a near-eye display for displaying the measured values ​​superimposed on the field of view of a surgeon of the patient's tissue at the location relative to the patient from which the measured values ​​were collected, At least one sensor for tracking the position of the patient, A memory for storing the aforementioned measurement values ​​and the locations associated therewith, Processor and Equipped with, During the aforementioned medical procedure, the following steps are taken: (a) A step of receiving measurements collected from the patient's tissue, wherein the measurements are collected by probes positioned adjacent to various regions of the patient's tissue, (b) Tracking the position of the probe while it is collecting measurements to determine the location where the measurements are collected, (c) Simultaneously with performing steps (a) and (b), the step of using at least one sensor to track the location of a biological landmark on the patient so as to determine the relationship between the location of the biological landmark and the location where the measurement is collected, (d) Display the measured value at a certain position on the near-eye display, To continuously track the location of the biological landmark throughout the entire medical procedure and obtain its current position, The position of the measured value on the display is updated based on the current position of the biological landmark. The steps include ensuring that throughout the entire medical procedure, the measurements are aligned within the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurements were collected, and An augmented reality (AR) system configured to perform the following actions.

2. The AR system according to claim 1, further comprising the step of creating a virtual map of the measured values, wherein the step of displaying the measured values ​​on the near-eye display includes the step of displaying the virtual map.

3. The AR system according to claim 1 or 2, wherein the step of tracking the location of the biological landmark on the patient further includes the steps of illuminating a region of the patient using light of a specific wavelength, and detecting a fluorescent signal emitted by the biological landmark in response to the illumination using the sensor.

4. The AR system according to claim 1 or 2, further comprising a probe.

5. The AR system according to claim 4, wherein the probe is a Raman probe.

6. The AR system according to claim 1 or 2, wherein the biological landmark is one or more features of the biological landmark.

7. To continuously track the position of the feature throughout the entire medical procedure and obtain its current position, The sensor captures an image of the patient, From the aforementioned image, one or more of the features of the biological landmark are extracted. Determine the correspondence between one or more features across consecutive frames of the aforementioned image. By using a transformation matrix, the motion of one or more features is estimated across consecutive frames of the image. The AR system according to claim 6, further comprising the following:

8. The AR system according to claim 7, wherein the step of estimating the motion of one or more features over consecutive frames of an image by using a transformation matrix further comprises the step of applying rigid or non-rigid registration.

9. The AR system according to claim 6, wherein one or more features of the biological landmark include an edge or intersection of the biological landmark.

10. The AR system according to claim 1 or 2, wherein the biological landmark includes at least one of the patient's blood vessels, the patient's lymph nodes, the patient's nervous system, and the surface of the patient's organs.

11. The step of tracking the position of the probe is, The steps include: detecting a marker on the probe via the sensor and calculating the position of the tip of the probe that represents the position where the measurement value is collected; A step of detecting a change in the magnetic field caused by the movement of a magnetic marker on the probe, wherein the change in the magnetic field identifies the position of the probe. The AR system according to claim 1 or 2, including the following:

12. The AR system according to claim 1 or 2, wherein the relationship between the position of the biological landmark and the position where the measurement is collected is determined by transforming the position of the biological landmark and the position where the measurement is collected into a common coordinate system.

13. The step of displaying the measurement at a certain location on the near-eye display so that throughout the medical procedure, the measurement is aligned within the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurement was collected, The steps include tracking the position of the surgeon's eye and adjusting the position of the measurement on the near-eye display based on the current position of the surgeon's eye. The AR system according to claim 1 or 2, further comprising:

14. A method for obtaining and displaying measurements from a patient to a surgeon during a medical procedure using an augmented reality (AR) system, (a) A step of receiving measurements collected from the patient's tissue, wherein the measurements are collected by probes positioned adjacent to various regions of the patient's tissue, (b) Tracking the position of the probe while it is collecting measurements to determine the location where the measurements are collected, (c) Simultaneously with performing steps (a) and (b), the step of using at least one sensor to track the location of a biological landmark on the patient so as to determine the relationship between the location of the biological landmark and the location where the measurement is collected, (d) Display the measured value at a certain location on the near-eye display of the AR headset worn by the surgeon, Throughout the entire medical procedure, the location of the biological landmark is continuously tracked to obtain its current position. The position of the measured value on the near-eye display is updated based on the current position of the biological landmark. The steps include ensuring that throughout the entire medical procedure, the measurements are aligned within the surgeon's field of view of the patient's tissue at the location relative to the patient where the measurements were collected, and Methods that include...

15. A computer program product that includes instructions, wherein when the program is executed by a computer, the instructions are sent to the computer. (a) A step of receiving measurements collected from the patient's tissue, wherein the measurements are collected by probes positioned adjacent to various regions of the patient's tissue, (b) Tracking the position of the probe while it is collecting measurements to determine the location where the measurements are collected, (c) Simultaneously with performing steps (a) and (b), the step of using at least one sensor to track the location of a biological landmark on the patient so as to determine the relationship between the location of the biological landmark and the location where the measurement is collected, (d) Display the measurement at a certain location on the near-eye display of an AR headset worn by the surgeon, so that the measurement is available throughout the medical procedure at the location relative to the patient from which the measurement was collected. To continuously track the location of the biological landmark throughout the entire medical procedure and obtain its current position, The position of the measured value on the near-eye display is updated based on the current position of the biological landmark. The steps include: aligning the patient's tissues within the surgeon's field of view; A computer program product that performs a certain action.

Images