Augmented reality headset for medical imaging

Abstract

To provide an augmented reality (AR) system for use in medical procedures.SOLUTION: An augmented reality (AR) system 100 comprises an AR headset 2 and a processor 12. The AR headset includes cameras 6a and 6b, near-eye displays 4a and 4b, and range sensors 10a and 10b. The processor is configured to adjust positions of images acquired by the cameras on the displays throughout medical procedures based on changes in the distance measured by the distance sensors.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an augmented reality system for use in medical procedures.

Background Art

[0002] Fluorescence-based intraoperative surgical guidance has become a widely used procedure. This popularity has particularly focused on a clinical approach using indocyanine green (ICG) as a fluorescence marker detected in the near-infrared spectrum (NIR: Near-Infrared Ray).

[0003] To enable ICG-based guidance in surgery, several medical imaging devices have been commercialized. This approach is used for sentinel lymph node identification and mapping, blood flow assessment, vascular patency, perfusion evaluation (especially in reconstructive and bypass surgeries), lymph imaging, and surgical procedures. Further research efforts are targeted at the potential of imaging molecular tracers that report on new vascular, structural, metabolic, immunological, or genetic features of tissues. The application rate and scope of ICG in medical procedures are rapidly increasing.

[0004] This process involves injecting a fluorescent dye molecule immediately before a surgical procedure. Next, near-infrared light is irradiated onto the target area, exciting the molecular ligand, and as a result, light of a specific wavelength is emitted. Then, a camera sensitive to the light spectrum in this range is used to detect the light and form an image clearly showing the target tissue.

[0005] This approach enables significant improvement in the detection and removal of metastatic foci remaining in sentinel lymph nodes, for example, in lung cancer surgery, not only for primary tumor nodules of various cancer types. Other uses include its use in breast cancer surgery involving mastectomy or lumpectomy and planned sentinel node biopsy procedures.

[0006] The current system is based on a mobile camera unit that collects the emitted light and visualizes the detected image on an external screen next to the operating table. During the procedure, the surgeon has to disengage the focus from the area of the patient being operated on and subjectively compare the display data taken from different viewpoints with the real-world view of the patient's actual body. This continuous comparison operation makes fluorescence imaging cumbersome and particularly limits the ability to accurately map what the surgeon can see of the displayed fluorescently labeled tissue. In this latter subjective step, a significant amount of accuracy, completeness, concentration, and time efficiency is lost, despite excellent specialized training.

Prior Art Documents

Non-Patent Documents

[0007]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] It is desirable to have a method that can more easily and accurately associate the view of the target tissue detected by fluorescence imaging with the surgeon's actual real-world view of the patient. This would result in a faster procedure and a reduced likelihood of human error.

Means for Solving the Problems

[0009] According to one aspect, an augmented reality (AR) system for use in a medical procedure is provided. The AR system includes an AR headset and a processor. The AR headset includes a camera configured to detect light from a target, a near-eye display positioned between the wearer's eyes and the target, the near-eye display being configured to display an image of the target based on the light detected by the camera such that the image overlaps the wearer's view of the target, and a distance sensor configured to determine a distance between the headset and the target throughout the medical procedure. The processor is configured to determine a mismatch between the image of the target acquired by the camera and the wearer's view of the target based on the value of the distance measured by the distance sensor and the position of the wearer's eyes, adjust the position such that the position of the image on the display is corrected based on the determined mismatch so that the image matches the wearer's view of the target, and repeat the determination of the mismatch and the adjustment of the position of the image throughout the medical procedure to account for changes in the distance measured by the distance sensor throughout the medical procedure.

[0010] By displaying an image of the target on the display of the augmented reality device, a view of the image generated by the light from the target in the direct line of sight is provided to the wearer performing the medical procedure. This eliminates the need for the wearer to adjust their gaze and switch between viewing the image on an external display and the patient's real-world view.

[0011] Differences between views can be adjusted based on the difference in distance between the position of the wearer's eyes and the target. This is due to the fact that the camera does not have the same view as the wearer's view of the target. This correction is performed throughout the procedure by a distance sensor that measures the distance between the headset and the target. Thereby, the position of the extended image on the display can be dynamically updated so that the displayed image is in the correct position to overlap with the real-world view of the wearer's target. Thereby, the extended image on the display surely coincides with the exact position on the original target from which it was radiated to the wearer's view.

[0012] Thereby, an accurate mapping between the extended generated image and the actual patient can be provided to a medical professional wearing the headset. Thereby, the accuracy in performing surgical procedures on the patient is significantly improved.

[0013] Preferably, the processor assigns a position in space to act as a fixed reference point, generates a 3D model of the target based on the light detected by the camera, determines the position and orientation of the target relative to the fixed reference point based on the distance measured by the distance sensor, determines the position of the wearer's eyes relative to the fixed reference point, and determines the position and orientation of the headset relative to the fixed reference point, and is further configured to determine the mismatch between the image of the target obtained from the camera and the wearer's view of the target.

[0014] By determining the position and orientation of the headset and the target, and the position of the wearer's eyes relative to the fixed reference point, the positions and orientations are converted into a fixed reference frame relative to each other. Thereby, through continuous updated measurements of the distance to the target, the relative positioning of the image of the target on the display can be adjusted through the geometric relationship between the wearer's eyes and the headset relative to the target. In addition to the position of the wearer's eyes, the orientation of the wearer's eyes can also be determined.

[0015] The fixed reference point may be on the headset. For example, it may be installed at a point between the locations where the wearer's eyes are placed. Alternatively, it may be set as the point where a camera, distance sensor, or display is installed on the headset. The fixed reference point does not have to be on the headset and may instead be a point outside the headset. The fixed reference point may be a position within a 3D space represented in 3D coordinates. The positions and orientations of the wearer's eyes, the target, and the headset can be converted into 3D coordinates.

[0016] The position and orientation of the headset with respect to the fixed reference point may be the position and orientation of at least one of the display, distance sensor, and camera. The positioning of the display, camera, and / or distance sensor relative to each other may be known. Thus, when adjusting the position of the image, the displacement between each of them can be considered. Since the positions relative to each other may be static, these positions may be known. The processor may receive the values of these positions. For example, these can be stored in memory. When the fixed reference point is a position on the headset, the distances between the display, distance sensor, and camera with respect to the fixed reference point may be known from knowing the shape of the headset.

[0017] Alternatively, if not known, the values of these positions and orientations can be measured using one or more sensors.

[0018] Preferably, the processor sets the position of the 3D model of the target with respect to the fixed reference point, renders the 3D model of the target to form an adjusted image based on the determined positions and orientations of the target and the headset, and the position of the wearer's eyes, and is further configured to adjust the position such that the position of the image on the display is corrected based on the determined discrepancy so as to be displayed on the display.

[0019] In this way, the generated 3D model of the target is in the same reference frame as the headset and the wearer's eyes. Thereby, the 3D model of the target can be rendered such that the images displayed on the display take into account the positions of the headset, the target, and the wearer's eyes.

[0020] Preferably, the processor is further configured to determine the parallax of the wearer's eyes. Parallax, i.e., binocular parallax, is the difference in the projection points of the images in the two eyes of the wearer. The parallax can be determined based on the determined distance to the target. It can also be determined based on the position of the wearer's eyes. This may be based on the interpupillary distance (IPD) and / or the distance between the headset and the wearer's eyes. Preferably, the processor is configured to determine the parallax of the wearer's eyes from the determined distance and the position of the wearer's eyes, and is further configured to determine the mismatch between the image of the target obtained from the camera and the wearer's view of the target.

[0021] The distance to the target is inversely proportional to the parallax. Knowing the distance to the target and the position of the wearer's eyes, the parallax of each eye can be determined. Thereby, the alignment of the generated images in the display can be updated to match the wearer's view of the target for each of the wearer's eyes. The adjustment of the images in the display may be different for each of the wearer's eyes.

[0022] In some embodiments, the AR headset may further include an eye-tracking sensor configured to continuously determine the position of the wearer's eyes throughout the medical procedure, whereby changes in the position of the wearer's eyes throughout the medical procedure are taken into account in repeated determinations of mismatches and adjustments of the position of the images throughout the medical procedure.

[0023] In this way, it can be used to track the position of the wearer's eyes throughout the medical procedure and update the position of the generated image within the display. Throughout the medical procedure, the position of the wearer's eyes does not change and become fixed as the view and gaze change throughout the procedure. The accuracy of positioning the generated image onto the wearer's view can be improved by the continuous determination of the position of the wearer's eyes and the target and the headset. The eye-tracking sensor can be configured to also determine the orientation of the wearer's eyes throughout the medical procedure.

[0024] Also, the eye-tracking sensor can determine the focus of the wearer's eyes, i.e., the position of the place where the wearer is concentrating at any given time. In this way, the generated image can be displayed so that it is always in focus within the wearer's view of the target.

[0025] The eye-tracking sensor may be a plurality of eye-tracking sensors. For example, there may be one sensor for tracking each eye. Alternatively, one tracking sensor may track the positions of both eyes.

[0026] The eye-tracking sensor may be an IR light source that scans the position of each eye to determine its position. This may be in the form of an LED or a laser. Alternatively, or in addition thereto, the eye-tracking sensor may be an electrooculogram eye-tracking sensor. The electrooculogram eye-tracking sensor uses electrodes placed around the eyes to measure the movement of the eyes.

[0027] The processor can receive the position of the wearer's eyes. Preferably, the processor is further configured to obtain the position of the wearer's eyes by obtaining the interpupillary distance of the wearer's eyes.

[0028] Interpupillary distance is the distance between the pupils of each of the wearer's eyes. Each user has a unique interpupillary distance. Knowing the interpupillary distance allows it to be used to determine the relative position of the wearer's eyes with respect to the headset and / or fixed reference points. It can also be used to determine the parallax of the wearer's eyes. This can help position the generated images at appropriate positions on the display for each of the wearer's eyes that overlap the view of the target of each eye.

[0029] Interpupillary distance may be automatically acquired by a processor. This can be done using an eye-tracking sensor. Alternatively, the wearer may manually provide the interpupillary distance to the processor. For example, the wearer or another person may have manually measured the interpupillary distance.

[0030] Alternatively, the position of the wearer's eyes may be determined by means other than using the interpupillary distance.

[0031] As described above, the position of the wearer's eyes can be determined using an eye-tracking sensor. This can be the position of the wearer's eyes with respect to a fixed reference point, such as a point on the headset. Alternatively, the position of the wearer's eyes with respect to the fixed reference point may be determined by a calibration procedure. This may be performed before a medical procedure or when it is determined that further calibration is needed during a medical procedure. This calibration may require the wearer to look at an external marker. Subsequently, the position of the image of the marker can be adjusted until it coincides with the wearer's view of the marker. This may require the wearer to manually adjust the parameters of the image generation so that the image of the marker is moved to overlap the wearer's view of the marker. The marker can be any type of marker, such as a dot or line, or a reference motif with a known shape. This calibration allows the processor to determine the position and / or orientation of the wearer's eyes with respect to the fixed reference point.

[0032] Preferably, the camera may include a distance sensor. Alternatively, the camera and the distance sensor may be separate sensors.

[0033] The distance sensor may be a time-of-flight distance sensor, and / or the sensor may be a depth sensor. Alternatively, the distance sensor may be a simultaneous localization and mapping (SLAM) sensor, a vSLAM sensor, a dot marker pattern sensor, or a sensor using the same principle as a Kinect device. The distance sensor may be a sensor whose sole purpose is to determine distance. Alternatively, the distance sensor may be a camera configured to serve the role of a distance sensor. For example, the camera and the distance sensor may be the same. In this way, the camera functions as both a camera and a distance sensor. Alternatively, there may be multiple cameras that function as distance sensors.

[0034] Preferably, the AR system further includes a light source, which is configured to emit light that is incident on the target and then detected by the camera.

[0035] The light source is configured to transmit light towards the target, so that when detected by the camera thereafter, the light can form an image representing the target or a part thereof. The light source may be a fluorescent light source. Alternatively, the light source may be such that an image is formed by reflection of light from the target.

[0036] The AR headset may include a light source. If there is a light source in the headset, it enables the camera and the light source to focus on the same area of the target. Thereby, the wearer can control the illumination of the light on the target. Alternatively, the light source may be an external light source not installed on the headset.

[0037] Preferably, the light is near-infrared light. The medical procedure can be a fluorescence-based guidance procedure. Near-infrared light can be used for fluorescence-based guidance procedures. The light source can be configured to emit within these bands. The wavelength of the NIR light can be about 780 nm. This is the excitation range of ICG. However, depending on the molecular marker used, other wavelengths of NIR can also be used.

[0038] Alternatively, the light may be visible light or infrared light.

[0039] The camera can be configured to detect NIR light. The camera can be configured to detect within the wavelength range at which the molecular marker used emits. In the case of ICG, this can be a wavelength of 810 - 860 nm depending on the tissue type. The camera can be configured to detect light in the same wavelength band as the light source.

[0040] The camera can continuously detect light throughout the medical procedure. In other arrangements, the camera can be configured to collect light at intervals throughout the medical procedure. In this way, the generated image in the AR display can be updated throughout the procedure.

[0041] In other embodiments, the image detected by the camera can be captured once during the medical procedure. This image is not updated throughout the procedure, and only the position of the image at that location on the near-eye display can be updated taking into account changes in the position of the headset. This can be the case when it is not assumed that the image of the target collected by the camera will change.

[0042] The processor can be further configured to convert the light detected by the camera into an image visible to the user. When the light emitted from the target is in the IR or NIR range, the wearer's eyes cannot see the conventional light. This may be the case when the light is excited by fluorescence from the target. By converting the light and displaying the image on the near-eye display, the light reflected from the target can be displayed so that the wearer can visualize it. This image provides information that the wearer could not see with their line of sight before, improving the wearer's view.

[0043] The AR headset may include a processor. This eliminates the need for the necessary leads or physical connections between the external processor and the headset.

[0044] Alternatively, the processor may not be installed in the headset. This can reduce the weight of the headset. For example, the processor may be installed in a server or computing system external to the headset. The processor may be connected to the headset by a wired connection. Alternatively, the processor can also be connected to the headset by a wireless connection such as a WiFi, Bluetooth® connection, or other type of RF connection.

[0045] In some embodiments, the headset may include multiple cameras configured to detect the excitation light. Having multiple cameras to detect the excitation light may allow images to be detected from two different viewpoints. This can improve the generation of the target model. Furthermore, the position of the headset relative to the target can be estimated more accurately.

[0046] The near-eye display may be a single display. Alternatively, the near-eye display may be two displays, with each eye's image being displayed on one display. The near-eye display may be a waveguide. Alternatively, the display may be a beam splitter display or a laser reflection display. In the display, a mirror can be utilized to project an image into the wearer's view. The display may be made of glass and / or plastic. In this way, the display is transparent. Alternatively, the near-eye display may be a lens. Or, the near-eye display may be a beam that projects an image into the wearer's eye so that the image is displayed on the retina. Thus, the near-eye display may be a virtual retinal display.

[0047] The display and / or camera may comprise one or more filters. The filter can be configured to improve the detection of light of a specific wavelength from a target while removing other signals of undesired wavelengths. Alternatively, or additionally, the filter may be positioned in front of a light source.

[0048] According to a further aspect, a method for adjusting the position of an image within an augmented reality (AR) system for use in a medical procedure is provided. The AR system comprises an AR headset and a processor. The method includes detecting light excited from a target, determining, throughout the medical procedure, the distance between the headset and the target, determining a discrepancy between an image of the target obtained from a camera and the wearer's view of the target based on the determined distance value and the position of the wearer's eyes, and adjusting the position such that the position of the image on the near-eye display is corrected based on the determined discrepancy, and displaying, on a near-eye display positioned between the wearer's eyes and the target, an image of the target based on the detected light such that the image overlaps the wearer's view of the target in the headset.

[0049] According to a further aspect, there is provided a non-transitory computer-readable medium configured to execute the above method when executed on a processor.

Brief Description of the Drawings

[0050]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0051] Figure 1 shows an augmented reality (AR) system 100 according to an embodiment of the present invention. The AR system 100 includes an AR headset 2 and a processor 12.

[0052] The augmented reality headset 2 has two displays, namely, a first display 4a and a second display 4b. The first display 4a is for displaying an image to the right eye of the headset wearer, and the second display 4b is for displaying an image to the left eye of the wearer of the headset 2. The displays 4a and 4b are attached to the housing 16 of the headset 2.

[0053] Two cameras 6a and 6b are installed in the housing 16 of the headset. Camera 6a is installed above the first display 4a, and camera 6b is installed above the second display 4b. Cameras 6a and 6b can detect near-infrared (NIR) light.

[0054] Also, a light source 8 is installed in the housing 16. The light source 8 is a NIR light source configured to emit NIR light. The light source 8 is installed between the cameras 6a and 6b, but can be installed at any position of the AR headset 2. Alternatively, the light source 8 may be installed outside the AR headset 2.

[0055] Two distance sensors 10a and 10b are installed in the housing 16 of the headset 2. The distance sensors are time-of-flight sensors configured to determine the distance from the headset 2 to an object.

[0056] The headset 2 further includes an eye gaze tracking sensor 18. The eye gaze tracking sensor is installed on the side of the headset facing the wearer's head. The eye gaze tracking sensor is configured to determine the position of the eyes of the wearer of the headset 2.

[0057] Processor 12 is installed outside the AR headset 2. The processor can be a processor of a computer or other data processing device. The AR headset 2 is connected to the processor 12 via a cable 14. The cable 14 is for transmitting signals between the headset and the processor 12. For example, data obtained from the cameras 6a, 6b, the gaze tracking sensor 18, and the distance sensors 10a, 10b can be transmitted to the processor 12 via the cable 14. The cable 14 is also for transmitting communication signals between the processor 12 and the headset 2 to control the cameras 6a, 6b, the distance sensors 10a, 10b, the light source 8, and the gaze tracking sensor 18 to execute their functions.

[0058] Figures 2, 3, 5, and 7 show top-down schematic views of the AR headset 2 according to the present invention, and each figure shows a headset 2 with a different arrangement of sensors. The headset 2 is shown as being used in a fluorescence-based guidance procedure.

[0059] The features of the AR headset 2 shown in Figures 2, 3, 5, and 7 are denoted with the same reference numerals as the features shown in Figure 1. Here, a part of the target 20, which is a patient, is also shown, from which a fluorescence image is detected.

[0060] Figure 2 shows an AR headset having two cameras 6a and 6b. The headset shown in Figure 2 has no separate distance sensors. In addition to detecting light from the target 20 to form an image, the two cameras 6a and 6b also serve as distance sensors.

[0061] Figure 2 shows the spatial relationships determined and used to adjust the position of the image generated on the displays 4a, 4b to match the view of the target 20 of the wearer.

[0062] The distance 22 between the target 20 and each of the cameras 6a, 6b is measured by the cameras 6a, 6b.

[0063] Since this AR headset 2 does not have a gaze tracking sensor, the distance 26 between each of the wearer's eyes and each of the displays 4a 4b is determined by a calibration procedure. The calibration procedure also involves determining the distance 28 between each of the cameras 6a and 6b and each of the wearer's eyes 30a 30b. The interpupillary distance (IPD) 24 is also determined. This can also be determined by the calibration procedure. Alternatively, the IPD may be known and input into the AR system by the wearer.

[0064] The distance 32 between the two displays 4a and 4b and the distance 34 between the two cameras 6a 6b are known from the shape of the headset 2. Thereby, the distance 40 between the target 20 and the display, and the distance 38 between the target and the wearer's eyes 30a 30b can be determined.

[0065] By determining each of these distances, they can be compared to the fixed reference points 36 on the headset 2. Thereby, the processor 12 can adjust the position of the image on the display so that the position of the image on the display for each eye coincides with the wearer's view of the target.

[0066] The wearer can move their head relative to the patient 20 throughout the procedure. By continuously measuring the distance 22 throughout the procedure, the above calculations can be continuously performed to adjust the position of the image on the display throughout the procedure to coincide with the wearer's actual real-world view of the target.

[0067] Figure 3 shows an AR headset having two cameras 6a and 6b and two distance sensors 10a and 10b. In this case, cameras 6a 6b serve to capture the light emitted from target 20 and form an image on displays 4a 4b. Distance sensors 10a 10b serve to determine the distance between patient 20 and headset 2. The distances collected by distance sensors 10a 10b are shown as 42 in Figure 3. The distance 22 between target 20 and cameras 6a 6b can be calculated from the determined distance value 42 and the known spatial relationship. Other spatial relationships shown in Figure 3 having reference numbers similar to those shown in Figure 2 are determined in the same way as described in connection with Figure 2.

[0068] Figure 4 shows the steps performed by AR system 100 of Figures 2 and 3 when displaying an image of the target.

[0069] In step 101, the IPD of the wearer's eyes is obtained using a calibration procedure. In step 103, the position and orientation of the wearer's eyes relative to the headset or any other point in the virtual 3D space are calculated and saved.

[0070] In step 105, using the known shape of the headset, the positions and orientations of the headset and the cameras, distance sensors, eye-tracking sensors, and AR / MR displays relative to each other are determined. This is based on the known shape as shown in Figures 2 and 3. Thereby, in step 107, the positions and orientations of the headset and all its components and sensors relative to each other or to any other point in the virtual 3D space can be determined.

[0071] Steps 101 - 107 are performed at the beginning of the medical procedure after the wearer places the AR headset on the head. Since these determined values and spatial relationships are considered not to change throughout the medical procedure, there is no need to recalculate these values throughout the procedure.

[0072] Step 109 involves obtaining the position and orientation of the headset relative to the target using measurements acquired by a distance sensor (such as shown in FIG. 3) and / or a camera sensor (such as shown in FIG. 2). The distance sensor can use time-of-flight or any known measurement type used to determine distance. The cameras 6a 6b can use vSLAM or any known method for determining distance using an image sensor. In step 111, the position and orientation of the target relative to the origin of the headset or any other point within the virtual 3D space are calculated and stored.

[0073] Step 113 involves obtaining light from the target using a camera, recording an image of the surgical wound, and detecting biomarkers of various wavelengths using fluorescence. In step 115, based on the light received by the camera, a 3D model shape of the target area can be constructed and stored relative to local coordinates. The local coordinates can be the same point on the headset or a point within the virtual space that serves as a reference for determining other positions and orientations.

[0074] Creation of the 3D model from the images acquired by the camera can be performed using photogrammetry. This requires the 3D reconstruction of the subject from 2D captures using computer vision and computational geometry algorithms.

[0075] In step 117, the origin of the virtual 3D space is determined. As shown in FIGS. 2 and 3, this is the point 36 of the headset positioned between the two displays 4a and 4b. This virtual 3D space can be the same as the point when determining the position and orientation of the headset, the wearer's eyes, and the 3D model in steps 103, 107, 111, and 115. Thereby, in step 119, the position and orientation of the wearer's eyes, and the target are transformed into the virtual 3D space relative to the origin of the virtual 3D space.

[0076] In step 121, the 3D model of the target within the virtual 3D space is rendered.

[0077] In step 123, the rendered 3D model is displayed on the displays 4a and 4b of the AR headset 2. As a result, the target 3D model is automatically displayed in a perspective view for each eye of the wearer.

[0078] Steps 109 and 111 are performed throughout the medical procedure. This may be continuous or at fixed points in time because the wearer's head, and thus the AR headset 2, may move throughout the procedure. As a result, the value determined in step 109 will change throughout the procedure.

[0079] Steps 113 and 115 may also be performed throughout the medical procedure. This may be continuous or at fixed points in time due to the fact that the light detected by the camera may change throughout the medical procedure as it progresses. Steps 109, 111, and 113, 115 may be performed in parallel throughout the medical procedure.

[0080] As a result, steps 119, 121, and 123 are also performed throughout the medical procedure, taking into account the updated data obtained from steps 109 - 115.

[0081] FIG. 5 shows the same as FIG. 3 but with an AR headset 2 further having two eye - tracking sensors 18a and 18b. The same spatial relationships as shown in FIG. 3 are shown in FIG. 5. However, the distances 44 between the eye - tracking sensors 18a, 18b and the wearer's eyes 30a, 30b are measured using the eye - tracking sensors 18a, 18b. Thereby, throughout the medical procedure, the exact position and orientation of the wearer's eyes can be determined. The distance 46 between the eye - tracking sensor and the display is also determined. This may be determined using the eye - tracking sensors 18a, 18b or may be a known spatial relationship based on the shape of the headset.

[0082] The measurement of distance 44 and the tracking of the wearer's eyes can be performed throughout the medical procedure. Thereby, the image of the target in the display can be updated to take into account the movement of the wearer's eyes. Thereby, since the position of the wearer's eyes may be known throughout the procedure, a more accurate match to the image displayed on the AR display of the wearer's view of the target can be provided. The eye tracking sensor can determine the position of the wearer's eyes continuously throughout the medical procedure or at fixed time intervals. The eye tracking sensor can determine the position of the wearer's eyes simultaneously when the distance to the target is measured. This may be every 0.5 seconds. Alternatively, it may be more frequent than every 0.5 seconds. Alternatively, it may be every 1 second. Using an eye tracking sensor can provide higher accuracy than when not using an eye tracking sensor, which allows for accuracy of less than 1 cm to be achieved since changes in the movement of the wearer's eyes are taken into account. This is compared to the headset of FIGS. 2 and 3 where accuracy at the cm level is obtained. Also, when the position of the headset moves on the wearer's head, the eye tracking sensor can correct this movement by performing re-calibration. In this way, with an eye tracking sensor, there is no need to perform an initial calibration, such as through the use of calibration criteria at the beginning of the procedure.

[0083] The eye tracking sensor can use near-infrared technology together with a camera (or other type of optical sensor) to track the gaze direction of the wearer's eyes. This may require the use of Pupil Center Corneal Reflection (PCCR). Also, the eye tracking sensor may utilize electrooculogram recording techniques. This requires dry electrodes that measure the electrical potential of the skin around the eyes. A small electronic device interprets the electrical signals to calculate eye movement. The sample rate can be approximately 256 samples per second, but it varies depending on the type of camera. Alternatively, any known eye tracking technology can be used.

[0084] FIG. 6 shows the steps performed by the AR system 100 of FIG. 5 when displaying an image of a target. Each step of FIG. 6 that is the same as a step of FIG. 4 is denoted by the same reference numeral.

[0085] Steps 201 and 203 of FIG. 6 are different from steps 101 and 103 of FIG. 4. Step 201 includes obtaining the IPD of the wearer's eyes and the position and orientation of the wearer's eyes from measurements made by an eye-tracking sensor. Next, step 203 includes calculating and storing the focus, position, and orientation of the wearer's eyes with respect to the headset or any other point in the virtual 3D space. This is based on measurements made by the eye-tracking sensors 18a 18b.

[0086] Steps 201 and 203 are performed throughout the medical procedure as described above. Next, this is sent to step 119, where, in steps 121 and 123, a corrected image is rendered and displayed.

[0087] FIG. 7 shows an AR headset 2 without a distance sensor or an eye-tracking sensor. The AR headset 2 of FIG. 7 is an AR headset 2 having only one camera 6. To use this AR headset, it is first necessary to perform additional calculations to obtain the 3D representation and position of the target. For this reason, at the start of the medical procedure, the wearer needs to move relative to the target from a plurality of viewing angles so that the camera can obtain a plurality of images of the target to reconstruct a 3D model of the target. Since this one camera 6 is the only sensor, this cannot be done in real time throughout the medical procedure.

[0088] As can be seen in FIG. 7, the distance 40 from the target to the display, the distance 38 to the user's eyes, and the distance 34 between the camera and the fixed reference point 36, unlike FIG. 2, are determined by calibration. This is determined, as in FIG. 2, in addition to the IPD 24, the distance 28 between the camera and the wearer's eyes, and the distance 26 between the wearer's eyes and the display. The distance 32 between the display and the fixed reference point 36 is known from the shape of the headset.

[0089] FIG. 8 shows the steps performed by the AR system 100 of FIG. 7 when displaying an image of the target. Each of the steps in FIG. 8 that is the same as the steps in FIG. 4 is indicated by the same reference numeral.

[0090] Similar to FIG. 4, in step 107, the position and orientation of the headset and the components of the headset are determined relative to the position within the headset or the 3D space. The position and orientation of the wearer's eyes are also determined in step 103.

[0091] In step 315, a 3D model of the target area is generated based on the light received by the camera 6. Since there is only one camera, images are collected at multiple different angles and the 3D model is formed by a photometric algorithm. Therefore, when using the AR headset of FIG. 7 having one camera, a 3D model cannot be generated in real time, which needs to be done by an initial procedure of scanning the target using a photometric algorithm to generate a 3D model. The position and orientation of the 3D model are set relative to the headset by manual calibration (309). The wearer views the state in which the target is scanning the surface. Thereby, the position and orientation of the 3D generated model of the target can be set relative to the headset or any other point within its virtual 3D space (311). Next, the image of the 3D model is converted (119), rendered (121), and displayed on the display (123) as described in connection with FIG. 4.

[0092] As described above, in the case where the AR headset 2, like the headsets shown in FIGS. 2, 3, and 7, does not have a gaze tracking sensor for determining the position of the wearer's eyes, a calibration procedure needs to be executed. For this, the wearer may need to look at an external calibration reference 46 as shown in FIG. 9. The calibration reference in FIG. 9 is a reference motif 48. FIG. 9 shows the actual position of the reference motif 48. The uncorrected image of the reference motif is also shown at 50. In this case, the adjustment is performed until the position of the image 50 of the reference motif coincides with the wearer's view 48 of the reference motif. For this, the wearer may need to manually adjust the parameters of the image generation so that the image 50 of the reference motif is moved so as to overlap the view 48 of the reference motif. By this calibration, the processor can determine the position of the wearer's eyes relative to a fixed reference point, such as a point on the headset.

[0093] Further details of a method for correcting the alignment of an image on a display will be described. This is one example of a method by which this can be achieved, and alternative algorithms and methods may be applied instead.

[0094] To correct an image on a display based on the perspective view of the wearer, the parameters of the camera, including the position and orientation, and the optical characteristics, are determined. Also, the 3D information of the target, and the parallax between the images displayed for each eye are determined for correcting the image. The position and orientation of the camera are determined based on steps 109 and 111. The 3D information of the target is determined based on steps 113 and 115. The parallax between the images displayed for each eye is shown in detail based on steps 101, 103, 201, 203, 105, 107, 109, and 111.

[0095] The parameters of the camera are determined based on the internal parameters representing the optical characteristics of the camera and the external parameters representing the position and orientation of the camera.

[0096] The internal parameters represent the optical characteristics and can be estimated using the pinhole camera model. This includes the focal length of the camera, the aspect ratio of the plane onto which the camera's view is projected (i.e., the display), and the location of the center of the image (its principal point) where the optical axis captures the image plane.

[0097] The internal characteristics of the pinhole camera model define the projective transformation from 3D space to the 2D coordinate space of the display.

Number

[0098] The position and orientation of the camera are determined by calculating the camera's pose. This can be calculated using the camera's sensors, such as a distance sensor. The pose is represented as follows.

Number

[0099] Based on the internal and external parameters of the pinhole camera model, 3D points can be mapped to 2D image coordinates. This is shown by the following matrix transformation: T cam as shown by. T cam = K * T pose This is the product of the camera's pose (i.e., the external parameters) and the projection matrix (i.e., the internal parameters).

[0100] In a physical implementation, the pinhole camera model is not always accurate. This is because various positions of the user's eyes relative to the display are possible during execution. Therefore, the following initial calibration is required. The translation vector t of the eye position relative to the displayeye =[x, y, z] t Assuming that is known, the internal matrix can be defined as follows for each eye.

Number

[0101] Vector t eye depends on the current position of the user's eyes relative to the display. Therefore, when the headset is repositioned on the user's head or when another user wears the headset and requires recalibration, the parameters of the internal matrix change. In this case, the old eye position t0 = [x0, y0, z0] t can be used to update the old matrix K0 based on to the new internal matrix K1.

Number

[0102] The initial internal and external matrices need to be estimated for a specific headset and user settings during calibration. There are various calibration procedures that use manual interaction to collect 3D and 2D correspondences by manually aligning the world reference point to the 2D points displayed on the screen. For example, Tuceryan and Navad (Tuceryan, Mihran & Navab, Nassir. (2000). Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. 149~158. 10.1109 / ISAR.2000.880938) introduced SPAAM (Single Point Active Alignment Method). They proposed solving all projection parameters simultaneously after collecting the correspondences of individual 2D-3D points one by one. To do this, the user has to align a 2D symbol (circle or cross) to a 3D object. The headset and the 3D object are spatially tracked. When at least six correspondences are obtained, they are used to create and solve a system of linear equations as the initial estimated values of the parameters of matrix K. When tracking the eye position, these values may instead be automatically calculated at runtime.

[0103] Here, a method for calculating the parallax of each of the wearer's eyes will be described. This is then used to adjust the position of the image within the display. The calculation of the parallax can be seen in Figure 10, in which the following terms are defined as follows. O l = Position of the left eye O l = Position of the left eye P = Position of the target f = Distance between the eye and the display p l and p r = Principal points of the left and right eyes c l and c r = Centers of the left and right eye displays T = IPD x l and x r= Difference between the positions of p and c for each eye Z = Distance to the target

[0104] From Figure 10, the parallax d = x l -x r It can be seen that

Equation

[0105] Figure 11 shows a further flowchart of a method for adjusting the position of an image within the display of the headset of Figure 2, 3, or 5.

[0106] In step 501, the origin in the 3D virtual space is determined. This is set at a position that serves as a reference for determining spatial relationships.

[0107] In step 503, the IPD of the wearer is determined. This may be input manually, such as by the wearer, or may be determined by a gaze tracking sensor if the headset has one.

[0108] In step 505, the position and orientation of the headset with respect to the target are determined. This may be done by receiving light from the target by the camera and analyzing this light as described above.

[0109] In step 507, the position of the wearer's eyes is determined based on the IPD and the position of the headset. This can be determined by using a calibration procedure before the medical procedure. Alternatively, this may be done by using a gaze tracking sensor to determine the distance between the wearer's eyes and the headset.

[0110] In step 509, the parallax between the two eyes is determined based on the distance to the target and the IPD. The distance to the target is determined by a distance sensor and / or a camera.

[0111] In step 511, a 3D model shape of the target is constructed based on the light received by the camera. Step 511 can be executed in parallel while steps 503 - 509 are being executed.

[0112] Step 513 includes rendering the 3D model of the target. This is based on the 3D model constructed in step 511, and the 3D model is rendered based on the parallax calculated in step 509 and the position of the headset relative to the target area as in step 505.

[0113] In step 515, it is determined whether the position of the headset has changed. If it has changed, steps 503 - 513 are repeated. If it has not changed, the same 3D model is rendered based on the previous calculated values.

[0114] In step 517, it is determined whether the target has changed. If it has changed, step 519 is executed to update the 3D model of the target, and the updated 3D model is rendered in step 513.

[0115] By having described the embodiments of the present disclosure in detail, it will become apparent that modifications and changes are possible without departing from the scope of the embodiments of the present disclosure as defined in the appended claims. Since various changes can be made to the above structures, products, and methods without departing from the scope of the embodiments of the present disclosure, all matters included in the above description and shown in the accompanying drawings are intended to be construed as illustrative rather than in a limiting sense.

[0116] Figures 2, 3, 5, and 7 show determined, calculated, or known spatial relationships, but any other type of spatial relationship can be determined in accordance with the present invention. All that is necessary is to be able to determine the relationship between the headset, the wearer, and the target so that the position on the image within the display can be corrected based on each view of the wearer's eyes.

[0117] Above, it has been described that the creation of a 3D model from an image acquired by a camera can be performed using photogrammetry. Alternatively, triangulation can also be used. This may include laser triangulation. This requires projecting a laser beam onto the surface of the target. Details of the shape of the target are provided by the measure of the deformation of the laser beam. Alternatively, it may be necessary to use the time of flight of the laser beam. The laser beam is projected onto the surface of the target and collected by a sensor. The shape information of the surface is obtained by the travel time of the laser between its emission and reception. In these methods, a headset having one or more laser light sources that can be used to perform these techniques may be required. Any of the AR headsets shown in the figures may also have such a laser light source.

[0118] The eye-tracking sensor has been shown to be used in combination with a distance sensor and a camera, but the eye-tracking sensor can be applied to any of the AR headsets described. For example, the eye-tracking sensor can be applied to an AR headset that has only one camera and no distance sensor. Alternatively, the eye-tracking sensor can also be applied to an AR headset that has multiple cameras and no distance sensor.

[0119] The detection of images and related operations are described as being performed by a camera. However, any type of image sensor / image sensing device can also be used. The camera may be configured to detect still images or videos.

[0120] The AR headset is shown in the figure as having two displays attached to a housing with two arms (temples). However, it will be understood that the AR headset of the present invention is not limited to such an arrangement, and conventional means for attaching a head-mounted display (HMD) to the wearer are also conceivable. This may include using straps that are wrapped around and / or over the head to hold the headset in place. Alternatively, attachment means such as a hat that attaches the device to the entire upper part of the head can also be used.

[0121] The device has been considered in relation to use during fluorescence-based guided medical procedures. However, the device can be used in any type of medical procedure aimed at detecting radiation from a patient's body or tissue to generate an image. It can also be used outside of medical procedures. For example, applications will be found in other industries where it is necessary to correct the position of the AR image on the display to match the wearer's view. Such industries could be architecture or construction.

[0122] The processor can be a processor for executing instructions within a data processing device. The instructions can be stored, for example, in memory. The processor can include one or more processing units (e.g., a multi-core configuration) for executing the instructions. The instructions can be executed within various different operating systems on the data processing device, such as UNIX (registered trademark), LINUX, Microsoft Windows (registered trademark), etc. More specifically, the instructions can bring about various data operations (e.g., create, read, update, and delete procedures) on the data stored in memory. It should also be understood that various instructions can be executed during initialization at the start of a computer-implemented method. There are operations necessary for performing one or more of the methods described herein, as well as operations that are more general and / or specific to a particular programming language (e.g., C, C#, C++, Java (registered trademark), or other suitable programming languages, etc.).

Claims

1. An augmented reality (AR) system for use in a medical procedure, comprising: A camera configured to detect light from a target; A near-eye display positioned between the wearer's eyes and the target, the near-eye display being configured to display an image of the target based on the light detected by the camera such that the image overlaps the wearer's view of the target; and An AR headset comprising a distance sensor configured to determine a distance between the headset and the target throughout the medical procedure; A processor configured to determine a disparity between the image of the target acquired by the camera and the wearer's view of the target based on the value of the distance measured by the distance sensor and the position of the wearer's eyes; Adjust the position of the image on the display based on the determined disparity such that the image coincides with the wearer's view of the target; and Repeat the determination of the disparity and the adjustment of the position of the image throughout the medical procedure to account for changes in the distance measured by the distance sensor throughout the medical procedure. An AR system.

2. The processor is configured to: Assign a position in space to act as a fixed reference point; Generate a 3D model of the target based on the light detected by the camera; Determine the position and orientation of the target relative to the fixed reference point based on the distance measured by the distance sensor; Determine the position of the wearer's eyes relative to the fixed reference point; and Determine the position and orientation of the headset relative to the fixed reference point, Thereby further configured to determine the disparity between the image of the target acquired by the camera and the wearer's view of the target. The AR system according to claim 1.

3. The AR system according to claim 2, wherein the position and orientation of the headset relative to the fixed reference point are the position and orientation of at least one of the display, the distance sensor, and the camera.

4. The processor is configured to: Set the position of the 3D model of the target relative to the fixed reference point, Render the 3D model of the target to form the adjusted image based on the determined positions and orientations of the target and the headset, and the position of the wearer's eyes, By being configured to display the adjusted image on the display, The AR system according to claim 2 or 3, further configured to adjust the position so that the position of the image on the display is corrected based on the determined discrepancy.

5. The AR headset further comprises an eye-tracking sensor, the eye-tracking sensor being configured to continuously determine the position of the wearer's eyes throughout the medical procedure, whereby, in the repetition of the determination of the discrepancy and the adjustment of the position of the image throughout the medical procedure, changes in the position of the wearer's eyes throughout the medical procedure are taken into account. The AR system according to any one of claims 1 to 4.

6. The processor is, configured to determine the parallax of the wearer's eyes from the determined distance and the position of the wearer's eyes, The AR system according to any one of claims 1 to 5, further configured to determine the discrepancy between the image of the target acquired from the camera and the wearer's view of the target.

7. The distance sensor is a time-of-flight distance sensor, or a simultaneous localization and mapping (SLAM) sensor, or a visual SLAM (vSLAM) sensor. The AR system according to any one of claims 1 to 6.

8. Further comprising a light source, the light source being configured to emit light that is incident on the target and then detected by the camera. The AR system according to any one of claims 1 to 7.

9. The AR headset comprises the light source. The AR system according to claim 8.

10. The light is near-infrared light. The AR system according to any one of claims 1 to 9.

11. The AR headset comprises the processor. The AR system according to any one of claims 1 to 10.

12. The AR system according to any one of claims 1 to 11, wherein the headset includes a plurality of cameras configured to detect the excited light.

13. The AR system according to any one of claims 1 to 12, wherein the camera includes the distance sensor.

14. A method for adjusting the position of an image in an augmented reality (AR) system for use in a medical procedure, the AR system including an AR headset and a processor, the method including: detecting light excited from a target; determining a distance between the headset and the target throughout the medical procedure; determining a discrepancy between an image of the target obtained from the camera and the wearer's view of the target based on the determined value of the distance and the position of the wearer's eyes; and adjusting the position such that the position of the image on the near-eye display is corrected based on the determined discrepancy. displaying, on the near-eye display positioned between the wearer's eyes and the target, the image of the target based on the detected light such that the image overlaps the wearer's view of the target of the headset; A method including the above.

15. A non-transitory computer-readable medium configured to perform the steps of claim 14 when executed on a processor.

Images