Augmented Reality Head-Mounted Device for Medical Imaging

Abstract

The present invention discloses a system (100) for a medical procedure. The system (100) includes a head-mounted display (2) and a processor (12). The head-mounted display (2) includes cameras (6a, 6b), near-eye displays (4a, 4b), and distance sensors (10a, 10b). The processor (12) is configured to adjust the position of an image obtained by the cameras (6a, 6b) on the displays (4a, 4b) throughout the medical procedure based on changes in the distances measured by the distance sensors (10a, 10b).

Description

Technical Field

[0001] The present invention relates to an augmented reality system for medical procedures. Background Art

[0002] Fluorescence-based intraoperative surgical guidance is becoming a widely used procedure. This growth has been particularly focused on clinical methods using indocyanine green (ICG) as a fluorescent label that can be detected in the near-infrared (NIR) spectrum.

[0003] Some medical imaging devices have been commercialized to allow ICG-based guidance during surgery. This method is used for blood flow assessment, vascular patency, perfusion assessment (especially in reconstructive and bypass surgeries), lymphatic imaging, and as a surgical procedure for sentinel lymph node identification and mapping. Further research efforts are directed at the potential of imaging molecular tracers in reflecting the neovascular, structural, metabolic, immune, or genetic characteristics of tissues. The application rate of ICG in medical procedures is growing rapidly, and the scope of application is expanding rapidly.

[0004] This process involves injecting a fluorophore before surgery. Then, near-infrared light is shone on the target area, exciting the molecular ligand, and in response, the molecular ligand emits light of a specific wavelength. Then, a camera sensitive to the spectral range of this light is used to detect the light to form an image clearly indicating the target tissue.

[0005] This method allows for a significant improvement in the detection and removal of primary tumor nodules and chronic metastases of sentinel lymph nodes for various cancer types (such as lung cancer surgery). Other uses include the use of mastectomy or lumpectomy in breast cancer surgery and planned sentinel lymph node biopsy procedures.

[0006] Current systems are based on a mobile camera unit that acquires the emitted light and displays the detected image on an external screen next to the operating table. During the surgery, the surgeon has to shift the focus away from the patient being operated on and subjectively compare the display data acquired from different angles with the real patient's body in the real-world view. This continuous comparison operation makes fluorescence imaging cumbersome, especially limiting its ability to accurately map the fluorescently labeled tissue, which shows what the surgeon can see. In this latter subjective step, even with excellent professional training, a significant amount of accuracy, integrity, focus, and time efficiency is lost.

[0007] Preferably, there is a way to more easily and accurately correlate the view of the target tissue detected by fluorescence imaging with the surgeon's real-world view of the patient. This has the potential to make the process faster and less prone to human error.

[0008] US 2019 / 289284 A1 discloses a mediated reality visualization system that includes a head-mounted display assembly that includes a display device and an imaging component that is separate and spaced from the head-mounted display assembly configured to capture light field data. US 2019 / 310705 A1 discloses an image processing method for a head-mounted display that involves adjusting the display position such that the image presented by the physical display corresponds to the user's interpupillary distance. WO 2020 / 023672 A1 discloses a head-mounted display for rendering three-dimensional virtual objects that appear to be located in the user's surrounding environment. Summary of the Invention

[0009] According to one aspect, there is provided an augmented reality (AR) system for a medical procedure. The AR system includes: an AR head-mounted device and a processor. The AR head-mounted device includes: a camera configured to detect light from a target; a near-eye display located between the wearer's eyes and the target, the near-eye display configured to display an image of the target based on the light detected by the camera such that the image overlays the wearer's view of the target; and a distance sensor configured to determine the distance between the AR head-mounted device and the target throughout the medical procedure. The processor is configured to: determine a mismatch between the image of the target obtained from the camera and the wearer's view of the target based on the distance value measured by the distance sensor and the position of the wearer's eyes; adjust the position of the image on the display to correct for the determined mismatch such that the image matches the wearer's view of the target; and repeat the determination of the mismatch and the adjustment of the image position throughout the medical procedure to account for changes in the distance measured by the distance sensor throughout the medical procedure.

[0010] The system provides the wearer performing the medical procedure with an image generated by light from the target within their direct line of sight by displaying the target image in the display of the augmented reality device. This eliminates the need for the wearer to adjust their gaze to switch between viewing an image on an external display and the real-world view of the patient.

[0011] The difference between the position of the wearer's eyes and the distance to the target can adjust the difference between views. This is because the field of view of the camera is different from the wearer's field of view of the target. Throughout the process, the distance sensor performs this correction by measuring the distance between the head-mounted device and the target. This allows for the dynamic update of the position of the enhanced image on the display so that the displayed image is in the correct position and thus overlaps with the wearer's real-world view of the target. This ensures that the enhanced image in the display matches the accurate position of the emitting target in the wearer's field of view.

[0012] This can provide a precise mapping between the enhanced generated image and the actual patient for a professional medical personnel wearing the head-mounted device. This can achieve higher precision during a surgical operation on the patient.

[0013] Preferably, the processor is further configured to determine the mismatch between the target image obtained from the camera and the wearer's target view by the following steps: specifying a position in space as a fixed reference point; generating a three-dimensional (3D) model of the target based on the light detected by the camera; determining the position and orientation of the target relative to the fixed reference point based on the distance measured by the distance sensor; determining the position of the wearer's eyes relative to the fixed reference point; determining the position and orientation of the head-mounted device relative to the fixed reference point.

[0014] By determining the position and orientation of the head-mounted device, and the target and position of the wearer's eyes relative to the fixed reference point, their positions and orientations are converted into a fixed reference coordinate system relative to each other. This allows for the adjustment of the relative position of the target image on the display through the geometric relationship between the wearer's eyes and the head-mounted device relative to the target, via continuously updated target distance measurements. In addition to the position of the wearer's eyes, the orientation of the wearer's eyes can also be determined.

[0015] The fixed reference point can be located on the head-mounted device. For example, the fixed reference point can be located between the positions of the wearer's eyes. Or, the fixed reference point can be set as the point where the camera, distance sensor, or display is located on the head-mounted device. The fixed reference point does not have to be located on the head-mounted device and can also be a point outside the head-mounted device. The fixed reference point can be a position in 3D space represented by 3D coordinates. The positions and orientations of the wearer's eyes, target, and head-mounted device can be converted into 3D coordinates.

[0016] The position and orientation of the head-mounted device relative to a fixed reference point can be the position and orientation of at least one of the display, the distance sensor, and the camera. The positions of the display, the camera, and / or the distance sensor relative to each other are known. This allows displacements between each of the display, the camera, and / or the distance sensor to be taken into account when adjusting the image position. Since their positions relative to each other are static, their positions are known. The processor can receive these position values. For example, these position values can be stored in a memory. When the fixed reference point is a position on the head-mounted device, the distances between the display, the distance sensor, and the camera relative to the fixed reference point can be obtained by knowing the geometry of the head-mounted device.

[0017] Alternatively, if the values of these positions and orientations are not yet known, these positions and orientations can be measured by one or more sensors.

[0018] Preferably, the processor is further configured to adjust the position of the image on the display by the following steps to correct based on the determined mismatch: set the position of the target 3D model relative to the fixed reference point; render the 3D model of the target based on the determined position and orientation of the target and the head-mounted device and the position of the wearer's eyes to form an adjusted image; and display the adjusted image on the display.

[0019] In this way, the generated target 3D model is in the same reference coordinate system as the head-mounted device and the wearer's eyes. This enables the 3D model of the target to be rendered so that the image displayed on the display takes into account the positions of the head-mounted device, the target, and the wearer's eyes.

[0020] Preferably, the processor is further configured to determine the parallax of the wearer's eyes. Parallax or binocular parallax is the difference between the image projection points in the wearer's two eyes. The parallax can be determined based on the determined distance to the target. The parallax can also be determined based on the position of the wearer's eyes. The position of the wearer's eyes can be determined based on the interpupillary distance (IPD) and / or the distance between the head-mounted device and the wearer's eyes. Preferably, the processor is further configured to determine the parallax of the wearer's eyes according to the determined distance and the position of the wearer's eyes, so as to determine the mismatch between the target image obtained from the camera and the wearer's target view.

[0021] The distance to the target is inversely proportional to the parallax. By obtaining the distance to the target and the position of the wearer's eyes, the parallax of each eye can be determined. This enables the alignment of the image generated in the display to be updated so that the wearer's target view matches the target view of each of the wearer's eyes. For each of the wearer's eyes, the adjustment of the image in the display can be different.

[0022] In some aspects, the AR head-mounted device may further include an eye tracking sensor configured to continuously determine the position of the wearer's eyes throughout the medical procedure. In this way, when repeatedly determining mismatches and adjusting the image position throughout the medical procedure, the changes in the position of the wearer's eyes throughout the medical procedure are taken into account.

[0023] In this way, the position of the wearer's eyes can be tracked throughout the medical procedure and used to update the position of the generated image on the display. Throughout the medical procedure, the position of the wearer's eyes is not fixed, and their view and gaze change throughout the procedure. By continuously determining the position of the wearer's eyes and their position relative to the target and the head-mounted device, the positioning accuracy of the generated image on the wearer's field of view can be improved. The eye tracking sensor may also be configured to determine the orientation of the wearer's eyes throughout the medical procedure.

[0024] The eye tracking sensor may also determine the focus of the wearer's eyes, i.e., the position on which they focus at any given time. In this way, the generated image can always be displayed at the focus position in the wearer's target view.

[0025] The number of eye tracking sensors may be multiple. For example, a single sensor may track each eye. Alternatively, a single tracking sensor may track the positions of both eyes.

[0026] The eye tracking sensor may be an infrared (IR) light source that scans the position of each eye to determine its position. The infrared light source may be in the form of a light-emitting diode (LED) or a laser. Alternatively, the eye tracking sensor may be an electro-oculography eye tracking sensor. The electro-oculography eye tracking sensor uses electrodes placed around the eyes to measure the movement of the eyes.

[0027] The processor may receive the position of the wearer's eyes. Preferably, the processor is further configured to obtain the position of the wearer's eyes by acquiring the pupillary distance of the wearer's eyes.

[0028] The pupillary distance is the distance between the pupils of each eye in the wearer's eyes. Each user has their unique pupillary distance. By acquiring the pupillary distance, it can be used to determine the relative position of the wearer's eyes with respect to the head-mounted device and / or a fixed reference point. The pupillary distance can also be used to determine the parallax of the wearer's eyes. This helps to position the generated image at the correct position on the display, covering each eye in the wearer's eyes in the target view of each eye.

[0029] The processor can automatically obtain the interpupillary distance. This can be achieved by using an eye-tracking sensor. Alternatively, the wearer can manually provide the interpupillary distance to the processor. For example, the wearer or someone else may have manually measured the interpupillary distance.

[0030] Alternatively, the position of the wearer's eyes can be determined by methods other than using the interpupillary distance.

[0031] As described above, the position of the wearer's eyes can be determined using an eye-tracking sensor. The position of the wearer's eyes can be the position of the wearer's eyes relative to a fixed reference point such as a point on the head-mounted device. Alternatively, the position of the wearer's eyes relative to the fixed reference point can be determined through a calibration process. The calibration process can be performed before the medical procedure or if it is determined during the medical procedure that further calibration is needed. This calibration may involve the wearer looking at an external marker. Then adjustments can be made until the image position of the marker matches the wearer's view of the marker. This may involve the wearer manually adjusting the image generation parameters to move the image of the marker to cover their view of the marker. The marker can be any type of marker, including a point, a line, or a reference pattern with a known shape. This calibration enables the processor to determine the position and / or orientation of the wearer's eyes relative to the fixed reference point.

[0032] Preferably, the camera can include a distance sensor. Alternatively, the camera and the distance sensor can be separate sensors.

[0033] The distance sensor can be a time-of-flight distance sensor and / or the sensor can be a depth sensor. Alternatively, the distance sensor can be a Simultaneous Localization and Mapping (SLAM) sensor, a Visual Simultaneous Localization and Mapping (vSLAM) sensor, a dot-marker pattern sensor, or a sensor using the same principle as the Kinect device. The distance sensor can be a sensor only for determining distance. Alternatively, the distance sensor can be a camera configured to perform the role of a distance sensor. For example, the camera and the distance sensor may be the same. In this way, the camera acts as both a camera and a distance sensor. Alternatively, multiple cameras can act as distance sensors.

[0034] Preferably, the AR system further includes a light source configured to emit light such that the light is incident on the target and then detected at the camera.

[0035] The light source can be configured to emit light towards a target such that when subsequently detected by a camera, the light forms an image representative of the target or a part of the target. The light source can be a fluorescent light source. Alternatively, the light source can form an image by reflection of light from the target.

[0036] The AR head-mounted device can include a light source. Placing the light source on the head-mounted device can enable the camera and the light source to be focused on the same target area. This ensures that the wearer can control the illumination of the light on the target. Alternatively, the light source can be an external light source that is not located on the head-mounted device.

[0037] Preferably, the light is near infra-red (NIR) light. The medical procedure can be a fluorescence-based guidance procedure. NIR light can be used for fluorescence-based guidance procedures. The light source can be configured to emit light within these frequency bands. The wavelength of the NIR light can be on the order of 780 nm. This is the excitation range of ICG. However, depending on the molecular marker used, other wavelengths of NIR can be used.

[0038] Alternatively, the light can be visible light or infrared light.

[0039] The camera can be configured to detect NIR light. The camera can be configured to detect within the wavelength range transmitted by the molecular marker used. In the case where the molecular marker is ICG, the wavelength can be from 810 nm to 860 nm, depending on the type of tissue. The camera can be configured to detect light in the same band as the light source.

[0040] The camera can continuously detect light throughout the medical procedure. In other arrangements, the camera can be arranged to acquire light at intervals during the medical procedure. In this way, the image generated in the AR display can be updated during the procedure.

[0041] In other embodiments, an image detected by the camera can be captured once during the medical procedure. During the entire procedure, this image may not be updated, and only its position on the near-eye display is updated to reflect changes in the position of the head-mounted device. In this case, the target image captured by the camera is expected not to change.

[0042] The processor can be further configured to convert the light detected by the camera into an image visible to the user. When the light emitted by the target is in the IR or NIR band, the wearer's eyes generally cannot see this light. This occurs when the light is excited by fluorescence from the target. By converting this light and displaying the image on the near-eye display, the wearer can see the light reflected by the target. This image enhances the wearer's field of view and provides information that they could not previously see in their line of sight.

[0043] The AR head-mounted device can include a processor. In this way, there is no need to rely on wires or physical connections between the external processor and the head-mounted device.

[0044] Alternatively, the processor may not be located on the head-mounted device. This can reduce the weight of the head-mounted device. For example, the processor may be located on a server or computing system external to the head-mounted device. The processor may be connected to the head-mounted device via a wired connection. Or, the processor may be connected to the head-mounted device via a wireless connection (such as Wireless Fidelity (WiFi), Bluetooth connection, or other types of radio frequency connections).

[0045] In some aspects, the head-mounted device may include multiple cameras configured to detect excited light. Having multiple cameras to detect excited light enables detecting images from two different viewpoints. This can improve the generated target model. Additionally, it can provide a more accurate estimate of the position of the head-mounted device relative to the target.

[0046] The near-eye display may be a single display. Or, the near-eye display may be two displays, with one display showing an image for one eye. The near-eye display may be a waveguide. Or, the display may be a beam splitter display or a laser reflection display. The display may use a mirror to project an image into the wearer's field of view. The display may be made of glass and / or plastic. In this way, the display is transparent. The near-eye display may also be a lens. Or, the near-eye display may be a light beam that projects an image into the wearer's eye, causing the image to be displayed on the retina. In this way, the near-eye display may be a virtual retinal display.

[0047] The display and / or camera may include one or more filters. The filters may be configured to improve the detection of light of a specific wavelength from the target while removing other signals of unwanted wavelengths. Or, alternatively or additionally, the filter may be located in front of the light source.

[0048] According to another aspect, there is provided a method for adjusting the position of an image in an augmented reality (AR) system for a medical procedure, the AR system including an AR head-mounted device and a processor, the method including: detecting light excited from a target; determining the distance between the head-mounted device and the target throughout the medical procedure; displaying a target image based on the detected light on a near-eye display located between the wearer's eyes and the target, such that the image covers the wearer's view of the target through the following steps: determining a mismatch between the target image obtained from the camera and the wearer's view of the target, and the position of the wearer's eyes, based on the determined distance value; and adjusting the position of the image on the display such that the position is corrected based on the determined mismatch.

[0049] According to another aspect, there is provided a non-transitory computer-readable medium that, when executed on a processor, is configured to perform the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Shows an exemplary augmented reality (AR) system according to the present invention;

[0051] Figure 2 Shows a top - down schematic view of an exemplary AR system according to the present invention;

[0052] Figure 3 Shows a top - down schematic view of another exemplary AR system according to the present invention;

[0053] Figure 4 Shows the steps of using an exemplary AR head - mounted device as Figure 2 or Figure 3 shown to display a target 3D image in a flowchart;

[0054] Figure 5 Shows a top - down schematic view of another exemplary AR system according to the present invention;

[0055] Figure 6 Shows the steps of using an exemplary AR head - mounted device as Figure 5 shown to display a target 3D image in a flowchart;

[0056] Figure 7 Shows a top - down schematic view of another exemplary AR system according to the present invention;

[0057] Figure 8 Shows the steps of using an exemplary AR head - mounted device as Figure 7 shown to display a target 3D image in a flowchart;

[0058] Figure 9 Shows a reference for a calibration process for using an exemplary AR head - mounted device to display a target 3D image;

[0059] Figure 10 Shows the geometric relationship between a head - mounted device according to the present invention and a target for calculating a difference; and

[0060] Figure 11 Shows the steps of using an exemplary AR head - mounted device as Figure 2 、 Figure 3 or Figure 5 shown to display a 3D image of a target in a further flowchart. Detailed Description

[0061] Figure 1 Shows an augmented reality (AR) system 100 according to an embodiment of the present invention. The AR system 100 includes an AR head - mounted device 2 and a processor 12.

[0062] The augmented reality head mounted device 2 has two displays: a first display 4a and a second display 4b, wherein the first display 4a is used to display images to the right eye of the wearer of the head mounted device, and the second display 4b is used to display images to the left eye of the wearer of the head mounted device 2. The displays 4a and 4b are connected to the housing 16 of the head mounted device 2.

[0063] Two cameras 6a, 6b are located on the housing 16 of the head mounted device. The camera 6a is located above the first display 4a and the camera 6b is located above the second display 4b. The cameras 6a, 6b are capable of detecting near infrared (NIR) light.

[0064] The light source 8 is also located on the housing 16. The light source 8 is a NIR light source configured to emit NIR light. The light source 8 is located between the cameras 6a and 6b, but the light source 8 can be located anywhere on the AR headset 2. Alternatively, the light source 8 can be located outside the AR headset 2.

[0065] Two distance sensors 10a and 10b are located on the housing 16 of the head mounted device 2. The distance sensors are time-of-flight sensors configured to determine the distance from the head mounted device 2 to an object.

[0066] The head mounted device 2 also includes an eye tracking sensor 18. The eye tracking sensor is located on the side of the head mounted device facing the wearer's head. The eye tracking sensor is configured to determine the position of the eyes of the wearer of the head mounted device 2.

[0067] The processor 12 is located outside the AR head-mounted device 2. The processor can be a processor of a computer or other data processing device. The AR head-mounted device 2 is connected to the processor 12 via a cable 14. The cable 14 is used to send signals between the head-mounted device and the processor 12. For example, data obtained from the cameras 6a, 6b, the eye tracking sensor 18, and the distance sensors 10a, 10b can be sent to the processor 12 via the cable 14. The cable 14 is also used to transmit communication signals between the processor 12 and the head-mounted device 2 to control the cameras 6a, 6b, the distance sensors 10a, 10b, the light source 8, and the eye tracking sensor 18 to perform their functions.

[0068] Figure 2 , Figure 3 , Figure 5 and Figure 7 A top-down schematic diagram of an AR headset 2 according to the present invention is shown, each figure showing a headset 2 with a different sensor arrangement from one another. The headset 2 is shown for use in a fluorescence-based guidance process.

[0069] Figure 2 , Figure 3 , Figure 5 and Figure 7The features of the AR head-mounted device 2 shown in are described with the same reference numerals as the features shown in Figure 1 Also shown is the part of the target 20 (in this case the patient) for detecting the fluorescence image.

[0070] Figure 2 An AR head-mounted device with two cameras 6a and 6b is shown. Figure 2 The head-mounted device shown in does not have a separate distance sensor. In addition to detecting light from the target 20 to form an image, the two cameras 6a and 6b act as distance sensors.

[0071] Figure 2 A spatial relationship is shown, which is determined and used to adjust the position of the images generated in the displays 4a, 4b so that the position matches the wearer's view of the target 20.

[0072] The distances 22 between the target 20 and each of the cameras 6a, 6b are measured by the cameras 6a, 6b.

[0073] Since the AR head-mounted device 2 does not have an eye-tracking sensor, the distances 26 between each eye in the wearer's eyes and each of the displays 4a, 4b are determined by a calibration process. The calibration process also involves determining the distances 28 between each of the cameras 6a and 6b and each eye in the wearer's eyes 30a and 30b. The interpupillary distance (IPD) can also be determined 24 by the calibration process. Alternatively, the wearer can know the IPD and input it into the AR system.

[0074] From the geometry of the head-mounted device 2, the distance 32 between the two displays 4a and 4b and the distance 34 between the two cameras 6a, 6b are known. This allows determination of the distance 40 between the target 20 and the displays and the distance 38 between the target and the wearer's eyes 30a, 30b.

[0075] By determining each of these distances, they can be compared with the fixed reference point 36 on the head-mounted device 2. This enables the processor 12 to adjust the position of the image on the display so that for each eye the position of the image on the display matches the wearer's target view.

[0076] Throughout the process, the wearer can move their head relative to the patient 20. By continuously measuring the distance 22 throughout the process, the above calculations can be continuously performed to adjust the position of the image on the display throughout the process so that the image position matches the wearer's actual true view of the target.

[0077] Figure 3Shows an AR head-mounted device, which has two cameras 6a and 6b and two distance sensors 10a and 10b. In this case, the cameras 6a, 6b function to capture the light emitted from the target 20 to form images in the displays 4a, 4b. The distance sensors 10a, 10b function to determine the distance between the patient 20 and the head-mounted device 2. The distances obtained by the distance sensors 10a, 10b are as shown in Figure 3 42 in. The distance 22 between the target 20 and the cameras 6a, 6b can be calculated based on the determined distance value 42 and the known spatial relationship. Figure 3 The other spatial relationships shown are with Figure 2 similar reference numerals to the spatial relationship shown in Figure 2 and are determined in the same way as described with respect to Figure 3 the other spatial relationships shown.

[0078] Figure 2 and Figure 3 The steps performed by the AR system 100 in displaying the target image are as shown in Figure 4 .

[0079] In step 101, the pupil distance of the wearer's eyes is obtained through a calibration process. In step 103, the position and orientation of the wearer's eyes relative to the head-mounted device or any other point in the virtual 3D space are calculated and saved.

[0080] In step 105, using the known geometry of the head-mounted device, the positions and orientations of the head-mounted device and the cameras, distance sensors, eye-tracking sensors, and AR / Mixed Reality (MR) displays relative to each other are determined. This is based on the known geometry shown in Figure 2 and Figure 3 . This enables the positions and orientations of the head-mounted device and all its components and sensors relative to each other or any other point in the virtual 3D space to be determined in step 107.

[0081] Once the wearer places the AR head-mounted device on their head, steps 101 to 107 are performed at the start of the medical procedure. It can be considered that these determined values and spatial relationships do not change throughout the medical procedure, so there is no need to calculate these values further throughout the process.

[0082] Step 109 involves using distance sensors (such as shown in Figure 3 ) and / or camera sensors (such as shown in Figure 2The measured values obtained as shown are used to obtain the position and orientation of the head-mounted device relative to the target. The distance sensor can use time-of-flight or any known measurement type for determining distance. The cameras 6a, 6b can use vSLAM or any known method to determine distance through the image sensors. At step 111, the position and orientation of the target relative to the origin of the head-mounted device or any other point in the virtual 3D space are calculated and saved.

[0083] Step 113 involves obtaining light from the target through the camera to record an image of the surgical wound, so as to detect biomarkers of different wavelengths through fluorescence. At step 115, the geometry of the 3D model of the target area can be constructed and saved based on the light received by the camera relative to the local coordinates. The local coordinates can be the same point on the head-mounted device or the same point in the virtual space that determines other positions and orientations.

[0084] Photogrammetry can be used to create a 3D model based on the images obtained by the camera. This involves using computer vision and computational geometry algorithms to reconstruct a 3D object from two-dimensional (2D) captures.

[0085] At step 117, the origin of the virtual 3D space is determined. As Figure 2 and Figure 3 shown, this is the point 36 on the head-mounted device that is located between the two displays 4a and 4b. This virtual 3D space can be the same point as the one where the positions and orientations of the head-mounted device, the wearer's eyes, and the 3D model are determined in steps 103, 107, 111, 115. This results in the position and orientation of the wearer's eyes and the target relative to the origin of the virtual 3D space being transformed into the virtual 3D space at step 119.

[0086] Then, at step 121, the 3D model of the target in the virtual 3D space is rendered.

[0087] At step 123, the rendered 3D model is then displayed in the displays 4a, 4b of the AR head-mounted device 2. This enables the 3D model of the target to be automatically displayed in perspective for each eye of the wearer.

[0088] Steps 109, 111 are performed throughout the medical procedure. This can be continuous or at fixed time points. This is because the wearer's head and the AR head-mounted device 2 may move throughout the procedure. This will cause the values determined in step 109 to change throughout the procedure.

[0089] Steps 113 and 115 can also be performed throughout the medical procedure. This can be continuous or at fixed time points. This is because as the medical procedure progresses, the light detected by the camera may change throughout the procedure. Steps 109, 111, 113, and 115 can run in parallel throughout the medical procedure.

[0090] Therefore, steps 119, 121, and 123 can also be performed throughout the medical procedure to take into account the updated data obtained from steps 109 to 115.

[0091] Figure 5 Shown is the AR head-mounted device 2, which is the same as Figure 3 but this AR head-mounted device 2 also has two eye tracking sensors 18a and 18b. The same spatial relationship as shown in Figure 3 is shown in Figure 5 . However, the distances 44 between the eye tracking sensors 18a, 18b and the wearer's eyes 30a, 30b are measured using the eye tracking sensors 18a, 18b. This enables the accurate position and orientation of the wearer's eyes to be determined throughout the medical procedure. The distance 46 between the eye tracking sensors and the display is also determined. This can be determined using the eye tracking sensors 18a, 18b or can be a known spatial relationship based on the geometry of the head-mounted device.

[0092] The determination of the distance 44 and the tracking of the wearer's eyes can be performed throughout the medical procedure. This enables the target image in the display to be updated according to the movement of the wearer's eyes. This can provide a more precise match between the wearer's view of the target and the image displayed on the AR display, as the position of the wearer's eyes is known throughout the procedure. The eye tracking sensors can continuously determine the position of the wearer's eyes throughout the medical procedure or at fixed time intervals. The eye tracking sensors can determine the position of the wearer's eyes while measuring the distance to the target. This can be once every 0.5 s. Alternatively, it can be more frequent than once every 0.5 s. It can also be once every 1 s. Using eye tracking sensors can provide higher precision than not using eye tracking sensors, and since the changes in the movement of the wearer's eyes are taken into account, the eye tracking sensors can allow sub-centimeter precision. This is in contrast to Figure 2 and Figure 3 where head-mounted devices can achieve centimeter-level precision. In addition, if the position of the head-mounted device moves on the wearer's head, the eye tracking sensors can correct this movement by performing re-calibration. In this way, having eye tracking sensors eliminates the need for an initial calibration such as that performed using a calibration reference at the start of the procedure.

[0093] An eye-tracking sensor can use near-infrared technology along with a camera (or other type of optical sensor) to track the gaze of the wearer's eyes. This can involve using Pupil Center Corneal Reflection (PCCR). Alternatively, the eye-tracking sensor can utilize electro-oculography technology. This involves dry electrodes that measure the skin potential around the eyes. The microelectronic device interprets the electrical signals to calculate eye movements. The sampling rate may be approximately 256 samples per second, but this depends on the type of camera. Alternatively, any known type of eye-tracking technology can be used.

[0094] Figure 5 The steps performed by the AR system 100 when displaying the target image are as Figure 6 shown. Figure 6 Each step that is the same as the steps in Figure 4 is represented by the same reference numeral.

[0095] Figure 6 Steps 201 and 203 in Figure 4 are different from steps 101 and 103 in

[0096] Step 201 involves obtaining the IPD of the wearer's eyes, as well as the position and orientation of the wearer's eyes, from the measurements made by the eye-tracking sensor. Then, step 203 involves calculating and saving the focus, position, and orientation of the wearer's eyes relative to the head-mounted device or any other point in the virtual 3D space. This is based on the measurements made by the eye-tracking sensors 18a, 18b.

[0097] Figure 7 Shows an AR head-mounted device 2 without a distance sensor or an eye-tracking sensor. Figure 7 The AR head-mounted device 2 in

[0098] is an AR head-mounted device 2 that only has a single camera 6. Using such an AR head-mounted device requires additional calculations initially to obtain a 3D representation and position of the target. This requires the wearer to move relative to the target from multiple perspectives at the start of the medical procedure so that the camera can obtain multiple images of the target to reconstruct a 3D model of the target. Since the single camera 6 is the only sensor, the above process cannot be completed in real time throughout the medical procedure. Figure 7 It can be seen from Figure 2In contrast, the distance 34 between the camera and the fixed reference point 36 is also determined through calibration. This is additional data apart from the IPD 24, the distance 28 between the camera and the wearer's eyes, and the distance 26 between the wearer's eyes and the display, as Figure 2 shown. The distance 32 between the display and the fixed reference point 36 can be known from the geometry of the head-mounted device.

[0099] Figure 7 The steps performed by the AR system 100 in displaying the target image are as Figure 8 shown. Figure 8 Each step in Figure 4 that is the same as the steps in

[0100] is represented by the same reference numeral. Figure 4 As shown in

[0101] Figure 4 , in step 107, the position and orientation of the head-mounted device and the head-mounted device components are determined relative to the position in the head-mounted device or the 3D space. The position and orientation of the wearer's eyes are also determined in step 103. Figure 7 In step 315, a 3D model of the target area is generated based on the light received by the camera 6. Since there is only one camera, images are acquired from multiple different angles through a photometric algorithm to form the 3D model. Therefore, with Figure 4 the AR head-mounted device with a single camera, it is impossible to generate the 3D model in real time, and the target must be scanned through an initial process using the photometric algorithm to generate the 3D model. The position and orientation of the 3D model relative to the head-mounted device are set through manual calibration 309. The wearer sees the target on the scanned surface. This enables setting the position and orientation of the 3D generation model of the target relative to the head-mounted device or any other point in its virtual 3D space 311. Then, as

[0102] Figure 2 Figure 3 , Figure 7 and Figure 9 shown, when the AR head-mounted device 2 does not have an eye tracking sensor for determining the position of the wearer's eyes (such as the head-mounted device shown in Figure 9 ), it is necessary to perform a calibration process. This may involve the wearer viewing an external calibration reference 46, as Figure 9 shown. The calibration reference in Figure 9As shown. Also, the uncorrected image of the base reference pattern is shown as 50. Then adjustments are made until the image position of the reference pattern 50 matches the wearer's view of the reference pattern 48. This may involve the wearer manually adjusting the parameters of the image generation to move the image of the reference pattern 50 to cover the wearer's view of the reference pattern 48. This calibration enables the processor to determine the position of the wearer's eyes relative to a fixed reference point (such as a point on the head-mounted device).

[0103] Further details on how to correct the image alignment on the display will now be described. This involves examples of alternative algorithms and methods for how to achieve this correction and how to apply it.

[0104] To correct the images on the display based on the wearer's perspective, the parameters of the camera need to be determined, including the position, orientation, and optical characteristics of the camera. To correct the images, the 3D information of the target and the differences between the images displayed for each eye also need to be determined. The position and orientation of the camera are determined based on steps 109 and 111. The 3D information of the target is determined based on steps 113 and 115. The differences between the images displayed for each eye are detailed based on steps 101, 103, 201, 203, 105, 107, 109, and 111.

[0105] The parameters of the camera are determined based on the internal parameters representing the optical characteristics of the camera and the external parameters representing the position and orientation of the camera.

[0106] The internal parameters represent the optical characteristics, and the internal parameters can be estimated using the pinhole camera model. The internal parameters include the focal length of the camera, the aspect ratio of the plane (i.e., the display) onto which the camera view is projected, and the position of the image center (its principal point) where the optical axis truncates the image plane.

[0107] The internal characteristics of the pinhole camera model define the projection transformation from the 3D space of the display to the 2D coordinate space:

[0108]

[0109] where f is the focal length starting from the camera center and perpendicular to the display, and c u,v are the coordinates of the center of the display plane.

[0110] The position and orientation of the camera are determined by calculating the pose of the camera. This can be calculated using sensors on the camera (such as distance sensors). The pose is represented as:

[0111]

[0112] where R is a 3x3 rotation matrix representing the camera orientation and T is a translation vector representing the camera translation.

[0113] Based on the internal and external parameters of the pinhole camera model, 3D points can be mapped to 2D image coordinates. The following matrix transformation T cam shows this:

[0114] T cam = K * T pose

[0115] This is the product of the camera pose (i.e., external parameters) and the projection matrix (i.e., internal parameters).

[0116] In a physical implementation, the pinhole camera model is not always exact because, at runtime, the possible positions of the user's eyes relative to the display are different. Therefore, the following initial calibration is required. Assume we know the translation vector t eye = [x, y, z] t of the eyes relative to the display. We can define the internal matrix for each eye as follows:

[0117]

[0118] The vector t eye depends on the current position of the user's eyes relative to the display. Therefore, when the headset is repositioned on the user's head or when another user wears the headset and needs to recalibrate, the parameters of the internal matrix will change. If this is the case, the old matrix K0 based on the old eye position t0 = [x0, y0, z0] t can be updated to the new internal matrix K1.

[0119]

[0120] During the calibration run, the initial internal and external matrices for a specific head-mounted device and user configuration should be estimated. There are different calibration procedures that use manual interaction to collect 3D and 2D correspondences by manually aligning world reference points with 2D points displayed on the screen. For example, Tuceryan and Navad (Tuceryan, Mihran & Navab, Nassir. (2000). Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. 149-158. 10.1109 / ISAR.2000.880938. (Tuceryan, Mihran and Navab, Nassir (2000)'s "Single Point Active Alignment Method (SPAAM) for Optical See-Through Head-Mounted Display Calibration for Augmented Reality", pp. 149–158, 10.1109 / ISAR.2000.880938)) introduced SPAAM (Single Point Active Alignment Method). They proposed collecting individual 2D-3D point correspondences one by one and then solving for all projection parameters simultaneously. To do this, the user must align 2D symbols (circles or crosses) with 3D objects. Spatial tracking of the head-mounted device and the 3D object is performed. Once we have at least 6 correspondences, they are used to create and solve a system of linear equations for an initial estimate of the parameters of matrix K. To track the eye position, these values can be automatically calculated at runtime.

[0121] Now, we will describe how to calculate the disparity for each eye in the wearer's eyes. This is then used to adjust the position of the image on the display. The calculation of the disparity is as Figure 10 shown, Figure 10 The following terms in

[0122] O l = Left eye position

[0123] O l = Right eye position

[0124] P = Target position

[0125] f = Distance between the eye and the display

[0126] p l and p r = Principal point of the left eye and principal point of the right eye

[0127] c l and c r = Center of the left eye display and center of the right eye display

[0128] T = IPD

[0129] x l and x r = the difference between the p and c positions of each eye

[0130] Z = the distance to the target

[0131] From Figure 10 It can be seen that the difference d = x l - x r :

[0132]

[0133] Therefore, the difference is inversely proportional to the distance to the target. By knowing the distance to the target and the positions of the wearer's eyes throughout the medical procedure, the alignment of the generated 3D model (image) of the target can be updated so that it is in the correct alignment with the wearer's binocular vision. This is further based on the estimated camera parameters as described above.

[0134] Figure 11 Shows the adjustment Figure 2 、 Figure 3 or Figure 5 A further flowchart of a method for the image position in the display of a head-mounted device.

[0135] In step 501, the origin in the 3D virtual space is determined. This is set as the position for determining the spatial relationship.

[0136] In step 503, the wearer's IPD is determined. This can be manually input, for example, by the wearer. Alternatively, if the head-mounted device has an eye-tracking sensor, it can be determined by the eye-tracking sensor.

[0137] In step 505, the position and orientation of the head-mounted device relative to the target are determined. This can be achieved by the camera receiving light from the target and analyzing the light as described above.

[0138] In step 507, the positions of the wearer's eyes are determined based on the IPD and the position of the head-mounted device. This can be determined using a calibration process before the medical procedure. Alternatively, the distance between the wearer's eyes and the head-mounted device can also be determined by using an eye-tracking sensor.

[0139] In step 509, the difference between the binocular vision is determined based on the distance to the target and the IPD. The distance to the target is determined by a distance sensor and / or a camera.

[0140] In step 511, the geometry of the 3D model of the target is constructed based on the light received at the camera. Step 511 can be executed in parallel while steps 503 to 509 are being performed.

[0141] Step 513 involves the 3D model of the rendering target. This is based on the 3D model constructed in step 511 and is rendered based on the difference calculated in step 509 and the position of the head-mounted device relative to the target area in step 505.

[0142] In step 515, it is determined whether the position of the head-mounted device has changed. If it has changed, steps 503 to 513 are repeated. If it has not changed, the same 3D model is rendered based on the previous calculated values.

[0143] In step 517, it is determined whether the target has changed. If it has changed, step 519 is performed to update the 3D model of the target with the updated 3D model rendered in step 513.

[0144] After various aspects of the present disclosure have been described in detail, it is obvious that modifications and variations can be made without departing from the scope of the various aspects of the present disclosure defined in the appended claims. Since various changes can be made to the above-described structures, products, and methods without departing from the scope of the various aspects of the present disclosure, all matters included in the above description and shown in the drawings should be construed as illustrative rather than restrictive.

[0145] Although Figure 2 , Figure 3 , Figure 5 and Figure 7 show determined, calculated, or known spatial relationships, any other type of spatial relationship can be determined according to the present invention. All that is required is that the relationship between the head-mounted device, the wearer, and the target can be determined so that the position on the image in the display can be corrected based on the view of each eye in the wearer's eyes.

[0146] As described above, photogrammetry can be used to create a 3D model based on the images obtained by the camera. Alternatively, triangulation can be used. This can include laser triangulation. This involves projecting a laser beam onto the target surface. Measuring the deformation of the laser rays provides detailed information about the target geometry. Alternatively, it can involve using the time of flight of the laser beam. The laser beam is projected onto the target surface and then collected by the sensor. The propagation time of the laser from emission to reception gives the geometric information of the surface. These methods can involve a head-mounted device having one or more laser sources that can be used to perform these techniques. Any AR head-mounted device shown in the figures can have such a laser source.

[0147] Although the eye tracking sensor is shown in combination with the distance sensor and the camera, the eye tracking sensor can be applied to any of the AR head-mounted devices described above. For example, the eye tracking sensor can be applied to an AR head-mounted device that has only a single camera and no distance sensor. Alternatively, the eye tracking sensor can be applied to an AR head-mounted device that has multiple cameras and no distance sensor.

[0148] The detection of the image and related actions are described as being performed by the camera. However, any type of image sensor / image sensing device can be used. The camera can be configured to detect still images or videos.

[0149] The AR head-mounted device is shown in the figure as having two displays, and the two displays are attached to the housing by two arms (brackets). However, it should be understood that the AR head-mounted device of the present invention is not limited to this arrangement, and any conventional way of attaching the head-mounted display (HMD) to the wearer can be envisioned. This may include using straps that go around and / or over the head to secure the head-mounted device in place. Alternatively, an attachment that attaches the device to the top of the entire head, such as a hat, can be used.

[0150] The use of the device in a fluorescence-based medical guidance process is discussed. However, the device can be used in any type of medical process that aims to detect radiation from a patient's body or tissue to generate an image. The device can also be used outside of medical processes. For example, the device can be applied in other industries that require correcting the position of the AR image on the display so that the AR image matches the wearer's view. Such industries may be the construction industry or the structural industry.

[0151] The processor can be a processor for executing instructions in a data processing device. For example, the instructions can be stored in the memory. The processor can include one or more processing units for executing the instructions (e.g., in a multi-core configuration). These instructions can be executed in various different operating systems such as UNIX, LINUX, Microsoft and the like on the data processing device. More specifically, these instructions may perform various data operations on the data stored in the memory (e.g., create, read, update, and delete processes). It should also be understood that various instructions can be executed during initialization when the computer-implemented method is started. In order to execute one or more of the methods described herein, some operations may be required, while other operations may be more general and / or specific to a particular programming language (e.g., C, C++, Java, or other suitable programming languages, etc.).

Claims

1. A system for a medical procedure, comprising: A head-mounted display, the head-mounted display comprising: A camera configured to detect light from a target; A near-eye display located between the wearer's eyes and the target, the display configured to display an image of the target based on the light detected by the camera such that the image covers the wearer's view of the target; and A distance sensor configured to determine the distance between the head-mounted display and the target throughout a medical procedure; and A processor configured to: Determine a mismatch between the image of the target obtained from the camera and the wearer's view of the target based on the distance value measured by the distance sensor and the position and orientation of the wearer's eyes relative to the head-mounted display; Adjust the position of the image on the display so as to correct the position based on the determined mismatch, thereby causing the image to match the wearer's view of the target; and Repeat the determination of the mismatch and the adjustment of the position of the image throughout the medical procedure to account for changes in the distance measured by the distance sensor throughout the medical procedure.

2. The system according to claim 1, wherein, The processor is further configured to determine a mismatch between the image of the target obtained from the camera and the wearer's view of the target by: Taking the position of the head-mounted display as a fixed reference point; Generating a 3D model of the target based on the light detected by the camera; Determining the position and orientation of the target relative to the fixed reference point based on the distance measured by the distance sensor; Determining the position and orientation of the wearer's eyes relative to the fixed reference point; Determining the position and orientation of the head-mounted display relative to the fixed reference point.

3. The system according to claim 2, wherein, The position and orientation of the head-mounted display relative to the fixed reference point is the position and orientation of at least one of the display, the distance sensor, and the camera.

4. The system according to claim 2 or 3, wherein, The processor is further configured to adjust the position of the image on the display so as to correct the image based on the determined mismatch by: Setting the position of the 3D model of the target relative to the fixed reference point; Rendering the 3D model of the target based on the determined position and orientation of the target and the position and orientation of the head-mounted display and the wearer's eyes to form an adjusted image; And Displaying the adjusted image on the display.

5. The system according to any one of claims 1 to 3, wherein, The head-mounted display further includes an eye tracking sensor configured to continuously determine the position of the wearer's eyes throughout the medical procedure such that changes in the position of the wearer's eyes throughout the medical procedure are taken into account when repeatedly determining the mismatch and adjusting the position of the image throughout the medical procedure.

6. The system according to any one of claims 1 to 3, wherein, The processor is further configured to determine a mismatch between the image of the target obtained from the camera and the wearer's view of the target by: Determining the difference in the wearer's eyes based on the determined distance and the position of the wearer's eyes.

7. The system according to any one of claims 1 to 3, wherein, The distance sensor is a time-of-flight distance sensor, or a simultaneous localization and mapping (SLAM) sensor, or a visual SLAM sensor.

8. The system according to any one of claims 1 to 3, further comprising: A light source configured to emit light that is incident on the target and then detected by the camera.

9. The system according to claim 8, wherein, The head-mounted display includes a light source.

10. The system according to any one of claims 1 to 3, wherein, The light is near-infrared light.

11. The system according to any one of claims 1 to 3, wherein, The head-mounted display includes the processor.

12. The system according to any one of claims 1 to 3, wherein,The head-mounted display includes a plurality of cameras configured to detect the excited light.

13. The system according to any one of claims 1 to 3, wherein, The camera includes the distance sensor.

14. A method for adjusting the position of an image in a system during a medical procedure, the system including a head-mounted display and a processor, the head-mounted display including a camera configured to detect light from a target, the method including: Detecting, with the camera, light excited from the target; Determining, throughout the medical procedure, the distance between the head-mounted display and the target; Displaying, on a near-eye display located between the wearer's eyes and the target, an image of the target based on the detected light such that the image overlays the wearer's head-mounted display view of the target by: Determining a mismatch between the image of the target obtained from the camera and the wearer's view of the target based on the value of the determined distance and the position and orientation of the wearer's eyes relative to the head-mounted display; And Adjusting the position of the image on the display to correct the image based on the determined mismatch.

15. A non-transitory computer-readable medium, which when executed on a processor, is configured to perform the steps in claim 14.

Images