Device with image detector and image projector

Abstract

A projection apparatus (102) for image projection of an image on a subject (114). The apparatus (102) includes an image detector (104) having a detector field of view, where the image detector (104) is configured to detect an input image of the subject 5 (114). The processor (204) is configured to generate an output image based on the input image, where the output image includes content representing at least a portion of the content of the input image. An image projector (106) has a projector field of view, where the projector field of view overlaps the detector field of view, and where the image projector (106) is configured to project the output image on a target surface (208) 10 of the subject (114).

Description

TECHNICAL FIELD

[0001] The present invention relates to image projection, more particularly to image projection using a single field of view for both image detection and image projection. In particular, the present invention relates to a projection device for image projection of an image on a subject. The present invention further relates to a method of generating a projected image. BACKGROUND

[0002] For a surgical procedure, it is important to locate features in a patient's body that can be visible on e.g. medical images.

[0003] The application WO 2016 / 127173 A1 discloses an optical imaging system that utilizes a three-dimensional (3D) light scanner to capture topographical information, color reflectance information, and fluorescence information of a target object being imaged, such as a surgical patient. The system also utilizes the topographical information of the target object to perform an image mapping process to project captured intraoperative images back onto the target object with enhanced resolution or clarity.

[0004] The prior art document has the disadvantage that the intraoperative images have to be calibrated using the topographical information of the target before the intraoperative images are projected back onto the target.

[0005] The patent US 9510914 B2 discloses providing surgical guidance and image registration, wherein three-dimensional image data associated with a subject or patient is registered to topographical image data obtained using a surface topography imaging device.

[0006] The patent US 9646423 B1 discloses a method for providing augmented reality in a minimally invasive surgical procedure, the method comprising: capturing preoperative image data of an internal organ of a patient, capturing intraoperative image data of the internal organ with an endoscope during a surgical procedure, registering the preoperative image data and the intraoperative data in real-time during the surgical procedure, tracking a position and orientation of the endoscope during the surgical procedure, and augmenting the intraoperative image data captured by the endoscope in real-time by rendering at least a portion of the internal organ of the patient that is registered from the real-time intraoperative image data from the endoscope but outside a field of view of the endoscope.

[0007] The patent US 11113825 B2 discloses a projected image item tracking system that analyzes projected camera images to determine items taken from, placed on, or moved on a shelf or other area in an autonomous store.

[0008] The patent application WO 2015 / 135985 A1 discloses a projection system for projecting a feature on a subject. An intraoperative position marker (10) is provided on a first body surface (6) and provided with a movable first detector (12) having a detection line of sight.

[0009] The prior art document has the disadvantage of requiring a position marker on the subject. In addition, the prior art can involve calibrating the projected image with the detected image, which can be cumbersome and / or reduce the accuracy and / or quality of the projected image. SUMMARY

[0010] The present invention seeks to provide an improved image projection system for projecting one or more image representations of a feature of interest on a body surface of a subject. A conventional image projector can ensure that the projected image representation is projected on the body surface at the same location as where the body surface image was captured or detected based on a calibration. The calibration process can result in a projected image having reduced accuracy and / or quality. The calibration step can also be inconvenient.

[0011] Known image projection devices can have the problem of aligning the projected image of the image projector with the detected image of the image detector. Such alignment can be difficult to perform perfectly. This can involve an image transformation which can reduce the accuracy and / or quality of the projected image and can consume a relatively large processing power.

[0012] According to an aspect of the present invention, a projection device for projecting an image on a subject comprises: an image detector having a detector field of view, wherein the image detector is configured to generate an input image of the subject; an image projector having a projector field of view, wherein the projector field of view overlaps the detector field of view, and wherein the image projector is configured to project an output image based on the input image on a target surface of the subject, wherein the output image is aligned with the input image.

[0013] Because the detector field of view and the projector field of view are configured to overlap, the projector can project an image at the same location as where the image was detected. In this way, the target of the detector can be projected with a visual representation of the detected image to indicate its location physically below the surface of the subject. The detector field of view and the projector field of view can overlap such that the projector field of view precisely covers the detector field of view such that the projected image is projected to the location where the image was detected.

[0014] Because the projected image is projected from (precisely or approximately) the location of the detector, any distortion caused by the curvature of the surface can be avoided or strongly reduced, thereby reducing or negating any image transformation to correct for such distortion.

[0015] The image detector and the image projector can be fixed relative to each other in a single apparatus. For example, the image projector and the image detector can be housed in a single housing. This can help to simplify or avoid a calibration procedure. Because the image detector and the image projector are fixed relative to each other, the distance and relative orientation between them is known and static. And because the image detector and the image projector are housed in the same apparatus, the distance between the image detector and the image projector is relatively small compared to the distance from the subject to the apparatus. In this way, spatial correction of the projected image can be avoided or minimized.

[0016] The image detector can comprise a first optical path within the projection device, the first optical path being associated with a detector field of view; the image projector can comprise a second optical path within the projection device, the second optical path being associated with a projector field of view; a distance from the first optical path within the projection device to the second optical path within the projection device can be at most 20 millimetres, preferably at most 10 millimetres, more preferably at most 5 millimetres, even more preferably at most 1 millimetre; and the image projector can be configured to project on a target surface, wherein a distance of the image projector to the target surface is at least 1 metre, preferably at least 2 metres. By setting the projector very close to the detector, the projected image does not need to be transformed much to correct for any spatial distortion that would otherwise occur. For example, distal portions of the optical paths within the projection device (where the optical paths have an optical interface to the outside of the device) can be substantially parallel to each other and close to each other, as indicated.

[0017] The image detector and the image projector can share at least a portion of the optical path within the projection device. In this way, spatial transformations for correctly aligning the projected image with the detected image can be omitted, because there is no distortion. Correction for differences in image resolution or other image properties between the detected image and the projected image can still be useful. For example, distal portions of the optical paths within the projection device (where the optical paths have an optical interface to the outside of the device) can be shared by the projector and the detector.

[0018] The apparatus can comprise a first optical element configured to direct an incident light beam from the subject through the optical path and towards an image sensor of the image detector; and a second optical element configured to direct an exit light beam from a light source of the image projector through the optical path and towards the target surface. This provides an efficient way of implementing a shared optical path.

[0019] The first optical element can comprise the second optical element and can comprise a reflective side and a transmissive side. This provides an efficient way of implementing a shared optical path.

[0020] The reflective side can be configured to direct the incoming signal towards the image sensor, and the transmissive side can be configured to receive a light signal from a light source of the image projector and allow the light signal to pass through. This provides an efficient way of implementing a shared optical path.

[0021] The device can comprise a lens, wherein the shared optical path passes through the lens to the target surface. The shared lens can define a center of the projected light rays and the detected light rays. The lens can be configured as an interface between the optical path within the device and outside the device.

[0022] The device can comprise any number of image detectors with an additional field of view of the subject, wherein the additional image detectors are configured to generate an additional detected image of the subject. The processor can be configured to generate an output image based on the multiple input images. For example, the additional image detectors can be used for distance measurements. Alternatively, the additional image detectors can generate input images with content that can be merged with the detected images of the other image detectors to generate an output image and project back onto the projection surface. The features of the additional image detectors, in particular their distance and orientation and positioning relative to the projector and the subject, can be similar to the features described with respect to the image detectors described. Alternatively, the additional image detectors can be positioned further away from the projector.

[0023] The projection device can comprise an emitter configured to emit radiation towards the target surface at a predefined frequency. This can be used, for example, to illuminate the subject or to excite certain elements in the field of view.

[0024] The emitter can be configured to excite radiation at a frequency of radiation corresponding to an excitation frequency of an element within the subject, wherein the image detector is configured to detect radiation at a frequency corresponding to the excitation light of the element. This can be useful for detecting radiation generated by an element within the subject and projecting it back onto the surface of the subject.

[0025] The image detector can comprise at least one of the following: an image detector of a hyperspectral or multispectral camera, an infrared camera, a thermal camera, or a laser Doppler imaging system. This can allow for detecting useful images and projecting them back onto the subject.

[0026] The processor can be configured to map a pixel of the output image to at least one pixel of the input image, wherein a slope of an output light ray associated with the pixel of the output image corresponds to a slope of an input light ray associated with the at least one pixel of the input image, said slopes being with respect to the projection device; and determine a value of the output pixel based on at least one value of the at least one input pixel that is mapped to the output pixel. This provides a suitable mapping of the detected and / or projected field of view to the projected field of view in terms of pixels of the images.

[0027] The projector field of view can be wider or narrower than the detector field of view. Thus, the fields of view need not be the same.

[0028] The image detector and the image projector can have overlapping optical paths to reduce or eliminate adjustments in mapping the detected image to the projected image. The optical paths can travel through a lens of the image projection device. The lens can be configured to generate the detector field of view and the projector field of view.

[0029] The image projection device can include an optical element configured to align the image detector optical path with the image projector optical path. For example, if the image detector is configured to detect a target surface and the two optical paths are aligned, no adjustments are needed in mapping the detected image pixels to the projected image pixels.

[0030] The optical element can include a reflective side and a transmissive side. The reflective side can reflect light beams emitted from the subject to the image detector. The transmissive side can allow light beams emitted from the image projector to travel through the optical element to the subject. It should be noted that the image projection device can include separate reflective and transmissive optical elements, rather than a combined optical element.

[0031] The image projection device can include one or more image detectors configured to take one or more detected images. Each detected image can represent a different characteristic, such as a different wavelength. The projected image can be generated based on the one or more detected images to generate a visual representation of the one or more characteristics.

[0032] The image projection device can also include a light emitter configured to emit radiation onto the field of view of the detector. This can facilitate imaging techniques including, but not limited to, fluorescent dyes, laser speckle contrast imaging (LSCI), and laser Doppler imaging.

[0033] Another aspect of the present disclosure provides a method of projecting an image, the method comprising: detecting, by an image detector having a detector field of view, an input image of a subject; generating, by a processor, an output image based on the input image, wherein the output image includes content representing at least a portion of content of the input image; projecting, by an image projector, the output image on a target surface of the subject by an image projector having a projector field of view, wherein the projector field of view overlaps the detector field of view.

[0034] Another aspect of the invention provides a method of generating a projection image, the method comprising: determining a first distance between an image detector and a target surface; determining a second distance between an image projector and the target surface; determining an angle between the image projector and the image detector, wherein a vertex of the angle is at the target surface; mapping a detected pixel of a detected image to a projection pixel of the projection image based on the first distance, the second distance and the angle, such that the projection pixel represents a same location of the target surface as the detected pixel.

[0035] The skilled person will appreciate that the features described above can be combined in any way deemed useful. Furthermore, modifications and variations described in relation to the apparatus can equally be applied to the method, and modifications and variations described in relation to the method can equally be applied to the apparatus. BRIEF DESCRIPTION OF DRAWINGS

[0036] The invention will be explained in more detail below based on a number of exemplary embodiments with reference to the attached drawings. The drawings are diagrammatic and can not be to scale. In all the drawings, similar references can be used to refer to similar items. In the drawings:

[0037] Figure 1 An image projection apparatus according to the present disclosure is shown;

[0038] Figure 2 An exemplary image detector and image projector according to the present disclosure are shown;

[0039] Figure 3 An exemplary detection of a thermal image and a corresponding projection image according to the present disclosure is shown;

[0040] Figure 4 An exemplary image projection apparatus according to the present disclosure is shown, the image projection apparatus comprising an optical element between the image detector and the image projector;

[0041] Figure 5 A first exemplary configuration of a projection apparatus comprising two image detectors according to the present disclosure is shown;

[0042] Figure 6 A first exemplary configuration of a projection apparatus comprising two image detectors according to the present disclosure is shown;

[0043] Figure 7 A first exemplary configuration of a projection apparatus comprising two image detectors according to the present disclosure is shown;

[0044] Figure 8 An exemplary projection apparatus configured to detect an image of a subcutaneous feature according to the present disclosure is shown;

[0045] Figure 9 Another exemplary configuration of a projection device according to the present disclosure is shown;

[0046] Figure 10 A sketch of an exemplary projected image on a surface of a subject according to the present disclosure is shown, the projected image comprising one or more data of interest;

[0047] Figure 11 An exemplary image projected on an arm of a subject is shown;

[0048] Figure 12 An exemplary image projected on a hand of a subject is shown;

[0049] Figure 13 An exemplary image of an excited dye projected on a surface of a subject is shown. DETAILED DESCRIPTION

[0050] In medicine, there are various difficulties in identifying patient information. For example, identifying and locating subcutaneous structures; and locating internal organs; patient vital signs, such as temperature, blood pressure, heart rate, etc.; blood flow; burns; infections; rheumatism; cancer cells; microorganisms; etc. For example, during a surgical operation, it can be advantageous to identify a part of the patient's vascular system before starting the surgical operation.

[0051] The image projection device disclosed herein is generally designed to project an image on an external surface of a subject. The image projection device can thus be located at a reasonable working distance from the subject, for example at most 3 meters. Moreover, the image projection device can have a fixture to fix the device to, for example, a wall or a ceiling or a stand, at least during operation. Alternatively or additionally, the device can have a handle that allows the device to be held in the hand during operation. Such a fixture or handle can be connected to the detector and the projector (including the optical elements along the optical path) by means of rigid connections and / or hinged connections.

[0052] The image projection device can comprise a laser projector configured to focus a projected image on a remote target surface, regardless of the distance between the laser projector and the target surface.

[0053] Alternatively, the image projection device can include a lens system to focus the projected image on the target surface. Such a projection device can for example include a light emitting diode (LED) and / or a liquid crystal display (LCD). The image projection device can include an auto-focus feature or a manual focus feature to focus the projected image on the target surface. The image projection device can include sensors configured to measure the distance between the LED projector and the target surface. This distance can be fed back to the image projection device to determine the alignment of the projected image.

[0054] Additionally or alternatively, the lens system can be configured to focus the image detector such that the detected image is in focus. In either case, the focusing lens can be optimized to focus to a distance of at least 1 meter and / or at most 3 meters from the projection device.

[0055] It should be noted that the term "detector field of view" can herein denote the area or volume of the subject that the image detector can detect. It should further be noted that the image detector can be able to detect radiation emitted or reflected below the surface of the subject, or signals that can originate from below the surface of the subject. For example, detecting vascular structures below the surface of the subject. It should be noted that the term "projector field of view" can herein denote the area of the surface of the subject on which the projector can project the projected image. It should further be noted that the detector field of view and the projector field of view can overlap. The field of view can have a conical shape with its center (apex) at the detector / projector.

[0056] It should further be noted that the surface of the subject (e.g. the skin of the patient) can typically not be a flat surface and can include a three-dimensional profile following the curvature of the body and organs etc. (e.g. it can be curved and can have upright walls, recesses etc.). The image detector can generate a two-dimensional (2D) image of the structures within the field of view. The projected image can be a 2D image projected on the surface within the field of view. Because the projected image is projected from (exactly or approximately) the position of the detector, any distortion caused by the curvature of the surface can be avoided or strongly reduced, thereby reducing or negating any image transformation to correct for such distortion.

[0057] Alternatively, the projected 2D image can be corrected for the 3D surface on which it is projected. The image projection device can include a 3D sensor, such as a time-of-flight camera, a stereo camera, etc. to adapt the projected image to the 3D profile of the surface of the subject.

[0058] The image projection device can also include a light emitter configured to emit radiation onto the field of view of the detector. This can facilitate imaging techniques including, but not limited to, fluorescent dye, laser speckle contrast imaging (LSCI), and laser Doppler imaging. For example, certain bacteria or tissue can emit light at a certain frequency when exposed to light from the light emitter. The emitted light is sensed to detect an anatomical feature of interest. In another example, the device can be configured to perform laser speckle contrast imaging (LSCI) with a laser that emits light. The detector measures the minimum reflection difference of the speckle pattern of the detected image. LSCI can be used to detect movement of, for example, red blood cells in the tissue of the subject. Additionally, certain bacteria or tissue of interest can be excited when exposed to the emitted light. The emitted light can be configured for a particular frequency to excite the object of interest. The detector can be configured to detect light reflected by the object of interest when exposed to the emitted light.

[0059] If the image detector is detecting images of features below the target surface, the distances between the target surface and the features can optionally be determined. The mapping algorithm can use these distances to map the detected images to the projected image. The projected image can be generated to align with the detected images. For example, the device can be configured to project a visual representation of a detected internal organ on the target surface to show where the organ is located below the surface.

[0060] There can be an error margin for the projected image. For example, the projected image can not perfectly align with the detected images. The allowed error margin can depend on the type of image detector. For example, an image detector configured to detect vascular structures below the surface of a subject can allow a smaller error margin than a thermal image detector configured to measure the temperature of a subject.

[0061] The projected image can align with the structures below detected by the image detector. For example, internal organs or cancerous tissue. When the projected image is projected on the surface, a visual representation of the detected image showing the structures below is projected on the target surface. This allows medical personnel to view the information directly on the patient in real-time without having to use a separate display screen. The image projection device can project at the location on the surface of the subject where the feature of interest is located to assist in surgical procedures.

[0062] Figure 1An image projection apparatus 102 according to this disclosure is shown. Subject 114 is not part of the image projection apparatus 102. The image projection apparatus 102 may include an image detector 104 and an image projector 106, which are described in further detail below. The image detector 104 may be configured to detect features, such as anatomical features (organs, etc.) and / or physiological features. The image projection apparatus 102 may further include a lens 108. The detected beam 110 and the projected beam 112 may pass through the lens 108 to reach and exit the image projection apparatus 102, respectively. The lens 108 may be configured to generate fields of view for the image detector 104 and the image projector 106. For example, the field of view of the image detector and the field of view of the projector may overlap. The lens 108 may be configured such that the field of view of the projector and the field of view of the detector overlap each other, such that each pixel of the projected image is projected to the exact same position as the corresponding pixel of the detected image. It should be noted that there may be an acceptable error margin between the projected image and the detected image. The acceptable error margin may depend on the application and, for example, the type of detected image. For example, in some applications, an image representation of subject 114's body temperature can have a larger error margin than an image representation of subject 114's blood vessels. The image projection device can be configured for an error margin of up to 1 cm. In other words, the projected image can be at most 1 cm from the actual location of the object of interest. Image detector 104 can generate a detected image based on a detected beam 110 from the image detector's field of view. The detected beam can be emitted or reflected by subject 114. Image projection device 102 can generate a projected image based on the detected image. Image projection device 102 can map information from the detected image to a visual representation of information in the projected image. Image projector 106 can project the projected image based on the detected image via beam 112. Image projector 106 can use beam 112 to project the projected image onto subject 114.

[0063] Image detector 104 may include a sensor, such as a sensor having a plurality of detection elements arranged in a matrix on a substrate. Each detection element may correspond to a pixel of the input image. An exemplary sensor is a complementary metal-oxide-semiconductor (CMOS) detector. Suitable filters may be provided such that the detector is configured to sense radiation within a specific frequency range. Image projector 106 may include a laser projector or a plurality of light-emitting elements (such as LEDs) arranged in a matrix on a substrate. The sensor may be any camera-based sensor system. Alternatively, the sensor may be a 2D or 3D ultrasound device, or an X-ray device.

[0064] For example, the image projector 106 may be a laser projector configured to project the image using a coherent parallel beam.

[0065] As another example, if the image projector 106 can be an LED projector, then each light emitting element can correspond to a pixel of the output image.

[0066] Although the image sensor can be configured to detect radiation in any suitable frequency range to capture any desired features, the projector can be configured to emit light in the visible spectrum such that the image projected on the surface of the subject is visible to the human eye.

[0067] The image detector 104 and the image projector 106 can have fixed, substantially parallel positions within the image projection device 102. The image projection device 102 can use the distance and angle between the image detector 104 and the image projector 106 to generate the projected image. Because the relative positions between the image detector 104 and the image projector 106 are fixed, the distance and angle between them are also fixed. A single calibration is sufficient to align the optical path and adjust the projected image onto the subject through the captured sensor data.

[0068] If the paths use optical elements to optically combine and align in all directions as described previously, the image projection device 102 uses these static values to generate the projected image without additional calculations or calibrations, as described below with respect to at least Figure 4 shown.

[0069] If the paths of the image detector 104 and the image projector 106 are parallel to each other, the image projection device 102 can use static values to shift the projected image based on the distance between the detector 104 and the projector 106.

[0070] The image projection device 102 can include more than one image detector 104. When the image projection device 102 includes multiple image detectors, each image detector can generate a detected image. The projected image can be based on the multiple detected images. For example, if the image projection device 102 includes a thermal image detector and an infrared image detector, for example, filtered for indocyanine green (ICG), the image projector 106 can project an image that represents the surface temperature of the subject 114 and indicates the presence of the dye in the subject 114.

[0071] Alternatively, the image detector 104 can be a laser Doppler imaging (LDI) system. The LDI system can include a laser exciter, a laser, and a camera configured to detect changes over time. The LDI system can be configured to detect movement of red blood cells by reflecting a particular wavelength to determine movement of red blood cells in the subject’s tissue. For example, when the laser reaches the tissue, moving blood cells generate a Doppler component in the reflected light.

[0072] For example, the LSCI can be configured to measure burn depth of a subject. Speckle can be produced by detecting when coherent light is reflected from the subject's tissue. Red blood cells can scatter the coherent light to cause the speckle pattern to blur. Movement of the red blood cells can be determined based on the degree of blurring in the detected image. Burn depth can be determined based on the presence of movement of the red blood cells.

[0073] Typically, the surface of the subject 114 is a skin surface. However, in certain applications of the image projection device 102, the surface of the subject 114 can alternatively be an internal body surface of the subject. For example, after a surgical procedure has begun and the subject 114 has an open cavity, images can be projected from outside the subject onto a surface within the cavity.

[0074] For example, the image detection device 102 can include a radiation source 116 configured to emit radiation, such as light 118. The radiation source 116 can illuminate the light 118 onto the subject 114 to, for example, excite bacteria or, for example, excite an injected dye in or on the subject 114. When excited by the light 118, the excited bacteria or injected dye can emit excited light. The image detector 104 can be configured to detect the excited light. The radiation source 116 can also be included in an LDI system.

[0075] Alternatively, the image projector 106 can also be the radiation source 116. For example, the image projector 106 can include an infrared channel in addition to the default RGB channels to project infrared light to excite an injected dye. By alternating between projecting from the infrared channel and projecting from the RGB channels, the image projector 106 can project a stable projected image and also excite, for example, an injected dye.

[0076] This is achieved by alternating between the infrared channel and the RGB channels (e.g., every 1 / 50th of a second). The image projection device 102 can also include the image detector 104 that is synchronized with the infrared channel of the image projector 106 to capture the excited light every, for example, 1 / 50th of a second. The human eye can only detect at approximately 50 frames per second (Hz), and because the image projector is able to project at a higher frequency, the projected image is still stable to the human eye even though the image projector 106 is alternating between projecting from the RGB channels and projecting from the infrared channel.

[0077] The image detector 104 can be capable of detecting the excited dye faster than 1 / 50th of a second and synchronized with the cycle of the image projector 106 to project infrared light to detect the light emitted by the excited dye. The image detector 104 can be configured to detect multiple wavelengths. While the image projector 106 and the image detector 104 are synchronized and capable of projecting and detecting fast enough to maintain a stable projected image, the image projector 106 can project two or more wavelengths between RGB projections. The following table demonstrates the alternating projections and detections:

[0078]

[0079] The image detection device 102 can include a processor (not shown) configured to generate a projected image based on the detected image. The image detection device 102 can use the processor to map the pixels of the detected image to the projected image. Each pixel of the detected image can be transformed to a pixel of the projected image. Each transformed pixel of the projected image can be a visual representation of the information detected by the image detector 104.

[0080] The image detection device 102 can be configured with a static measurement of the relative orientation (angle) and distance between the image detector 104 and the image projector 106. The image detection device 102 can be configured to map the pixels of the detected image to the pixels of the projected image based on the relative orientation and distance. The resulting projected image then accounts for and corrects for any positional differences between the image detector 104 and the image projector 106. Because the positional differences are fixed, the mapping algorithm can generate the projected image without having to obtain a measurement. Thus, generating the projected image requires less computation and less time than generating an image that requires a measurement, such as the distance between the image detector and the surface of the subject. It should be understood that the distance and orientation are actually related to the distal portions of the optical path of the image detector 104, which terminates at the lens 108, and the distal portion of the optical path of the image projector 106, which also terminates at the lens 108. That is, any internal mirrors, etc. are not considered. In particular, the distance between the center (apex) of the projection field of view and the detection field of view can be static. It is observed that, in certain embodiments, the optical path of the image detector 104 has a different lens than the optical path of the image projector.

[0081] The image detection device 102 can further include a mounting apparatus. The mounting apparatus can be configured to statically mount the image detection device 102 to a wall or a ceiling. Additionally, the mounting apparatus can include a ball joint connector to allow the image detection device 102 to be pointed in different directions. It should be noted that other types of connectors can be used. Furthermore, the mounting apparatus can include an extension to move the image detection device 102 towards or away from the target subject. It should be noted that any combination of mounts and extensions can be included in the image detection device 102. The image detection device 102 can have 6 degrees of freedom to point the image detection device 102 at any angle at different distances from the target subject.

[0082] Alternatively, the image detection device 102 can include a handle to hold the image detection device 102. Even without an extension arm or a connector, the user can use the handle to hold the image detection device 102 firmly to easily point the image detection device 102 in any direction desired by the user.

[0083] Figure 2 An exemplary image projection device 102 is shown that includes an apparatus having a housing 202. The apparatus can include an image detector 104 having a lens 108a and an image projector 106 having a lens 108b. The image detector 104 and the image projector 106 can be coupled to a processor 204. The processor 204 can be configured to execute an algorithm for mapping pixels of a projected image to pixels of a detected image and calculating values of the pixels of the projected image based on the pixels of the detected image that are mapped to the pixels of the projected image.

[0084] Because the image detector 104 and the image projector 106 are fixed within a single apparatus, the distance 206 between the image detector 104 and the image projector 106 is known and static. In particular, the distance between the lens 108a of the image detector 104 and the lens 108b of the image projector is static. The entire optical path inside the apparatus shown by way of light beams 210 and 212 can be statically arranged within the apparatus. With this fixed and known arrangement, only the distance 216 between the apparatus / device 102 and the surface 208 of the surface of the subject 220 can be used as an additional measurement to calculate the output image from the input image. In certain embodiments, a default value for the distance 216 can be assumed in the algorithm so that the measurement of the distance 216 can be omitted. This distance 216 can also be used to focus the projected image on the surface 208.

[0085] For example, the distance 206 between the image detector 104 and the image projector 106, in particular between their lenses and between their optical paths inside the housing 202 of the device 102, can be at most 20 mm, preferably at most 10 mm, more preferably at most 5 mm, even more preferably at most 1 mm, even more preferably they coincide. These limits on the distance 206 can provide sufficient accuracy when using the apparatus 102 at a distance 216 from the surface of the subject 208 that is common in, for example, surgical procedures. For example, the distance 216 can be in the range of 1 to 3 meters. For example, the focal length of the projector can be focused to project on a surface somewhere in the range of 1 to 3 meters from the apparatus 102. This small distance helps to make the projection accurate, even when the device 102 is movable. The distance can be large enough to provide space for, for example, a surgical procedure or other treatment. Alternatively, the limits on the distance 206 can be expressed relative to the distance 214 to the surface of the subject. Assuming a minimum working distance 216 of 0.25 meters, the relative distance 206 between the detector and the projector can be at most 2%, preferably at most 1%, more preferably at most 0.5%, even more preferably at most 0.1% relative to the distance between the apparatus 102 and the surface of the subject 208.

[0086] For ease of discussion, the lenses 108a and 108b have been illustrated as single lenses. However, each illustrated lens can comprise a plurality of lenses arranged in series.

[0087] The image projection device 102 can comprise one or more sensors fixed to the housing 202 to determine the distance 216 between the image projection device or apparatus (lenses of) and the surface of the subject 208. For example, the image projection device can comprise an ultrasonic sensor or an optical sensor to determine the distance 216. The processor 204 can calculate the angle 214 between the image detector 104 and the image projector 106 with the angle vertex 218 on the surface of the subject 208 based on the distances 206 and 216.

[0088] The image detector 104 and the image projector 106 can have the same field of view of the target surface 220. The processor 204 can map the detected image of the image detector 104 to a projected image of the image projector 106. The processor 204 can execute a mapping algorithm to generate the projected image. The mapping algorithm can receive the angle 214 and the detected image. The mapping algorithm can generate the projected image based on the detected image and the angle 214 such that a visual representation of the detected image is displayed on the subject surface 208. The image projector 106 projects the projected image in the field of view of the target surface 220 that the image detector 104 uses to detect the detected image. Because the distance 206 between the image detector 104 and the image projector 106 (lenses 108a, 108b) is constant, and the distances 210 and 212 and the angle 214 can be quickly determined, the mapping algorithm can be able to quickly and accurately generate the projected image compared to the calibration of separately housed image detectors and image projectors.

[0089] This can result in fewer processing resources for implementing the image projection device 102. Additionally, the reduced complexity of generating the output image can result in less distortion of the output image and thus better quality. Any manual configuration is further simplified.

[0090] The processor 204 can be configured to execute the mapping algorithm such that the projected image displays information on the subject surface 208 in the same location as where the image detector 104 detected the information. The mapping algorithm can allow for an error margin, and the error margin can depend on the type of image detection. In other words, the projected image can not be in the exact same location on the subject surface 208 as the detected image.

[0091] Figure 3A detected image 300a and a projected image 300b are shown for thermal image data on a surface of a subject. The detected image 300a can represent the field of view of the target surface 220. The image detector 104 can be a thermal image camera configured to detect the temperature of the subject surface. The detected image 300a can detect that region 302a is 33 °C, region 304a is 34 °C, region 306a is 35 °C, region 308a is 36 °C, and region 310a is 37 °C. The mapping algorithm can generate the projected image 300b, which is a visual representation of the detected image 300a. For example, the projected image 300b can include 5 regions to represent the different temperatures of the detected image 300a. Each region 302b-310b can correspond to regions 302a-310a. Each region 302b can have a different color to quickly represent the temperature without having to read. For example, region 310b can be red to represent the hottest temperature. Region 308b can be orange, region 306b can be yellow, region 304b can be green, and region 302b can be blue to show a step down in temperature. In the figure, these colors have been shown with the aid of gray scales.

[0092] The image projector 106 can be configured to project the projected image 300b on the subject surface 208 in the same location as the detected image 300a to show the temperature of the subject surface 208 directly on the subject surface 208.

[0093] Figure 4 An exemplary image projection device 402 is shown that includes an optical element 404. The optical element 404 includes a transmissive side 404a that is configured to allow the light beam 412 to travel through the optical element 404. The optical element 404 can also include a reflective side 404b that is configured to reflect the light beam 410. It should be noted that the transmissive side 404a and the reflective side 404b can be separate optical elements.

[0094] The optical element 404 can be positioned between the lens 108, the image detector 104, and the image projector 106. The optical element 404 can be configured such that the image detector 104 and the image projector 106 have the same field of view 406. A light beam 410 emanating from or originating at the subject surface 414 can reach the lens 108, which directs the light beam 410 toward a reflective side 404b of the optical element 404. The reflective side 404b can reflect the light beam 410 to the image detector 104 such that the image detector 104 can detect an image of the subject surface 414 from the field of view 406. The projector 106 emits a light beam that passes through a transmissive side 404a of the optical element 404 and through the lens 108 toward the subject surface 414. Due to the optical element 404, the image projector 106 can project the projected image in exactly the same location where the image detector 104 detects the image, thereby providing a mapping between input image pixels and output image pixels that does not depend on the distance between the device 402 (or the lens 108) and the subject surface. Because the image projector 106 is configured to project in the same location where the image detector 104 detects, the processor 204 can perform the second mapping algorithm without needing to determine the distance between the image detector 104 and the subject surface 414. Similarly, the processor 204 can perform the second mapping algorithm without needing to determine the distance between the image projector 106 and the subject surface 414.

[0095] For example, the second mapping algorithm can take as input a detected image and generate a projected image based on the detected image without needing to adjust the mapping of pixels based on a distance. However, in certain embodiments, if the image detector 104 is configured to detect a feature below the subject surface 208, the sensor can determine a distance between the subject surface and the feature. The mapping algorithm can use this distance to generate the projected image.

[0096] Figure 5An image projection device 402 is shown. The image projection device can include an image detector 104a and an image detector 104b. The image detectors 104a and 104b have the same field of view as the image projector 106. Both image detectors 104a and 104b can be configured to detect an image of a subject surface 414. A beamsplitter 802 can be configured to receive the light beam 410 and split the light beam between the image detectors 104a and 104b. The beamsplitter 802 can be configured to equally split the light beam 410 between the image detectors 104a and 104b. Alternatively, the beamsplitter 802 can be configured to unequally split the light beam 410 between the image detectors 104a and 104b. The image detectors 104a and 104b can be configured to generate detected images based on the split light beams 410a and 410b, respectively. The image detectors can be configured to generate different detected images. It should be noted that the image projection device 402 can include any number of beamsplitters to accommodate any number of image detectors. The processor 204 can be coupled to the image detectors 104a and 104b and configured to generate an output image based on a combination of the input images from all of the image detectors.

[0097] Figure 6 An image projection device 402 is shown including an image detector 104a and an image detector 104b. Both image detectors 104a and 104b can be configured to detect an image of a subject surface 414. The image projection device 402 can include optical elements 604a and 604b configured with a reflective side and a transmissive side as described with respect to the optical element 404. Figure 4 The optical element 404. The optical element 604b can be in a position between the lens 108 and the optical element 604a to receive the light beam 410 before it reaches the optical element 404a. The optical element 404b can be configured to reflect only certain wavelengths of the light beam 410 to the image detector 104b. The reflected light beam 410b can be a single wavelength or a range of wavelengths. The remaining wavelengths of the light beam 410 pass through the optical element 604b and reach the optical element 604a. The optical element 604a can be configured to reflect certain wavelengths or a range of wavelengths to the image detector 410a. Alternatively, the optical element can be configured to reflect all wavelengths. It should be noted that any number of optical elements can be used to reflect certain wavelengths to any number of image detectors configured to detect the corresponding wavelengths. For example, the optical element 604b can be configured to reflect infrared wavelengths to an image detector 104b configured to detect infrared light, and the optical element 604a can be configured to reflect visible wavelengths to an image detector 104a configured to detect visible light. The detected images from the image detectors 104a and 104b can be combined to generate a visible projection image including a representation of the detected infrared light and visible light.

[0098] Figure 7 An image projection device 402 is shown that includes image detector 104a and image detector 104b. Image detectors 104a and 104b can be configured to detect different features. The respective optical paths of image detector 104a, image detector 104b, and image projector 106 can be configured in parallel. Image detectors 104a and 104b can have a fixed distance and / or orientation with respect to each other. The detected images of 104a and 104b can be corrected based on the horizontal and / or vertical distance to be combined. For example, the detected image of image detector 104a can be shifted based on the horizontal and / or vertical distance with respect to image detector 104b to align with the detected image of image detector 104b. The combined image can be further shifted based on the horizontal and / or vertical distance between image detector 104b and image projector 106 to generate the projected image.

[0099] Figure 8 An image projection device 102 is shown that is configured to include image detector 104 and image projector 106. Although Figure 8 Optical element 404 is not shown in FIG. 8, but can be included. Image detector 104 can be configured to detect features 804 under target surface 802. Image projection device 102 can be configured to determine a distance 808 between the image detection device and features 804. Image projection device 102 can be configured to determine a distance 806 between target surface 802 and features 804. Alternatively, the image projection device can be configured to determine different distances, such as the distance between features 804 and image detector 104, image projector 106 and target surface 802, or any distance necessary to generate the projected image. Mapping algorithms can use distances 808 and 806, and any other determined distances, to generate the projected image such that the projected image aligns with features 804 underneath. For example, a distance 810 between light beams 210 and 212 at target surface 802 (indicated by the dashed line on target surface 802). Mapping algorithms can also use distance 810 to shift the position of pixels when generating the projected image such that the projected image aligns with features 804 underneath.

[0100] Figure 9An image projection device configured for a case where distance to subcutaneous structures is unavailable is shown. Since distance to the surface of the subject can be measured, an incident light beam (i.e., a pixel of the input image) can be mapped to a point 218 on the surface of the subject 220 at which the incident light beam intersects the surface of the subject 220. Via this point 218, an angle 214 can be determined and / or a corresponding pixel on the output image can be calculated by the processor. In this way, correct mapping with negligible error can be achieved without the need to detect any depth information of the image.

[0101] Figure 10 A projected image 1002 is shown projected on the subject 114. The projected image 1002 can be a visual representation of the detected image and include a visual representation 1004 of the detected feature 804. The projected image 1002 can visually represent the feature 804 below the target surface of the subject 114 to show where the feature 804 is located below the target surface. Alternatively, in another example, the projected image 1006 can include more than a visual representation of the detected image. The projected image 1006 includes the projected image 1002, which is a visual representation of the detected image. In addition, the projected image 1006 can also include data 1008 and project that data on the target surface. The data 1008 can be projected on the target surface in a location that does not interfere with the projection of the feature of interest 804.

[0102] In configurations where the projector field of view is larger than the detector field of view, the data 1008 can be included on the projected image 1006. This allows the image projector 106 to project the data 1008 to the surface of the subject without interfering with the visual representation of the feature 1004. Alternatively, the detector field of view can be larger, but the projector can only project back a portion of the detector field of view.

[0103] Figure 11An image 1100 of a subject's (e.g., subject 114) arm is shown. An image of a visual representation of thermal data is projected back onto the surface of the arm as shown in image 1100. Image 100 is a result of cooling the subject's arm. A gel pack stored at 4 degrees Celsius, positioned on the surface of the subject's arm for 30 seconds, cools the surface of the arm. The image projection device is configured to detect and project a visual representation of thermal data on the subject. The image projection device includes a thermal camera configured to detect thermal data of the subject's arm. The image projection device is positioned about 50 centimeters above a table on which the subject's arm rests. As the subject's arm warms up, the thermal camera detects areas of the arm that warm up first or hot spots 1102. As the surface of the subject's arm warms up, the hot spots 1102 reveal various blood vessels under the surface of the skin to show a pattern of blood vessels in image 1100. The hand has not cooled down and is represented by a white or light color shading in image 1100. The cooled down arm is represented by a dark or black color shading in image 1100. The image projection device also includes a laser projector to project a visual representation of the thermal data to image on the surface of the subject's arm. As the arm warms up, a pattern 1102 of lighter shading emerges throughout the dark arm, revealing a pattern of blood vessels as shown in image 1100.

[0104] Figure 12 An image 1200 of a subject's (e.g., subject 114) hand is shown. The subject is injected with indocyanine green (ICG) dye to reveal the lymphatic system in the subject's hand. ICG is injected into two web spaces of the hand. A bandage is placed over the injection site as shown in image 1200. The ICG dye diffuses through the subject's lymphatic system. The image projection device is configured to detect and project a visual representation of the image 1200 generated by the ICG dye. The image projection device includes an infrared LED as a radiation emitter 116 configured to excite the ICG injected into the subject's hand with light, such as light 118. The LED can be configured for different intensities to detect more or less fluorescence. The image projection device also includes a camera configured to detect fluorescence emitted from the excited ICG dye. The camera can be configured for different image contrast, brightness, or other image settings. The image projection device is positioned about 30 centimeters above the subject's hand. The camera detects fluorescence emitted from the excited ICG. The image projection device includes a projector to project an image of a visual representation of the fluorescence on the surface of the subject's hand. Throughout the hand, green light represented by light 1202 reveals fluorescence emitted from the excited ICG in the subject's hand's lymphatic system.

[0105] Figure 13An image 1300 of an experiment with ICG dye is shown. ICG dye is injected into an experimental pad at point 1302. A material 1304 is soaked in the ICG dye. An image projection device is configured to detect and project a visual representation of an image 1300 generated by the ICG dye. The image projection device (similar to the image projection device of Figure 12 FIG. 1) is placed 30 centimeters above the experimental pad and the material 1304. The image 1300 shows the ICG dye at point 1302 of the experimental pad and the ICG dye in the soaked material 1304. The image projection device includes an LED configured to excite the ICG dye and a camera configured to detect light emitted from the excited ICG dye. The image projection device also includes a projector configured to project a visual representation of the detected fluorescence emitted from the excited ICG dye on a surface of the subject, as shown in image 1300. For example, the experimental pad and the soaked material.

[0106] Some or all aspects of the present invention can be suitable for implementation in the form of software, in particular a computer program product. This applies to, for example, software of the control unit 204 as well as software for controlling the image detector and the projector, and software for processing the detected image and converting it into a projected image. The computer program product can comprise a computer program stored on a non-transitory computer readable medium. Also, the computer program can be represented by a signal (such as an optical or electromagnetic signal) carried by a transmission medium (such as a fiber optic cable or air). The computer program can partly or entirely be in the form of source code, object code, or pseudo code suitable for execution by a computer system. For example, the code can be executed by one or more processors.

[0107] The examples and embodiments described herein are used to illustrate and not to limit the present invention. Those skilled in the art will be able to design alternative embodiments without departing from the spirit and scope of the disclosure defined by the appended claims and their equivalents. Reference signs placed in parentheses in the claims shall not be construed as limiting the scope of the claims. Items described as separate entities in the claims or the specification can be implemented as a single hardware or software item combining features of the described items.

[0108] Thus, modifications and alternative implementations are possible, and are included within the scope of the protection defined by the appended claims.

[0109] Certain examples are disclosed in the following clauses.

[0110] 1. A projection device for image projection of an image on a subject, comprising:

[0111] an image detector having a detector field of view, wherein the image detector is configured to detect an input image of the subject;

[0112] a processor configured to generate an output image based on the input image, wherein the output image comprises content representing at least a portion of the content of the input image;

[0113] an image projector having a projector field of view, wherein the projector field of view overlaps the detector field of view, and wherein the image projector is configured to project the output image on a target surface of the subject, wherein the output image is aligned with the input image.

[0114] 2. The projection device of clause 1, wherein the image detector and the image projector are fixed relative to each other in a single apparatus.

[0115] 3. The projection device of any preceding clause, wherein,

[0116] the image detector comprises a first optical path within the projection device, the first optical path being associated with the detector field of view;

[0117] the image projector comprises a second optical path within the projection device, the second optical path being associated with the projector field of view;

[0118] a distance from the first optical path within the projection device to the second optical path within the projection device is at most 20 millimeters, preferably at most 10 millimeters, more preferably at most 5 millimeters, even more preferably at most 1 millimeter; and

[0119] the image projector is configured to project on the target surface, wherein a distance of the image projector to the target surface is at least 1 meter, preferably at least 2 meters.

[0120] 4. The projection device of any preceding clause, wherein the image detector and the image projector share at least a portion of an optical path within the projection device.

[0121] 5. The projection device of clause 4, further comprising

[0122] a first optical element configured to direct an incident light beam from the subject through the optical path and towards an image sensor of the image detector; and

[0123] a second optical element configured to direct an exit light beam from a light source of the image projector through the optical path and towards the target surface.

[0124] 6. The projection device of clause 5, wherein the first optical element comprises the second optical element and comprises a reflective side and a transmissive side.

[0125] 7. The projection device of clause 6, wherein the reflective side is configured to direct an incident signal toward the image sensor, and the transmissive side is configured to receive a light signal from a light source of the image projector and allow the light signal to pass through.

[0126] 8. The projection device of any of clauses 4-7, further comprising a lens, wherein the shared optical path passes through the lens to the target surface.

[0127] 9. The projection device of any of the preceding clauses,

[0128] further comprising a second image detector having a second field of view of the subject, wherein the second image detector is configured to generate a second input image of the subject, and

[0129] wherein the processor is configured to generate the output image further based on the second input image.

[0130] 10. The projection device of any of the preceding clauses, further comprising an emitter configured to emit radiation toward the target surface at a predefined frequency.

[0131] 11. The projection device of clause 10, wherein the emitter is configured to excite the radiation at a radiation frequency corresponding to an excitation frequency of an element within the subject, wherein the image detector is configured to detect an excitation light of the element.

[0132] 12. The projection device of any of the preceding clauses, wherein the image detector comprises at least one of: a hyperspectral or multispectral camera, an infrared camera, a thermal camera, or an image detector of a laser Doppler imaging system.

[0133] 13. The projection device of any of the preceding clauses, wherein the processor is configured to:

[0134] map a pixel of the output image to at least one pixel of the input image, wherein a tilt of an output ray associated with the pixel of the output image corresponds to a tilt of an input ray associated with the at least one pixel of the input image, said tilts being with respect to the projection device; and

[0135] determine a value of the output pixel based on at least one value of the at least one input pixel that is mapped to the output pixel.

[0136] 14. The projection device of any of the preceding clauses, further comprising a fixture for fixing the device during operation or a handle for holding the device in a hand during operation.

[0137] 15. A method of projecting an image, the method comprising:

[0138] detecting, by an image detector having a detector field of view, an input image of a subject;

[0139] generating, by a processor, an output image based on the input image, wherein the output image comprises content representing at least a portion of content of the input image;

[0140] projecting, by an image projector, the output image on a target surface of the subject by an image projector having a projector field of view,

[0141] wherein the projector field of view overlaps the detector field of view.

Claims

1. A projection device for image projection on a subject, comprising: an image detector having a detector field of view, wherein the image detector is configured to detect an input image of the subject; a processor configured to generate an output image based on the input image, wherein the output image comprises content representing at least a portion of the content of the input image, wherein the processor is configured to map a pixel of the output image to at least one pixel of the input image, wherein a tilt of an output light ray associated with the pixel of the output image corresponds to a tilt of an input light ray associated with the at least one pixel of the input image, the tilt being with respect to the projection device, and a value of the output pixel is determined based on at least one value of the at least one input pixel that is mapped to the output pixel; an image projector having a projector field of view, wherein the projector field of view overlaps with the detector field of view, and wherein the image projector is configured to project the output image on a target surface of the subject, wherein the output image is aligned with the input image.

2. The projection apparatus of claim 1, wherein, The image detector and the image projector are fixed relative to each other in a single apparatus.

3. The projection device of any preceding claim, wherein the image detector comprises a first optical path within the projection device, the first optical path being associated with the detector field of view; the image projector comprises a second optical path within the projection device, the second optical path being associated with the projector field of view; a distance from the first optical path within the projection device to the second optical path within the projection device is at most 20 millimeters, preferably at most 10 millimeters, more preferably at most 5 millimeters, even more preferably at most 1 millimeter; and the image projector is configured to project on the target surface, wherein a distance of the image projector to the target surface is at least 1 meter, preferably at least 2 meters.

4. The projection device of any preceding claim, wherein, The image detector and the image projector share at least a portion of an optical path within the projection device.

5. The projection device of claim 4, further comprising a first optical element configured to direct an incident light beam from the subject through at least a portion of the shared optical path and towards an image sensor of the image detector; and a second optical element configured to direct an exit light beam from a light source of the image projector through at least a portion of the shared optical path and towards the target surface.

6. The projection apparatus of claim 5, wherein, The first optical element is the second optical element and comprises a reflective side and a transmissive side.

7. The projection apparatus of claim 6, wherein, The reflective side is configured to direct an incident signal towards the image sensor, and the transmissive side is configured to receive a light signal from the light source of the image projector and allow the light signal to pass through.

8. The projection device of any of claims 4 to 7, further comprising a lens, wherein the shared optical path passes through the lens to the target surface.

9. The projection device of any one of the preceding claims, further comprising a second image detector having a second detector field of view of the subject, wherein, the second image detector is configured to generate a second input image of the subject, and wherein the processor is configured to generate the output image further based on the second input image.

10. The projection device of any one of the preceding claims, further comprising an emitter configured to emit radiation towards the target surface at a predefined frequency.

11. The projection apparatus of claim 10, wherein, the emitter is configured to excite the radiation at a radiation frequency corresponding to an excitation frequency of an element in the subject, wherein the image detector is configured to detect an excitation light of the element.

12. The projection device of any of the preceding claims, wherein, the image detector comprises at least one of: an image detector of a hyperspectral or multispectral camera, an infrared camera, a thermal camera, or a laser Doppler imaging system.

13. The projection device of any of the preceding claims, wherein, the processor is configured to: determine a first distance between an image detector and a target surface; determine a second distance between an image projector and the target surface; determine an angle between the image projector and the image detector, wherein a vertex of the angle is at the target surface; and map detected pixels of a detected image to projected pixels of the projected image based on the first distance, the second distance, and the angle.

14. The projection device of any one of the preceding claims, further comprising a fixture for fixing the device during operation or a handle for holding the device in a hand during operation.

15. A method of projecting an image, the method comprising: detecting, by an image detector having a detector field of view, an input image of a subject; generating, by a processor, an output image based on the input image, wherein the output image comprises content representing at least a portion of content of the input image, wherein the generating comprises mapping a pixel of the output image to at least one pixel of the input image, wherein a tilt of an output light ray associated with the pixel of the output image corresponds to a tilt of an input light ray associated with the at least one pixel of the input image, the tilt being with respect to the projection device, and determining a value of the output pixel based on at least one value of the at least one input pixel mapped to the output pixel; projecting, by an image projector, the output image on a target surface of the subject by an image projector having a projector field of view, wherein the projector field of view overlaps the detector field of view.

Images