Head mounted display apparatus and method of modulating a source of illumination thereof

Abstract

A head mounted display (HMD) device comprising a source of illumination and a method of modulating a source of illumination for the HMD are disclosed. The source of illumination emits light frontally of the HMD towards a target in a scene. An imaging sensor captures light re-emitted by at least a portion of the target in consequence of its illumination by the emitted light, as image data. An intensity of the emitted light is calibrated as an illumination constraint. The HMD device modulates light emission at the source of illumination by comparing either distance data computed from image data captured by the imaging sensor, or sensor data output by an environmental sensor of the HMD, with the illumination constraint. Processed image data is output to display means of the HMD.

Description

HEAD MOUNTED DISPLAY APPARATUS AND METHOD OF MODULATING A SOURCE OF ILLUMINATION THEREOFFIELD OF INVENTION

[0001] The invention belongs to the field of head mounted display (‘HMD’) apparatuses and more particularly, but not exclusively, to HMD apparatuses operating and / or configured with sources of illumination.BACKGROUND TO INVENTION

[0002] Head mounted display (‘HMD’) devices have gained popularity in surgical procedures, for their capacity to provide the wearer with surgical context-relevant digital information within the wearer’s field of view during a procedure. HMDs are digital display devices with some onboard data processing capacity, which project digital video content in a user’s direct field of view, either in superposition to the user’s real physical environment in the case of see-through augmented reality (‘AR’) glasses, or as computer-generated imagery in replacement of that real physical environment in the case of virtual reality (‘VR’) headsets or still, lately, as a combination of captured physical environment and computer-generated imagery in the case of mixed reality (‘MR’) headsets.

[0003] Functional capacities of HMD devices are in a state of constant evolution, as improvements to miniature electronic components open opportunities to accommodate new fields of application. A specific example well suited to HMDs intended for medical procedures, is fluorescence-based imaging, which has gained comparable popularity as intra-operative imaging technology. This technique involves injecting a fluorophore or contrast agent, for instance indocyanine green (ICG), in a patient to allow Near-Infrared (NIR)-based detection and visualization of target tissue intended for observation or surgical removal. In use, the ICG absorbs light emitted by a light source, typically with a wavelength of e.g. 780 nm and is excited into fluorescence, whereby the light is re-emitted at a slightly higher wavelength, e.g. in the range 830 to 850 nm. A NIR-sensitive imaging sensor configured with suitable cut-off and band-pass filters detects the re-emitted light and generates corresponding NIR image data.

[0004] Fluorescence-based imaging systems based upon the above principles use a NIR illuminator and camera arrangement tethered to a remote processing unit, outputting captured image data inclusive of fluorescent tissue to a remote display unit. Such NIR arrangements of the prior art are typically either a static or semi-static device mounted to an adjustable cantilevered arm,or a mobile wand-like device, in either case held at a predetermined distance of the target tissue, wherein the NIR light source illuminates the tissue with a predetermined intensity of at least 1 mW per cm2across the illuminated area, the camera captures excited tissue at that predetermined distance and outputs it to the remote processing unit, which processes it for output to the remote display unit.

[0005] Given the ergonomic disadvantages of such fluorescence-based imaging systems, notably the transient occlusion of some or all of the light source by a surgeon’s or nurse’s head or hand during a procedure, and the requirement for a surgeon or nurse to hold a wand-like NIR device in situ throughout the procedure, equipping an HMD with a NIR illuminator and camera arrangement appears desirable, moreover to prevent any spatial disconnect for the surgeon when periodically shifting their field of vision between the surgery site with the actual tissue to work upon, and the remote display unit showing the fluorescent tissue.

[0006] However, as an HMD device is located on the wearer’s head, accordingly when a surgeon wears a HMD device during a procedure, the distance between the headset and the surgical site may range from typically 25 to 35 cm when the surgeon is sitting adjacent the operating table, up to 45 to 60 cm when the surgeon is standing instead, and any variations between or beyond as surgeons move frequently during procedures for adjusting their observation of surgical tasks and interim results.

[0007] This diversity of poses necessarily affects the intensity of any illumination cast by the illumination source at the surgical site, since the larger the distance between the illuminator and the tissue gets, the larger the illuminated area gets, wherein the illumination intensity per cm2reduces in proportion, introducing variance in the rate of illumination of the contrast agent-containing tissue and, therefore, a non-trivial risk of fluorescent tissue escaping the surgeon’s attention.

[0008] Accordingly, there is a requirement for augmenting the functional capacities of conventional HMD apparatuses combining imaging and environmental sensors.SUMMARY OF INVENTION

[0009] Aspects of the invention are set out in the accompanying claims, respectively aimed at various embodiments of a head mounted display (HMD) apparatus, and various embodiments of a method of detecting a target with the HMD apparatus.

[0010] In a first aspect, the present invention provides a head mounted display (HMD) device comprising a source of illumination capable of emitting light frontally of the HMD and towards a target; an imaging sensor capable of capturing light, re-emitted by at least a portion of the target in consequence of its illumination by the emitted light, as image data ; display means ; and data processing means operably connected to the source of illumination, the imaging sensor and the display means, configured to calibrate an intensity of the emitted light as an illumination constraint, compute distance data representative of a distance between the imaging sensor and the at least portion of the target based on the image data, modulate light emission at the source of illumination by comparing the computed distance data with the illumination constraint, and output processed image data to the display means.

[0011] This technique advantageously automates the regulation of the intensity of illumination to maintain a rate or level of illumination substantially constant at the level of the target, accordingly maintaining optimal illumination of the region of interest corresponding to the target for the HMD wearer without any requirement of interacting with a menu, selection or other form of interface.

[0012] In embodiments of the HMD apparatus, the imaging sensor may comprise a monocular camera, whereby the image data is a sequence of image frames, and the data processing means is configured to compute distance data by processing successive image frames. Alternatively the imaging sensor may comprise a pair of cameras in a stereoscopic arrangement, whereby the image data is a sequence of left and right image frames, and the data processing means is configured to compute distance data by processing left and right image frames.

[0013] Herein, re-emission should be understood as the process by which a target absorbs light, of any type including visible, infrared, and ultraviolet light, and then emits it back out in a different direction. Suitably, the source of illumination may comprise at least one near infrared (‘NIR’) light source emitting light at a specific NIR wavelength, the imaging sensor may be a NIR imaging sensor configured to output NIR image data, and the target or portion thereof may comprise tissue excited by, absorbing, or reflecting the emitted NIR light.

[0014] The source of illumination may also, or instead, comprise at least one visible light source emitting light at a specific wavelength. A wavelength of the re-emitted light may be in the same or a different range than the specific wavelength. For example light absorbed by tissue, such as veins during vascularization analysis, may be re-emitted or reflected in substantially the same specific wavelength as the emitted light, whereas NIR light absorbed by tissues injected with afluorophore may be re-emitted in a wavelength different to that of the emitted NIR light, or reflected to a varying extent by surrounding tissue.

[0015] In embodiments of the HMD apparatus, the source of illumination may comprise at least first and second light sources, with the first light source emitting light in a specific wavelength outside the visible spectrum, the second light source emitting light in a specific wavelength within the visible spectrum, the first and second light sources having substantially the same collimation as each other and the second light source being configured to implement a visible edge with light in the visible spectrum peripherally of an area of illumination of the first light source. This configuration advantageously provides the HMD user with a visible reference frame within which to observe tissue excited by, absorbing, or reflecting the emitted non-visible, e.g. NIR, light.

[0016] In embodiments of the HMD apparatus, the source of illumination may be either integral with the HMD device, or releasably secured thereto, for example as an illumination module. Accordingly embodiments of the HMD apparatus may further comprise a modular interface, wherein the source of illumination is releasably secured to the HMD apparatus via the modular interface for powering and receiving light intensity modulating commands from the data processing means. This feature advantageously increases the range of illumination types for use with a given HMD, wherein illumination modules of varying types may be configured with distinct light sources of specific wavelengths, e.g. for use with different fluorophores, and / or in combination with monochromatic illumination, e.g. white.

[0017] In embodiments of the HMD apparatus, the source of illumination may be located in a housing having an aperture distal the source of illumination and an internal surface intermediate the source of illumination and the aperture, configured as a light guide having maximal internal reflection. Light guides advantageously focus the initial illumination without little to no loss relative to conventional mirror-like configurations, whereby they are accordingly more energy efficient, advantageously for autonomous HMD devices powered with an internal battery. Embodiments are considered wherein the source of illumination may comprise a plurality of light sources with respective light guides, combining energy efficiency with simplicity of modulation through the selective switching of individual light sources to effect the adjustment required by the modulation.

[0018] Various embodiments of the light guide configuration are considered. In embodiments of the HMD apparatus, a shape of the or each aperture may have an aspect ratio in proportion with an image-capturing ratio of the imaging sensor, wherein the or each light guide is preferably collimated with the imaging sensor. The light guide may be configured as a hollow reflector, withthe internal surface having a root mean square (RMS) height (Sq) in the range of 50 picometers (pm) to 5 micrometers (pm), through finishing the entire internal surface of the housing intermediate the light source and the aperture uniformly to the requisite measure of smoothness, analogous to polished glass; alternatively, through the application of a coating with similar properties to the entire internal surface. Alternatively, or additionally, the light guide may comprise one or more lens-like members, i.e. the volume bounded by the entire internal surface of the housing intermediate the light source and the aperture may be filled with a solid member, made of glass, polycarbonate or other fully transparent material, effectively embodying a homogeneous lenslike member, having a surface boundary with the requisite degree of smoothness. Such an embodiment may be further enhanced with configuring the immediately adjacent internal surface of the housing with a high-reflection coating.

[0019] In certain embodiments of the HMD, the data processing means may be further configured to process the image data to detect oversaturated pixels and decrease light emission at the source of illumination when a count of oversaturated pixels exceeds a threshold count. This technique can be relied upon to omit both the computation of distance data from image data and the modulation of light emission at the source of illumination by comparing that computed distance data, advantageously reducing the computational requirements when automatically adjusting the rate of illumination of the illumination module.

[0020] In another aspect, the present invention provides a head mounted display (HMD) apparatus comprising a source of illumination capable of emitting light frontally of the HMD apparatus and towards a target ; an imaging sensor capable of capturing light, re-emitted by at least a portion of the target in consequence of its illumination by the emitted light, as image data ; an environmental sensor capable of detecting an environmental characteristic frontally of the HMD apparatus and outputting sensor data representative of the detected characteristic ; display means ; and data processing means operably connected to the source of illumination, the imaging sensor, the environmental sensor and the display means, configured to calibrate an intensity of the emitted light as an illumination constraint, modulate light emission at the source of illumination by comparing the sensor data with the illumination constraint, and output processed image data to the display means.

[0021] In embodiments of the HMD apparatus, the environmental sensor may be a ranging sensor, wherein the environmental characteristic is a surface of the target and the sensor data is a distance value between the environmental sensor and the target surface.

[0022] In embodiments of the HMD apparatus, the data processing means may be further configured to selectively modulate light emission at the source of illumination by comparing the sensor data with a safety constraint. The safety constraint may be a maximum permissible distance value, past which the ratio of modulation extends to switching off the visible light source.

[0023] Alternatively, or additionally, when the environmental sensor includes a low-resolution imaging capacity, the environmental characteristic may be a color of the target, wherein the sensor data is a color of the target surface and the safety constraint is a predetermined color, or color range. These techniques are particularly suitable for embodiments of the apparatus incorporating a highly-focused source of illumination emitting illumination at a level apt to blind an onlooker, wherein the automation of the illumination rate is provided with a safety protocol if the HMD wearer should momentarily orient the source of illumination away from the target towards said onlooker.

[0024] In a further aspect, the present invention provides a method of modulating a source of illumination for a head mounted display (HMD) device, comprising steps of emitting light frontally of the HMD and towards a target with a source of illumination, whereby at least a portion of the target re-emits light in consequence of its illumination by the emitted light ; capturing at least the re-emitted light with an imaging sensor of the HMD device as image data ; and with data processing means of the HMD device, calibrating an intensity of the emitted light as an illumination constraint, computing distance data representative of a distance between the imaging sensor and the at least portion of the target based on the image data, modulating light emission at the source of illumination by comparing the computed distance data with the illumination constraint, and outputting processed image data to display means of the HMD device.

[0025] With embodiments of the HMD apparatus wherein the imaging sensor comprises a monocular camera and the image data is a sequence of image frames, the step of computing distance data may further comprise processing successive image frames. Alternatively, with embodiments of the HMD apparatus wherein the imaging sensor comprises a pair of cameras in a stereoscopic arrangement and the image data is a sequence of left and right image frames, the step of computing distance data may further comprise processing left and right image frames.

[0026] With embodiments of the HMD apparatus wherein the source of illumination comprises at least one near infrared (‘NIR’) light source, the imaging sensor is a NIR imaging sensor and the target or portion thereof comprises tissue excited by emitted NIR light, the step of emitting light may further comprise emitting NIR light with the NIR light source at a specific NIRwavelength, and the steps of capturing and outputting may further comprise capturing the light re-emitted in a different range than the specific NIR wavelength and outputting NIR image data.

[0027] With embodiments of the HMD apparatus wherein the source of illumination comprises at least a first light source emitting light in a specific wavelength outside the visible spectrum, and at least a second light source emitting light in a specific wavelength within the visible spectrum, the step of emitting light further comprises emitting light outside the visible spectrum with the first light source and emitting light within the visible spectrum with the second light source, and the method may comprise the further steps of arranging the first and second light sources to have substantially the same collimation as each other, and configuring the second light source to implement a visible edge with visible light peripherally of an area of illumination of the first light source.

[0028] With embodiments of the HMD apparatus, wherein a shape of the or each aperture has an aspect ratio in proportion with an image-capturing ratio of the imaging sensor, the method may wherein the further step of collimating the or each light guide with the imaging sensor.

[0029] With embodiments of the HMD apparatus comprising a modular interface, the method may comprise the further steps of releasably securing the source of illumination to the HMD device via the modular interface; and powering the source of illumination and modulating its light emission through the modular interface.

[0030] Embodiments of the method may comprise the further steps of processing the image data to detect oversaturated pixels and decreasing light emission at the source of illumination when a count of oversaturated pixels exceeds a threshold count wherein, when the count of oversaturated pixels exceeds the threshold count, the steps of computing the distance data and modulating light emission at the source of illumination by comparing the computed distance data with the illumination constraint are omitted.

[0031] In still another aspect, the present invention provides a method of modulating a source of illumination for a head mounted display (HMD) device, comprising steps of emitting light frontally of the HMD and towards a target with a source of illumination, whereby at least a portion of the target re-emits light in consequence of its illumination by the emitted light ; capturing at least the re-emitted light with an imaging sensor of the HMD device as image data ; detecting an environmental characteristic with an environmental sensor of the HMD device frontally thereof and outputting sensor data representative of the detected characteristic ; and with data processing means of the HMD device, calibrating an intensity of the emitted light as an illumination constraint, modulating light emission at the source of illumination by comparing the sensor datawith the illumination constraint, and outputting processed image data to display means of the HMD device.

[0032] With embodiments of the HMD apparatus wherein the environmental sensor is a ranging sensor, the environmental characteristic is a surface of the target, and the sensor data is a distance value between the environmental sensor and the target surface, the step of modulating further comprises selectively modulating light emission at the source of illumination by comparing the distance value with the illumination constraint.

[0033] With embodiments of the HMD apparatus wherein the source of illumination is potentially harmful to onlookers within the field of illumination, the step of modulating may further comprise selectively modulating light emission at the or each light source by comparing the sensor data with a safety constraint.

[0034] Other aspects of the invention are set out in the accompanying claims.BRIEF DESCRIPTION OF DRAWINGS

[0035] The invention will be more clearly understood from the following description of an embodiment thereof, given by way of example only, with reference to the accompanying drawings, in which: -Figure 1 illustrates a first embodiment of a head mounted display (HMD) device according to the invention, comprising a source of illumination, a camera and an optional environmental sensor.Figure 2 is a block diagram of a hardware architecture for the HMD device shown in Figure 1 , including data processing means, a memory and displays.Figure 3 shows a user wearing the HMD of Figures 1 and 2 and illustrates the illumination range of the NIR light source and the respective fields of view of the camera and ranging sensor.Figure 4A details data processing steps performed by the HMD device of Figures 1 to 3 in a first embodiment of the method of illumination according to the invention, when the HMD device excludes an environmental sensor.Figure 4B details data processing steps performed by the HMD device of Figures 1 to 3 in a second embodiment of the method of illumination according to the invention, when the HMD device includes an environmental sensor.Figure 5 illustrates the contents of the memory of Figure 2 at runtime when performing the steps of either Figure 4A or Figure 4B.Figure 6 illustrates a first embodiment of the source of illumination shown in Figures 1 to 4B, with a point source of light wherein the housing is integral with the HMD and configured as a hollow reflector.Figure 7 illustrates a second embodiment of the source of illumination shown in Figures 1 to 4B, with several point sources of light and respective homogeneous lens-like members within the housing, configured with a modular interface for releasable securing to the HMD.Figure 8 shows a detail of a homogenous lens-like member shown in Figure 7.Figure 9 illustrates a third embodiment of the source of illumination shown in Figures 1 to 4B, with several point sources of light of varying types and respective, substantially oblong homogeneous lens-like members.Figure 10 details data processing steps performed by the HMD device of Figures 1 to 3 in a further embodiment of the method of illumination according to the invention.Figure 11 illustrates the contents of the memory of Figure 2 at runtime when performing the steps of Figure 10.DETAILED DESCRIPTION OF DRAWINGS

[0036] There will now be described by way of example specific modes contemplated by the inventor. In the following description and accompanying figures, numerous specific details are set forth in order to provide a thorough understanding, wherein like reference numerals designate like features. It will be readily apparent to one skilled in the art, that the present invention may be practiced without limitation to these specific details. In other instances, well known methods and structures have not been described in detail, to avoid obscuring the description unnecessarily.

[0037] The inventive concept, based upon the principle of illuminating a target area in a scene frontally of an HMD device with one or more lights of a specific wavelength, wherein the emitted light creates a reflection or excitation in the target area, such as patient tissue, in the same or a different wavelength, relates to the automatic modulation of the emitted light for adjusting its intensity at the target area, as a function of one or more characteristics detected by one or more sensors of the HMD device in its environment.

[0038] With reference to Figures 1 to 5 initially, wherein like numerals reference like features, an embodiment of a head mounted display (‘HMD’) device is shown, in the example an augmented reality (‘AR’) HMD device 10. The AR HMD device comprises a wearer visor 20, which includes optical combiners to optically combine computer-generated content with the real view of the world observable through a main see-though portion 22. The virtual content is rendered on eye-respective micro video displays 24A, 24B located equidistantly of a central bridge portion overlying a wearer’s nose in use, that are placed outside the wearer’s field of view.

[0039] Each micro video display portion 24A, 24B consists of a respective video display unit, in the example a micro OLED panel with a minimum 60 Hz frame refresh rate and a resolution of 1920x1080 pixels, located proximate a lower edge of the visor so as to leave the see-though portion 22 extending above it and up to its upper edge. Collimation lenses are placed between each micro display 24A, 24B and optical combiner to focus the virtual 2D image whereby the display portions 24A, 24B implement, perceptually, a single video display 24 occupying a subset of the front aspect of the HMD device, wherein the wearer can observe both the ambient physical environment across the see-though portion 22 and the computer-generated content at a predefined and comfortable viewing distance on a virtual image plane, namely the display focal plane.

[0040] The HMD 10 comprises at least one source of illumination 30 capable of emitting light 32 of a specific wavelength towards a target in the environment frontally of the HMD, e.g. a surgery room wherein an HMD wearer 36 faces a patient 38 lying atop an examination table in the example shown in Figure 3. In the first embodiment of the HMD shown and described, the source of illumination is a near infrared (‘NIR’) light source 30, which emits NIR light 32 in a specific wavelength selected in the range 700 to 1 ,100 nm, for example 780 nm. The NIR light source 30 of the example is selected for fluorescence-based imaging and may be focused for energetic efficiency. Accordingly, upon administering a contrast agent to tissue of the patient 38, all or a portion of the patient tissue 39 that is targeted for observation, surgical removal or other purpose, re-emits the received light in a different wavelength, for example in the range 830 to 850 nm. Skilled readers will appreciate that the principle disclosed herein readily extends to any other type of target, including non-biological such as markers, for example a passive NIR marker of the Aruco™ type releasably secured to a location of the patient 38 selected for an orthopaedic procedure ; likewise to other types of illumination, wherein further embodiments described herein incorporate one or more visible light source(s) which emit light in a specific wavelength selected in the range 350 to 700 nm.

[0041] The HMD device 10 further comprises at least one imaging sensor 40 capable of capturing light re-emitted by the target tissue or portion thereof 39 in consequence of its illumination by the source of illumination 30. Adverting to the source of NIR illumination of the example, accordingly the imaging sensor 40 is a high-resolution or full-resolution (‘NIR’) camera 40, with a field of view (‘FoV’) 42 of typically 70 to 90 degrees and which captures light in a wavelength range of typically 350 to 1100 nm. An alternative embodiment contemplates at least a pair of cameras 40 arranged in a stereoscopic configuration, shown in dotted lines in Figure 1 . The (or each) NIR camera 40 outputs the captured re-emitted light as image data 340, namely asequence of image frames 442, each at a resolution of 1280x1024 pixels or higher, and at a rate of 60 frames per second or more, wherein only light that is re-emitted by the target tissue or portion thereof 39 is encoded, illustrated as a solid grey color in Figure 3 for contrast relative to the patient and operating table content shown in dotted lines to denote their absence in each image frame 340.

[0042] Optionally, the HMD device 10 further comprises an environmental sensor 50 capable of detecting at least a portion of a target frontally of the HMD device and outputting sensor data representative of the target or portion thereof. In the embodiment shown and described, the environmental sensor 50 is a ranging sensor, for example a low-power single-zone or multi-zone time of flight (ToF) sensor, which does not require any specific computing unit, as even a microcontroller can process its output data, including under a relatively high capture or polling rate of 60 Hz or more. The optional environmental sensor 50 has a respective FoV 52 similar to the FoV 42 of the imaging sensor 40, comprising a single detection zone or cell or several such zones e.g. arranged in a matrix, having a detection range d. The environmental sensor continuously polls the environment frontally of the HMD device 10 for targets 39 located in the range d within its FoV 52, as illustrated in Figure 3, for instance in the range 200 to 700 centimetres wherein targets of interest are expected to appear, for example tissue 39 of the patient 38 injected with a contrast agent and fluorescing under NIR illumination.

[0043] The skilled reader will understand that the present technique may be practiced with various types of environmental sensors which, subject to their characteristics and capacities, may allow a detection of target according to type and / or operational distance. By way of non-limitative example, the inventive concept disclosed herein is considered readily implementable by the skilled person with stereoscopic sensors, ToF sensors and matrix array sensors, with which sensor data can be selected as one or more of e.g. distance data and color data ; likewise depth- capable (3D perceptive) sensors with which different distances can be measured for different parts of the target tissue when the line of sight of the HMD device 10 is not substantially orthogonal to the target.

[0044] The HMD device 10 further comprises data processing means operably connected to the (or each) source of illumination 30, the (or each) imaging sensor 40, the optional environmental sensor 50 and the displays 24A, 24B and, optionally, a data connectivity capacity. In particular, the HMD device 10 includes a data processing unit (‘CPU’) 201 , which is a general-purpose microprocessor, for instance according to the Cortex™ architecture manufactured by ARM™, acting as the main controller of the HMD device 10. The CPU 201 may further include a dedicated image signal processing (‘ISP’) unit or module to receive and pre-process image datagenerated by the imaging sensor 40 before outputting the corresponding image data to the CPU 201. When present, this ISP unit is either integral or coexists with the CPU 201 , that is programmed to perform other data processing tasks described hereafter. The CPU 201 is coupled with memory means 202, comprising volatile random-access memory (RAM), non-volatile random-access memory (NVRAM) or a combination thereof. The CPU 201 and memory 202 communicate data bilaterally over a data input / output bus 203, to which the other components of the HMD 10 are similarly connected, in order to provide headset functionality and receive user commands.

[0045] The data connection between the (or each) imaging sensor 40 and the CPU 201 , via the bus 203 or another, is a high-frequency data communication interface which is sensitive to external electromagnetic interference (EMI) and must be shielded accordingly. The respective data connections between the CPU 201 and the source of illumination 30 and between the CPU 201 and the optional environmental sensor 50, via the bus 203 or another, are implemented as low- frequency data communication interfaces, for instance according to the l2C protocol, wherein any EMI over that interface is negligible considering the integer data type and volume output by the ranging sensor.

[0046] User input data may be received directly from a physical input interface 204, which may be one or more buttons, including at least an on / off switch, and / or a portion of the HMD casing configured for haptic interaction with a wearer’s touch. User input data may also be received as analogue sound wave data by a microphone 205, for which the CPU 201 (or a DSP unit or module, not shown) implements an analogue-to-digital converting function, and processed audio data is output to a speaker unit 206.

[0047] Power is supplied to all components by an electrical circuit 207, which is interfaced with an internal battery module 208, wherein the battery is periodically recharged by an electrical converter 209. Power is supplied to the above components by an electrical circuit 207, which is interfaced with an internal battery module 208, wherein the battery is periodically recharged by an electrical converter 209.

[0048] Embodiments of the HMD device may further include networking means 210, shown in dotted line in the figure as a wireless network interface card or module (WNIC) also connected to the data input / output bus 203 and the electrical circuit 207, apt to interface the HMD with a wireless local area network (‘WLAN’) generated by a local wireless router. Alternative or additional wireless data communication functionality may be provided by the same or anothermodule, for example implementing a short-range data communication according to the Bluetooth™ and / or Near Field Communication (NFC) interoperability and data communication protocol.

[0049] Data processing configuration and basic functionality of the HMD device 10 of Figures 1 to 3 is now described by reference both to Figure 4A, which shows a logic implementable by a HMD device 10 without an environmental sensor 50 and to Figure 4B, which shows a logic implementable by a HMD device 10 comprising an environmental sensor 50, wherein data structures stored in the memory 202 and processed by the CPU 201 under the respective methods are commonly shown in Figure 5, and wherein like numerals reference likes features.

[0050] When powering up the HMD 10, an operating system (‘OS’) 501 is initially loaded at step 401 , for governing basic data processing, interdependence and interoperability of HMD components 201 to 209, including the WNIC 210 when present. The HMD OS may be based on Android™ distributed by Google™ of Mountain View, California, United States. The OS 501 includes subroutines for reading and processing input and output data, optionally including subroutines 502 to configure the HMD device 10 for bilateral network communication with remote terminals via the WNIC 210 interfacing with a network router device. Still at step 401 , a set of instructions 503 embodying a controlling application for the source of illumination 30 is loaded, either as a subroutine of the OS 501 or in a higher computational layer. When loaded in a higher computational layer, the controlling application 503 is interfaced with the source of illumination 30 and the ranging sensor 50 through the OS 501 via one or more Application Programmer Interfaces (API) 504.

[0051] At any specific time at runtime, data stored and circulating within the example architecture shown in Figure 2 according to the principles described herein includes the image data 340 output by the imaging sensor(s) 40, light source-modulating command data 530 generated by the CPU 201 , calibration data 533 representative of an intensity of the light emitted by the source of illumination 30, then either computer distance data 540 generated by the set of instructions 503 by reference to Figure 4A specifically, or sensor data 550 generated by the environmental sensor 50 by reference to Figure 4B specifically, and also image data processed by the CPU 201 and output to the display units 24A, 24B plus, optionally, processed audio data output to the speaker unit 206.

[0052] By reference to Figure 4A, at step 402 the HMD 10 initializes the source of illumination 30 and the imaging sensor 40. The source of illumination 30 continuously emits NIR light in a specific NIR wavelength, initially at a default intensity, whereby any target tissue 39 frontally of the HMD 10 gets illuminated by same, tissue injected with a contrast agent therein gets excitedby the emitted NIR light, and re-emits light in a wavelength or range thereof different from the specific NIR wavelength. The imaging sensor 40 continuously captures the light re-emitted by at least a portion of the target 39 in consequence of its illumination by the NIR light, as the image data 440, and outputs that image data to the CPU 201 .

[0053] By reference to Figure 4B, the HMD 10 also initializes the environmental sensor 50 at step 402, whereby the environmental sensor 50 loads a discrete set of operating instructions 505, e.g. a sensor firmware, with which it initializes its detecting cell(s) for detecting one or more environmental characteristics : whenever a target 39 is present within its respective FoV 52 within the distance interval d, the ranging sensor 50 of the example generates a value representative of the environmental characteristic dependent upon the detected target 38, in the example a distance between the ranging sensor 50 and a surface of the detected target within its FoV 52, comprising target tissue 39 injected with the contrast agent. By reference to the example scene depicted in Figure 3, the patient tissue 38 is located approximately 60 centimetres from the sensor 50 wherein the or each sensor cell corresponding to same within the FoV 52 would be assigned a distance value of e.g. ‘60’ by way of sensor data.

[0054] In a computing context, the processing required for target detection as described herein with reference to Figure 4A typically resides in higher computing layers, e.g. at the application level, which is advantageous for processing operations involving complex models, but requires significant computational resources and power draw. The technique presented in Figure 4B improves this approach by displacing the operational requirement of detecting targets to a lower computing layer, at the OS-kernel level, with significant relief on computing and power resources, at the expense of an additional component 50. This is made possible by the low-frequency interface of the environmental sensor 50 to the CPU 201 , recalling that even so simple a data processing unit as a microcontroller can process sensor values and output same to the CPU across that interface.

[0055] The HMD 10 may optionally load an application program 506 for additional data processing functionality, for example post-processing and rendering local and / or remote data to a user interface 507 which is initialized at step 403 and output to the VDUs 24A, 24B. In the absence of an optional application program 506, the HMD 10 still initializes a user interface 507 for the OS at said step 403, which is output to the VDUs 24A, 24B.

[0056] At step 404, the illumination controlling application 503 invokes a calibration routine for the source of illumination 30, in order to adapt its intensity currently at the default level to specifics of the situational and environmental circumstances. By reference to illumination ratesdeemed optimal for ICG-based NIR applications, the routine can require the HMD wearer to locate the front of the HMD device 10 at a predetermined distance from the target tissue 39, wherein user adjustment instructions are relayed in the user interface 507 and are updated in quasi real-time by processed distance data 540, or substantially in real-time by sensor data 550 from the ranging sensor 50, alternatively at a predetermined distance from an external calibration reference, for example a physical reference pattern wherein adjustments are made until the position of image data 340 of the reference pattern in the GUI 507 is matched to the wearer’s view of the physical reference pattern through the see-through portion 22, over a predetermined period of time.

[0057] The predetermined distance is preferably set as a function of the default intensity of the source of illumination 30, for instance by reference to a stored distance value known to achieve the desired illumination rate with the default intensity at the target tissue 39. Alternatively the predetermined distance can be set with the physical reference pattern, for instance by reference to physical indicia printed on the pattern and corresponding registration indicia displayed as display data on the initialized user interface 507, which the wearer needs to adjust in coincidence by displacing their head.

[0058] With reference to Figure 4B specifically, the predetermined period of time can be set as a function of the known time-to-complete of the initialization routine of the ranging sensor 50. The calibration routine establishes coincidence between the stored distance and the actual distance (SD) measured by the ranging sensor 50 through the target detection, over the predetermined period deemed to provide sufficient operational accuracy from the ranging sensor 50 since starting the HMD 10 and which, when achieved, is set by the illumination controlling application 503 as a situational datum embodying an illumination constraint (IC) 533 at step 405.

[0059] Skilled readers will appreciate that the calibration of step 404 may be performed once, at the time of assembling and configuring the HMD device 10 into new equipment. The HMD imaging and environmental sensors and the source of illumination are pre-calibrated with regard to their intrinsic parameters, likewise the 6 degrees of freedom (‘DoF’) transformations therebetween, as 3x4 matrices. In the example, a three-dimensional coordinate system L originates at the source of illumination 30, a three-dimensional coordinate system E originates at the imaging sensor 40 and a three-dimensional coordinate system R originates at the environmental sensor 50. The 6 DoF transformations are known between R and E, i.e. Tf and between E and L, i.e. T and are accordingly pre-computed. Since the respective intrinsic parameters of the environmental and imaging sensors, for instance based on a pinhole (or other) camera model, are known, then these transformations are respectively defined as:wherein fx ,fy ,fE,fE, correspond to the sensors’ focal lengths in the x and y dimensions and Cx>cy>cx>cy> correspond to the sensors’ center of projections, likewise in the x and y dimensions, respectively. To calculate real distance data of the target area 39 in the presence of precomputed calibration, given the depth image data IR= {pf, . . . , p„) of resolution n e N captured by the environmental sensor 50, where each 2D point p = (xE,yE) represents a pixel location and corresponds to depth value d(p ), wherein 0 < i < n, then the 3D position PE= (x , YiR,zr) corresponding to each 2D position p , is extracted by:

[0060] Each 3D position PEis transformed from R to E via PE= TRPEand projected to the imaging sensor’s image plane to get the corresponding 2D position:which implies that= Z . Given the image data IEcaptured by the imaging sensor 40, a set of m e N relevant points illuminated by the illumination source 30 in the environment frontally of the HMD device 10 are detected and their corresponding 2D positions {pf, ... , pE} are extracted. For each 2D point pE, where 0 < j < m, the corresponding 3D position PEis extracted by looking up the matching 2D position pR^E. Each 3D position PEis next transformed from E to L via PE= TEPE. Given {PE, P^}, processed distance value(s) 540 or actual distance value(s) 550 for the target 39 area is or are extracted and used to correct the illumination constraint.

[0061] Upon completing the calibrating routine, image data 340 received by the CPU 201 from the imaging sensor 40 is processed by the CPU configured by the OS 501 at step 406 for out- putting to the user interface 507 at step 407, wherein the processed image data is optionally augmented with the application program 506 when present.

[0062] In parallel with the output to step 407, with reference to Figure 4A specifically, the imaging sensor 40 remains operative to capture and output image data 340, inclusive of target tissue 39 present within it FoV 42, as soon as it is initialised whereby, as soon as the set of instructions 503 is loaded, image data 340 received by the CPU 201 from the imaging sensor 40 and comprising data representative of target tissue 39 is processed by the CPU configured by the illumination controlling application 503 at step 406 for extracting distance data between the targettissue 39 and the imaging sensor 40. The processing of image data to extract distance data 540 therefrom at step 406 in the method of Figure 4A, is for instance based on techniques assessing the parallax present between two corresponding images of the same scene frontally of the HMD 10 and which, subject to the technical embodiment of the imaging sensor 40, for example either a monocular camera 40 or a pair of cameras 40 in a stereoscopic arrangement, may comprise successive frames in a sequence thereof, or simultaneously-captured left and right image frames (or a sequence of same). The illumination controlling application 503 accordingly processes each successive set of image frames to extract a distance value between the imaging sensor 40 and the surface of the target tissue 39 encoded in the image frames, and thus continuously outputs processed distance data 540 to the CPU 201 at step 408A.

[0063] In an alternative embodiment, which optimizes the logic of step 406 to immediately adjust the illumination rate of the source of illumination in case of oversaturation, and optionally reduce computational requirements associated with distance determination and accordingly reduce power consumption within the HMD device 10, the CPU is further configured to process the image data to detect oversaturation, indicative of an excessive rate of illumination at the surface of the target 39 by the illumination module 30, at step 406, prior to extracting distance data 540 from the image data 340. Where image data 340 comprises pixels containing single channel intensity value, representative of oversatured pixels in the image frame, and I is the intensity image represented as a two-dimensional array, wherein each element represents the intensity value of a pixel, WEN is the width of the image frame, hEN is the height of the image frame, and Ti is a threshold intensity value, then the number of pixels above Ti can be computed via: c = Y y T ' / H J l=l. 0 otherwise

[0064] Where L is a value representative of a rate of illumination of the source of illumination 30, that value can then be adjusted based on Ci at step 408 with using:wherein K is a constant factor. Another technique for adjusting the rate of illumination value, may be via a predefined look-up table, which may use L =f (Ci), wherein L is the illumination intensity and f (Ci) is the look-up table function, which maps the count of pixels to illumination intensity. Yet another technique for adjusting the rate of illumination value via a predefined look-up table, may instead use Lnew=Lold*f(Ci), wherein Laid is the current or last value, Lnew is the adjusted value and f (Ci) is the look-up table function, which maps the count of pixels to an intensity adjustment factor. Where these techniques determine an oversaturation from excessiveillumination of the target 39 based on pixel count in the image data 340, the logic may proceed directly to step 410 for immediately adjusting the rate of illumination to compensate the oversaturation, whilst the steps of processing of image data to determine a distance value (408A) and comparing (409) are performed. Embodiments may instead implement an interrupt for steps 408A, 409, since these are computationally more onerous and may be bypassed for computational efficiency and a reduced power draw.

[0065] Alternatively still, and still in parallel with the sequence of steps 406-407, but now with reference to Figure 4B specifically, as the ranging sensor 50 remains operative as soon as initialised independently of the CPU 201 activity and tasks whilever the HMD device 10 remains in use, target tissue 39 present within the FoV 52 of the ranging sensor 50 continuously triggers its detection zone(s), representative of a detection event, in real time. The ranging sensor 50 processes each detection event with determining a value for the one or more zone(s) involved by that respective detection event, in the example the distance between itself and the surface of the target tissue 39, and thus continuously outputs distance data (SD) as environmental sensor data 550 to the CPU 201 at step 408B. In this embodiment, the processing of image data 340 at step 406 advantageously excludes the computation (extrapolation) of distance data, accordingly relieving the data processing load placed upon the CPU 201.

[0066] At step 409, the CPU 201 configured by the illumination control application 503 compares, according to the embodiment, either the computed distance data 540 of step 408A or the real-time distance data (SD) 550 of step 408B, with the illumination constraint (IC) distance data 533 of step 405 to compute a positive or negative difference relative to the situational datum, wherein a positive difference is representative of movement of the HMD 10 away from the target tissue 39, requiring an increase in the intensity of the source of illumination 30 for maintaining the desired illumination rate at the target tissue 39 and, reciprocally, a negative difference is representative of the reverse situation, requiring an decrease in the intensity of the source of illumination.

[0067] At step 410, the CPU 201 configured by the illumination control application 503 computes command data 530 for modulating the light intensity of the source of illumination 30. In embodiments wherein oversaturation is detected at step 406, the command data 530 is computed according to the adjustment of the value representative of a rate of illumination, as an intensity decrease. Where no oversaturation is determined, and in embodiments omitting such oversaturation determination, the command data 530 is computed according to the comparison result of step 409, wherein the command data encodes an intensity increase in case of a positive difference, respectively an intensity decrease in case of a negative difference, and wherein the valuefor the intensity increase or decrease is computed in proportion to the scale of the difference relative to the normative reference that is the illumination constraint 533. The computed command data 530 is then output to the source of illumination 30 for modulating its intensity at step 411.

[0068] Control thereafter returns to a next iteration of step 406, wherein the next image data 340 (N+1) received from the imaging sensor 40 is processed, with reference to Figure 4A, for extracting the next processed distance data 540 from the next sequence of image frames 340 and, with reference to Figures 4A or 4B irrespective, for outputting to the user interface 507 at a next iteration of step 407.

[0069] This next image data is captured by the imaging sensor 40 in optimal lighting conditions irrespective of a change in the distance between the HMD 10 and the target tissue 39 since the previous image data 340 was captured, processed and output, as a result of the real-time adjustment of the source of illumination intensity per steps 409 to 411 , which maintains the rate of illumination of the target tissue 39 substantially at the rate embodied by the illumination constraint 533.

[0070] By reference to Figure 4B specifically, the environmental sensor 50 remains operative independently of the CPU 201 activity and tasks whilever the HMD device 10 remains in use, and its firmware may implement additional functionality, notably a switching between active and idle states, inclusive of the source of illumination 30 and / or imaging sensor(s) 40, according to preset periods of non-detection, for enhanced power conservation.

[0071] The skilled reader will appreciate that the automatic adjustment of the rate of illumination of the HMD’s illumination module 30, has been described by reference to a distance-based comparison technique in Figure 4A and to 4B by way of non-limitative example, and that both the illumination constraint (IC) 533 and alternative environmental factors may be determined from the image data 340 (408A) or detected by the sensor 50 may differ from a distance values, without departing from the inventive concept described herein. In particular, the illumination constraint (IC) 533 may be calibrated at step 404 as a level of brightness in the scene captured by the imaging sensor 40 and the alternative environmental determined from the image data 340 (408A) may be either a level of brightness of the target 39 or, in the case of a NIR illumination, e.g. the absence of target 39 or portion thereof in an image frame, indicative of a requirement to increase the intensity of illumination.

[0072] With reference to Figures 6 to 9 now, wherein like numerals reference like features, specific embodiments of the source of illumination 30 are contemplated by the inventors, which are optimized for minimal draw on the battery 208 by combining a low-power light source with a light guide optimized for maximum reflection, collimating the emitted light according to the optical principle of Total Internal Reflection (‘TIR’) for defining an area of illumination at the target having a sharp peripheral edge between adjacent lit and unlit surfaces.

[0073] Total internal reflection occurs when light within a transparent volume, for example a homogenous bloc of glass or polycarbonate, encounters a boundary with a material having a lower index of refraction, such as air, at an angle greater than the critical angle. According to Snell’s Law, the angle at which light emerges from the material depends upon both the angle at which the light hits the boundary of the material, and the difference in the refractive index of the material: the higher the index difference and the greater the angle of incidence, the greater the bending. Thus, for certain shapes with linear planes or boundaries, such as rectangular plates, pipes and cylinders, it is possible for light to become trapped inside the guide until it reaches edges or an end of the guide opposed to the light source, wherein it is transported with minimal losses.

[0074] A first embodiment of the source of illumination 30 shown both in a front aspect and a side elevation in Figure 6 comprises an illumination module 30 embodied as a permanent feature of the HMD front fascia. The module 30 comprises a housing 610 internally of the HMD device 10, wherein a point source of light 620, for instance a light emitting diode (‘LED’) or an arrangement or array of closely-adjacent LEDs, is contained within the housing proximate a first closed extremity thereof, distal the HMD device front surface. The LED 620 is supplied with power by the electrical circuit 207 of the HMD device 10 and interfaced with the CPU 201 through the data bus 203. Lasers are considered another suitable example of point source of light compatible with the principles and embodiments presently described herein, which may be preferred for certain applications but at the cost of a higher power consumption relative to LEDs and the like.

[0075] The housing 610 comprises an aperture 630 at a second extremity opposed to the first, accordingly distal the LED 620 and opening frontally of the HMD device 10 towards potential target(s) 39. The LED 620 has a smaller dimension than the aperture 630 and the internal volume 640 of the housing extending therebetween is substantially circular, accordingly a shape of the internal volume 640 is generally frusto-conical, with the LED 620 located substantially at its apex and wherein an inner surface 650 of the internal volume tapers outward, towards and up to the aperture 630.

[0076] The internal volume 640 is configured as a light guide, to collimate substantially all of the light emitted by the LED 620 frontally of the HMD device 10 towards potential target(s) 39. Depending on the illumination wavelength of the LED 620 and the intended application, the internal volume 640 may be hollow, wherein the inner surface 650 is preferably straight with high smoothness e.g. with a root mean square (RMS) height (Sq) in the range 50 pm to 5 pm, or by applying a coating to the surface having corresponding properties: materials configured into light guides can bend and take on more complex shapes, but light leakage occurs most at any bends, dots and / or other surface asperities, at which light trapped by TIR encounters the guide boundary at less than the critical angle and thus escapes the light guide.

[0077] A known formula for the total fraction of scattered light (Bennett and Porteus, 01-Feb- 1961 , Journal of the Optical Society of America) is:wherein TISBP is the total integrated scatter, Ro is the nominal reflection of the surface (assumed 1.0 for TIR), 0j is the angle of incidence, Rqis the surface roughness and A is the wavelength. Using this formula requires that the units of Rq and A must be the same, e.g. both nanometres, microns, or others. Although the formula is exponential, it is effectively linear when (Rq A) < Angles of incidence within the internal volume 640 range from about 45 degrees to more than 80 degrees, whereby an angle of 1 radian is typical. At this angle of incidence for e.g. NIR light with a wavelength of 780nm, a surface roughness of 1nm gives a TIS value under 1 %. However, as most of this scatter is forward scatter, i.e. light that is scattered only a limited angle of deviation away from the specular beam, this light is not lost, because it will still land within the illumination field, even though it may blur marginally the edge of that field.

[0078] The interface between the LED 620 and the light guide 640 is designed to minimize Fresnel loss. Suitably the LED 620 may be a flat-type LED rather than a lens-type LED, affixed centrally of a substantially planar entrance surface 660 of the light guide, i.e. co-axially with a main axis A of the internal volume 640, for an optimal combination of light extraction properties with a simple and compact design. A front aspect or surface 670 of the light guide 640 at the level of the aperture 630 and orthogonal to the entrance surface 660 is designed as a diffusor.

[0079] Accordingly, light rays emitted by the LED 620 strike the input face 660 at any angle and are bent at the interface such that they experience TIR at the walls 650 of the light guide. Once light rays have entered the light guide, they bounce about and between the curved surfaces 650and keep travelling until they reach the diffusor 670 at the far end of the light guide, wherein any light passing through the light guide entrance 660 face undergoes TIR inside the guide.

[0080] A second embodiment of the source of illumination 30 shown in a front aspect and side elevation in Figure 7 comprises an illumination module 700 configured as a removable feature of the HMD device 10. For this embodiment, the HMD device comprises a modular interface (not shown) adapted to releasably secure the module 700 thereto, and to transmit both power and light intensity commands to the module in use, thus interfaced with both the electrical circuit 207 and the CPU 201 through the data bus 203.

[0081] In this embodiment, the illumination module 700 again comprises a housing 610, an underside of which is configured with a module interface 710 designed to engage with and releasably secure to the HMD device’s modular interface, for instance with a tongue-and-groove configuration 720 comprising suitable power and data interconnects, such as spring-loaded (or ‘pogo’) pins, and a releasable clip or magnetic mechanism. In this embodiment still, the module 700 comprises a plurality of point sources of light 620I.N, for instance several light emitting diodes (‘LED’) of lower individual power relative to the unique LED of the first embodiment, arranged in a matrix, contained within the housing proximate a first lateral extremity thereof, and supplied with power and interfaced with the CPU 201 through the mated modular interfaces.

[0082] The housing 700 comprises a plurality of light guides, one respective for each point light source 620I.N, each having an aperture 630N at a second lateral extremity opposed to the first, thus distal the corresponding point light source 620Nand opening frontally of the illumination module 700 towards potential target(s) 39 frontally of the HMD device 10. The internal surface 640 of each light guide extending peripherally of each point light source 620I-N towards and up to the corresponding aperture 630N is again configured for Total Internal Reflection (‘TIR’), as a hollow reflector in order to collimate light emitted by the point light source 620.

[0083] Skilled readers will appreciate that the principles disclosed herein contemplate sources of illumination 30 distinct from the HMD device 10, but operatively connected therewith for at least data communication. The principles disclosed herein also contemplate sources of illumination 30 configured with an internal power source for supplying the light source(s), in order to relieve the draw on the HMD device’s battery 208, preferably factoring in ergonomic requirements, in particular the total weight of an illumination module with its own battery when designed for removable attachment to and frontally of an HMD device 10.

[0084] With reference to Figure 8, rather than be configured as a hollow reflector, the or each internal volume 640 may be occupied by a bloc 800 of solid material with a high refractive index, for instance an optically transparent plastic or polycarbonate material, in the example having a frusto-conical outer shape matching the frusto-conical shape of the inner surface 650 of each light guide, and provided with its high reflection characteristic by suitable surface finishing of the bloc’s outer surface to the requisite degree of smoothness, or by applying a suitable coating thereto, i.e. intermediate the outer longitudinal surface of the bloc 800 and the housing inner curved surface 650.

[0085] The skilled person will readily understand that the principles of a light guide implementing TIR and a modular interface described by reference to the source of illumination 30 may be extended to other types of illuminators and sensors, advantageously accommodating a variety of illuminators with respective wavelengths of illumination, for example for illuminating and detecting contrast agents with different fluorophore wavelengths, and extending to visible light sources and combinations of multiple light types, including laser light sources. For example, for illuminators comprising light source(s) with a specific wavelength outside the visible spectrum, the addition of at least one visible light source having substantially the same collimation as that of the non-visible light-source(s), and highly focused to implement a visible edge at the periphery of the juxtaposed area of illumination, advantageously informs the user about the area of illumination of the non-visible light source(s).

[0086] A simple variant with multiple type light sources is shown in Figure 9, wherein like numerals still reference like features. A third embodiment of the source of illumination 30 is shown in a front aspect, which comprises a removable illumination module 900 with a housing 910, having an underside configured with a module interface 710 as previously described. The module 900 also comprises a plurality of point sources of light, in this embodiment combining five NIR light LEDs 620 and a visible light LED 920, e.g. a white light, arranged in a matrix, with the visible light LED mounted centrally of a top row of three point light sources, the five NIR light LEDs forming a U-shape about the white light LED.

[0087] The housing 910 again comprises a plurality of light guides, one respective for each point light source 620, 920, wherein the internal surface 940 of each light guide extending peripherally of each point light source 620, 920 towards and up to the corresponding aperture is again configured for Total Internal Reflection (‘TIR’), either as a hollow reflector or with a lens-like bloc 800, in order to collimate light emitted by the point light source 620, 920 but, in this embodiment, having a square cross-section whereby a shape of each internal volume corresponding to arespective LED, likewise a shape of each bloc 800 if used, is generally a trapezoidal prism matching the trapezoidal prism shape of the inner surface.

[0088] The apertures, and optional lens members 800 when present, have substantially the same collimation as each other, i.e. each light guide is collimated with the imaging sensor 40, and the cross-section of each aperture, e.g. oblong or square, has an aspect ratio in proportion with the image(-capturing) aspect ratio of the imaging sensor 40. Accordingly, the area respectively illuminated by each of the visible (920) and non-visible (620) light sources is e.g. suitably oblong itself, suitably matching the aspect ratio of the image data 340 captured by NIR camera 40 and wherein the white light generated by the visible light LED 920 defines a sharp edge, which remains visible peripherally of the total area of illumination of the module 900, itself illuminated in a homogenous manner.

[0089] There are multiple advantages to this specific configuration. As the NIR surface illumination is matched and effectively bounded by the visible surface illumination area, wherein any illumination beyond that periphery is minimal and drop offs sharply, the HMD wearer 36 remains visually aware at all times, of which portion of the target 39 within their FoV is NIR illuminated, whilst the rate of illumination is independently adjusted in real-time. Moreover the NIR surface illumination remains matched with the registration area of the NIR camera, inclusive of any zooming in or out thereof, accordingly facilitating an intuitive self-correction by the HMD wearer 36, of any difference perceived between the imaging sensor FOV 42 and the illumination field of the light module at different distances, again whilst the rate of illumination is adjusted independently in real-time.

[0090] Given the modularity of design for the source of illumination 30 and the multiplicity of light types implementable therein, the inventors contemplate alternative embodiments of the method of illumination control. By way of non-limitative example, for variants sources of illumination 30 incorporating a plurality of light sources, e.g. point light sources 620, of a same illumination type, e.g. NIR of a specific wavelength, the controlling command may be computed at step 410, and the command data 530 may encode accordingly, to selectively switch one or more discrete light sources 620I.Non or off according to the adjustment in light intensity determined from the comparison of step 409.

[0091] Alternatively, for variants sources of illumination 30 incorporating a plurality of light sources with respective illumination types, e.g. wherein one or more point light sources are NIR illuminators with a distinct specific wavelength relative to others, the method may comprise an initial step of selecting the light source(s) suitable for the situational and environmental circumstances amongst all others, at or prior to the calibration and setting of steps 404, 405, whereinthe controlling command is computed at step 410, and the command data 530 encode accordingly, to selectively modulate only the selected light source(s) 620I.Naccording to the adjustment in light intensity determined from the comparison of step 409.

[0092] Alternatively still, variant sources of illumination 30 incorporating one or more light source(s) may implement respective illumination types with an optical filtration arrangement modifying the specific wavelength of the one or more light source(s) according to the HMD wearer’s specific requirement during use.

[0093] Alternatively still, or additionally, embodiments of the method may implement a safety protocol for variants sources of illumination 30 incorporating highly-focused light source(s), such as the module 900. In such embodiments, one of which illustrated in Figures 10 and 11 , the data processing means is further configured by the controlling application 503 to selectively modulate the light intensity of the light source with the processed distance data 540 or the sensor data 550 according to one or more safety constraint(s) 1100, integrated in the illuminator firmware 503 loaded at step 402 (1002).

[0094] Subject to the type and capacities of environmental sensor 50, at least one safety constraint 1100 may be a distance threshold representative of a maximum permissible distance 550 for detecting a target 39, at or past which the controlling application 503 overrides the computation of controlling command data of step 410 according to the comparison of step 409, for the command data 530 to encode significant dimming of the light source instead, for example through selectively dimming or even switching off the, or a majority of, LEDs of the source of illumination.

[0095] The illuminator firmware 503 includes a comparison step 1009 wherein the detected environmental data, i.e. the processed distance data 540 of step 408A or the real-time distance data 550 of step 408B, is compared to the safety constraint (SC) 1100 for determining a positive or negative difference, wherein a positive difference is representative of a safe level of illumination frontally of the HMD 10 and the logic resumes with steps 409 to 411 , but a negative difference is representative of a unsafe level of illumination frontally of the HMD 10 and, irrespective of any command data computed at step 410 according to the comparison result of step 409, a computation of override command data 530 is triggered at step 1010, which throttles the rate of illumination to a maximum safe level, at or below the safety constraint 1100. The override command data 530 is then output to the source of illumination 30 for modulating its intensity to a safe level at step 411 as before.

[0096] Alternatively still, or additionally, embodiments of the method may implement a remote modulation protocol for embodiments of the HMD 10 apparatus including a data networking capacity, for example via the WNIC 210 when present. In such embodiments, the data processing means 201 is further configured to connect with a remote terminal, e.g. a smartphone, tablet computer or another HMD device, processing a local instance of the controlling application 503, and receiving and displaying image data 340 received from the HMD device 10 over the network connection. By reference to the example shown in Figures 1 to 3, such a remote terminal may be operated e.g. by a healthcare professional observing and / or advising the HMD wearer 36 from a remote location. The remote data processing terminal is configured by its local controlling application 503 to process user input, e.g. a light intensity selection, into network override command data 530 and to communicate the network override command data 530 to the HMD device 10 over the network connection. At the HMD device 10, the network override command data 530 is then output to the source of illumination 30 for modulating its intensity at step 411 as before, to the level instructed by the remote observer.

[0097] Environmental sensors 30 are known to include a color detection capacity, in addition to or alternatively to a distance detection capacity, wherein the safety constraint 1100 may comprise one or more color ranges representative of human skin color, either as an alternative to the distance threshold, or complementarily to the distance threshold as a secondary variable for validating a decision to override the main illumination modulation logic for safety. These techniques and color-based variants advantageously provide safety protocols for sources of illumination 30 implementing highly focused light, which mitigate a risk of blinding persons near or facing the wearer 36 in the environment frontally of the HMD device 10, should the wearer lift their head upwards during use.

[0098] The combination of an environmental sensor with a source of illumination and methods described herein accordingly optimises user machine interaction, providing faster, accurate and automated techniques for focusing the visual perception of the HMD wearer onto targets in their line of sight in the environment, that are targeted for interaction such as a surgical procedure, e.g. breast cancer nodules. Medical applications generally expected to benefit from all embodiments of the HMD device with one or more sources of illumination modulated according to the technique disclosed herein include, non-exhaustively, those based upon auto-fluorescence of veins (e.g. taking blood, blood perfusion and vascularization analysis in any surgery), fluoro- phore detection (oncology, lymph nodes, blood perfusion and vascularization analysis in any surgery) and target markers (i.e. orthopaedics, oncology, robotics).

[0099] In the specification the terms "comprise, comprises, comprised and comprising" or any variation thereof and the terms include, includes, included and including" or any variation thereof are considered to be totally interchangeable and they should all be afforded the widest possible interpretation and vice versa. The invention is not limited to the embodiments hereinbefore de- scribed but may be varied in both construction and detail.

Claims

CLAIMS1. A head mounted display (HMD) device comprising- a source of illumination capable of emitting light frontally of the HMD and towards a target; an imaging sensor capable of capturing light, re-emitted by at least a portion of the target in consequence of its illumination by the emitted light, as image data ; display means ; and data processing means operably connected to the source of illumination, the imaging sensor and the display means, configured to calibrate an intensity of the emitted light as an illumination constraint, compute distance data representative of a distance between the imaging sensor and the at least portion of the target based on the image data, modulate light emission at the source of illumination by comparing the computed distance data with the illumination constraint, and output processed image data to the display means.

2. The HMD device according to claim 1 , wherein the imaging sensor comprises a monocular camera, the image data is a sequence of image frames, and the data processing means is configured to compute distance data by processing successive image frames.

3. The HMD device according to claim 1 , wherein the imaging sensor comprises a pair of cameras in a stereoscopic arrangement, the image data is a sequence of left and right image frames, and the data processing means is configured to compute distance data by processing left and right image frames.

4. The HMD device according to any of claims 1 or 3, wherein the source of illumination comprises at least one near infrared (‘NIR’) light source emitting light at a specific NIR wavelength, the imaging sensor is a NIR imaging sensor configured to output NIR image data, and the target or portion thereof comprises tissue excited by, absorbing and / or reflecting emitted NIR light, a wavelength of the re-emitted light being in the same or a different range than the specific NIR wavelength.

5. The HMD device according to any of claims 1 to 4, wherein the source of illumination comprises at least one visible light source emitting light at a specific wavelength, and the target or portion thereof comprises tissue excited by, absorbing and / or reflecting the emitted light, a wavelength of the re-emitted light being in the same or a different range than the specific wavelength of the visible light.

6. The HMD device according to claim 4 or 5, wherein the source of illumination comprises at least a first light source emitting light in a specific wavelength outside the visible spectrum, at least a second light source emitting light in a specific wavelength within the visible spectrum, the first and second light sources having substantially the same collimation as each other and the second light source being configured to implement a visible edge with visible light peripherally of an area of illumination of the first light source.

7. The HMD device according to any of claims 1 to 6, wherein a shape of the or each aperture has an aspect ratio in proportion with an image-capturing ratio of the imaging sensor and wherein the or each light guide is collimated with the imaging sensor.

8. The HMD device according to any of claims 1 to 7, wherein the source of illumination is integral with the HMD device or releasably secured thereto.

9. The HMD device according to claim 8, wherein the source of illumination comprises a housing having an aperture distal the or each light source and an internal surface intermediate the or each light source and its respective aperture, the or each internal surface being configured as a light guide.

10. The HMD device according to any of claims 1 to 9, wherein the data processing means is further configured to process the image data to detect oversaturated pixels, decrease light emission at the source of illumination when a count of oversaturated pixels exceeds a threshold count, and optionally omit to compute the distance data and to modulate light emission at the source of illumination by comparing the computed distance data with the illumination constraint.

11. A head mounted display (HMD) device comprising- a source of illumination capable of emitting light frontally of the HMD and towards a target; an imaging sensor capable of capturing light, re-emitted by at least a portion of the target in consequence of its illumination by the emitted light, as image data ; an environmental sensor capable of detecting an environmental characteristic frontally of the HMD and outputting sensor data representative of the detected characteristic ; display means ; and data processing means operably connected to the source of illumination, the imagingsensor, the environmental sensor and the display means, configured to calibrate an intensity of the emitted light as an illumination constraint, modulate light emission at the source of illumination by comparing the sensor data with the illumination constraint, and output processed image data to the display means.

12. The HMD device according to claim 11 , wherein the environmental sensor is a ranging sensor, the environmental characteristic is a surface of the target, and the sensor data is a distance value between the environmental sensor and the target surface, whereby the data processing means is further configured to selectively modulate light emission at the source of illumination by comparing the distance value with the illumination constraint.

13. The HMD according to claim 11 or 12, wherein the data processing means is further configured to selectively modulate light emission at the source of illumination by comparing the sensor data with a safety constraint.

14. The HMD device according to claim 13, wherein the environmental sensor comprises a low-resolution imaging sensor, the environmental characteristic is a color of the target, the sensor data is a color of the target surface and the safety constraint is a predetermined color or range thereof, whereby the data processing means is further configured to selectively modulate light emission at the source of illumination by comparing the color with the predetermined color or range thereof.

15. The HMD device according to any of claims 11 to 14, wherein the source of illumination is integral with to the HMD device or releasably secured thereto.

16. The HMD device according to claim 15, further comprising a modular interface adapted to power the source of illumination and to transmit light intensity modulating commands from the data processing means thereto, wherein the source of illumination is releasably secured to the HMD device via the modular interface.

17. An illumination module comprising a source of illumination for use with the HMD device according to any of claims 1 to 16.

18. A method of modulating a source of illumination for a head mounted display (HMD) device, comprising the steps of- emitting light frontally of the HMD and towards a target with a source of illumination,whereby at least a portion of the target re-emits light in consequence of its illumination by the emitted light ; capturing at least re-emitted light with an imaging sensor of the HMD device as image data ; and with data processing means of the HMD device, calibrating an intensity of the emitted light as an illumination constraint, computing distance data representative of a distance between the imaging sensor and the at least portion of the target based on the image data, modulating light emission at the source of illumination by comparing the computed distance data with the illumination constraint, and outputting processed image data to display means of the HMD device.

19. The method according to claim 18, wherein the imaging sensor comprises a monocular camera, the image data is a sequence of image frames, and the step of computing distance data further comprises processing successive image frames.

20. The method according to claim 18, wherein the imaging sensor comprises a pair of cameras in a stereoscopic arrangement, the image data is a sequence of left and right image frames, and the step of computing distance data further comprises processing left and right image frames.

21. The method according to any of claims 18 to 20, wherein the source of illumination comprises at least one near infrared (‘NIR’) light source, the imaging sensor is a NIR imaging sensor and the target or portion thereof comprises tissue excited by emitted NIR light, the step of emitting light further comprising emitting NIR light with the NIR light source at a specific NIR wavelength, and the steps of capturing and outputting further comprising capturing the light re-emitted, in a same or different range than the specific NIR wavelength and outputting NIR image data.

22. The method according to any of claims 18 to 21 , wherein the source of illumination comprises at least one visible light source and the target or portion thereof comprises tissue excited by, absorbing and / or reflecting the emitted light, the step of emitting light further comprises emitting visible light with the visible light source at a specific wavelength, and the steps of capturing and outputting further comprise capturing the light re-emitted, in a same or different range than the specific wavelength and outputting image data.

23. The method according to claim 21 or 22, wherein the source of illumination comprises at least a first light source emitting light in a specific wavelength outside the visible spectrum, and at least a second light source emitting light in a specific wavelength within the visible spectrum, the step of emitting light further comprises emitting light outside the visible spectrum with the first light source and emitting light within the visible spectrum with the second light source, the method comprising the further steps of- arranging the first and second light sources to have substantially the same collimation as each other, and configuring the second light source to implement a visible edge with visible light peripherally of an area of illumination of the first light source.

24. The method according to any claims 18 to 23, wherein a shape of the or each aperture has an aspect ratio in proportion with an image-capturing ratio of the imaging sensor and wherein the step of combining further comprises collimating the or each light guide with the imaging sensor.

25. The method according to any of claims 18 to 24, comprising the further steps of- providing the source of illumination with a housing having an aperture distal the or each light source and an internal surface intermediate the or each light source and its respective aperture, and configuring the or each internal surface as a light guide.

26. The method according to any of claims 18 to 25, wherein the HMD device comprises a modular interface, the method comprising the further steps of- releasably securing the source of illumination to the HMD device via the modular interface ; and optionally powering the source of illumination via the modular interface ; and optionally transmitting light intensity modulating commands from the data processing means to the source of illumination via the modular interface.

27. The method according to any of claims 18 to 26, comprising the further steps of processing the image data to detect oversaturated pixels ; decreasing light emission at the source of illumination when a count of oversaturated pixels exceeds a threshold count ; and wherein, when the count of oversaturated pixels exceeds the threshold count, the method optionally further comprises omitting the steps of computing the distance data and modulating light emission at the source of illumination by comparing the computed distance data with the illumination constraint.

28. A method of modulating a source of illumination for a head mounted display (HMD) device, comprising the steps of- emitting light frontally of the HMD and towards a target with a source of illumination, whereby at least a portion of the target re-emits light in consequence of its illumination by the emitted light ; capturing at least re-emitted light with an imaging sensor of the HMD device as image data ; and with data processing means of the HMD device, calibrating an intensity of the emitted light as an illumination constraint, detecting an environmental characteristic with an environmental sensor of the HMD device frontally thereof and outputting sensor data representative of the detected characteristic ; and with data processing means of the HMD device, calibrating an intensity of the emitted light as an illumination constraint, modulating light emission at the source of illumination by comparing the sensor data with the illumination constraint, and outputting processed image data to display means of the HMD device.

29. The method according to claim 28, wherein the environmental sensor is a ranging sensor, the environmental characteristic is a surface of the target, and the sensor data is a distance value between the environmental sensor and the target surface, wherein the step of modulating further comprises selectively modulating light emission at the source of illumination by comparing the distance value with the illumination constraint.

30. The method according to claim 28 or 29, wherein the step of modulating further comprises selectively modulating light emission at the source of illumination by comparing the sensor data with a safety constraint.

Images