Binocular Device

Abstract

A binocular device for visualizing optical radiation includes a support structure and left and right cameras coupled to the support structure, the left camera including left optics and a left image sensor, and the right camera including right optics and a right image sensor, the left and right image sensors configured to produce left and right video signals from optical radiation received from corresponding left and right input optics for the same field of view along corresponding left and right input optical axes, and a specular reflection is detected.

Description

[Technical field]

[0001] The present invention relates to a binocular device for visualizing optical radiation.The present invention further relates to a binocular device for visualizing visible and invisible radiation. [Background technology]

[0002] When performing surgery, surgeons use bright lights in the operating room to be able to identify as much as possible the tissues to be treated. However, not everything can be seen clearly that way. For example, certain tissue types, such as certain tumors, are not visible to the human eye. These tissue types can sometimes be visualized by near-infrared imaging techniques. This may be the case for tumor tissue.

[0003] Suman B. Mondal et al., "Binocular Goggle Augmented Imaging and Navigation System provides real-time fluorescence image guidance for tumor resection and sentinel lymph node mapping," Scientific Reports, vol. 5, no. 1, July 2015, discloses a system with a near-infrared (NIR) light source with LEDs and band-pass filters, and a white flashlight or surgical light covered with a short-pass filter as a white light source. The imaging module collects the combined color NIR signal through a custom glass lens. The input signal was split into visible and NIR components by a custom dichroic beam splitter cube and directed to separate color and NIR sensors. The NIR and color sensors were co-aligned. A Windows x64 PC produces a GUI that generates the superimposed color NIR image and accesses the display, storage, and processing functions of the image data, and replicates the image for simultaneous display on the PC and the head-mounted display module. The display module consists of a head-mounted display. Summary of the Invention

[0004] It would be advantageous to provide an improved visualization device. To address this problem, according to one aspect of the present invention, there is provided a binocular device for visualizing optical radiation, comprising: A support structure; a left camera and a right camera coupled to the support structure, the left camera comprising a left optical system and a left image sensor, and the right camera comprising a right optical system and a right image sensor, the left image sensor and the right image sensor configured to produce left and right video signals from detected optical radiation received from corresponding left and right input optics for a same field of view along corresponding left and right input optical axes; The device further comprises a processing unit, the processing unit comprising: receiving signals representative of a left image from the left camera and a right image from the right camera, the left image and the right image being captured by the left camera and the right camera, respectively, substantially simultaneously; To compare the left image with the right image, and A binocular device is provided that is configured to detect the specular reflection based on a result of the comparison.

[0005] According to another aspect, Light emission A method for visualizing receiving radiation along respective left and right input optical axes for the same field of view by left and right input optics coupled to a support structure and transmitting the light to a left image sensor of a left camera and a right image sensor of a right camera, respectively, the left and right cameras being coupled to the support structure; producing left and right video signals from the detected radiation received by the left and right cameras, respectively; receiving, by a processor, signals representative of a left image from the left camera and a right image from the right camera captured substantially simultaneously by the left and right cameras, respectively; comparing the left and right images by a processor; detecting, by the processor, a specular reflection based on a result of the comparison.

[0006] According to another aspect, there is provided a binocular device for visualizing optical radiation, comprising: A support structure; a left camera and a right camera coupled to the support structure, the left camera comprising a left optical system and a left image sensor, and the right camera comprising a right optical system and a right image sensor, the left image sensor and the right image sensor being configured to produce left and right video signals from optical radiation received and detected from corresponding left and right input optics for a same field of view along corresponding left and right input optical axes; a left display and a right display coupled to the support structure, the left display and the right display arranged to be viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display, and configured to present left and right video images formed in visible light by the left and right displays based on the left and right video signals, respectively; A binocular device is provided which is configured, in a particular visualization mode, to alternately show a left image based on a left video signal on a left display and a right image based on a right video signal on a right display.

[0007] This allows the user to distinguish specular reflections.

[0008] According to another aspect, Light emission A method for visualizing receiving radiation along respective left and right input optical axes for the same field of view by left and right input optics coupled to a support structure and transmitting the light to a left image sensor of a left camera and a right image sensor of a right camera, respectively, the left and right cameras being coupled to the support structure; producing left and right video signals from the detected radiation received by the left and right cameras, respectively; presenting, by a left display and a right display, left and right video images formed in visible light based on the left and right video signals, respectively, the left and right displays coupled to a support structure and viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display; A method is provided in which the presenting step includes alternately showing, in a particular visualization mode, a left image based on a left video signal on a left display and a right image based on a right video signal on a right display.

[0009] This allows the user to distinguish specular reflections.

[0010] It would be advantageous to provide an improved visualization device. To address this problem, in accordance with one aspect of the present invention, a binocular device for visualizing visible and invisible radiation is provided. The binocular device comprises: A support structure; a left camera and a right camera coupled to a support structure, the left camera comprising left optics and a left image sensor, and the right camera comprising right optics and a right image sensor; the left and right image sensors are configured to produce left and right video signals from optical radiation received and detected from the corresponding left and right input optics along corresponding left and right input optical axes and for the same field of view; At least one of the cameras is sensitive to both radiation in the invisible radiation wavelength band and radiation in the visible light radiation wavelength band, and the input optics of at least one of the cameras is transparent to the invisible wavelength band and transparent to the visible light wavelength band. Reduction It is sexuality.

[0011] The combination of features can be useful in presenting a stereoscopically realistic depiction of features in the visible and invisible wavelength ranges. Reduction Input optics that are reflective can help improve image quality. This is based on the idea that the intensity of received invisible radiation, such as infrared radiation, is in most cases much smaller than the intensity of visible light. In many practical situations, visible light is abundant, but the intensity of the invisible radiation wavelength band is much smaller. The reduction of the visible light radiation wavelength band brings the intensity levels of both wavelength bands of radiation closer to each other, without excessively reducing the reduction of the invisible radiation wavelength band. This can improve image quality, especially in combination with low-cost and / or lightweight optics and image sensors.

[0012] For example, the infrared may be near infrared (NIR). In applications involving viewing diffuse reflection or fluoroscopic radiation in the near infrared wavelength band of radiation, the intensity of the associated near infrared radiation received is often much less than radiation received in the visible wavelength band.

[0013] The binocular device may further comprise a left display and a right display coupled to the support structure, the left display and the right display arranged to be viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display, and configured to present left and right video images formed in visible light by the left and right displays based on the left and right video signals, respectively, thereby creating a display device, for example in a head-mounted or handheld device, with a built-in camera and display that improves visualization of invisible wavelength bands.

[0014] The input optics of a camera sensitive to radiation in invisible wavelength bands can include a polarizing filter that includes at least one layer of polarizing material. Polarizing filters with desired properties can be constructed of particularly lightweight and cost-effective materials. For example, the materials used in many sunglasses significantly reduce the intensity of visible light while transmitting most of certain invisible radiation wavelength bands, such as near infrared and infrared.

[0015] A polarizing filter can comprise at least two layers of polarizing material with mutually orthogonal polarization directions. In this way, about 98%-99% of visible light can be blocked in the form of a relative spectral linearity. Furthermore, about 90%-95% of radiation of a certain invisible wavelength, such as the near infrared wavelength range, can be transmitted through the polarizing filter.

[0016] The binocular device may further include a light source coupled to the support structure capable of generating radiation within at least the invisible and visible wavelength bands.

[0017] Preferably, the light source is configured to generate an emission beam of visible light and an emission beam of invisible radiation that are aligned, for example by optical elements, to be substantially identical in geometry and position, in this way the detected image may be more consistent.

[0018] The light source can include a polarizing filter configured to polarize the visible light output by the light source within the visible wavelength band and transmit radiation in the invisible wavelength band, the polarization direction of the light source polarizing filter being approximately orthogonal to the polarization direction of the polarizing filter of the input optics, which is another way to significantly reduce the amount of visible light while retaining most of the invisible radiation.

[0019] The input optics for a camera sensitive to invisible radiation may include a diaphragm having an aperture, the diaphragm surrounding the aperture being sensitive to light in the visible wavelength range. Reduction The aperture is transparent to light in the infrared wavelength band, while the aperture is transparent to light in the visible wavelength band. This allows for selective application of the aperture to the visible light wavelength band, while passing radiation in the invisible wavelength band, substantially without the effect of the aperture. In addition to reducing the intensity of visible light compared to the intensity of invisible radiation, this feature allows for improved utilization of the optical system. Conventional lenses are known to have different focal points for different radiation wavelengths due to dispersion. To allow optimal focusing for each wavelength band of radiation (e.g., red, green, blue, and infrared), complex optical systems are required, for example, by separating each wavelength band into separate bundles and focusing each bundle separately using separate optics. Using an aperture with a relatively small aperture increases the depth of focus, thereby reducing this problem, but also reduces the intensity of the radiation. Considering that the intensity of visible light is much higher than that of radiation in the invisible wavelength band, the aperture described herein has the advantages of an aperture for visible light without reducing the low intensity radiation in the invisible wavelength band. The depth of focus is increased for the visible light radiation wavelength band, providing a sharp image of the visible light. This allows for optimizing the focus of the input optical system for the invisible radiation wavelength band. This makes it possible to simplify the input optical system since it is not necessary to take dispersion into consideration.

[0020] The input optics may comprise a lens with an autofocus configured to focus radiation in the invisible wavelength bands. This may be done using known autofocus functions. In this way, each wavelength band recorded by the camera may be imaged sharply.

[0021] The input optical system of an infrared camera is designed to detect light in the visible wavelength range in addition to the aperture. Reduction To further reduce the visible light intensity, an additional filter for visible light can be added, where the additional filter does not have an aperture.

[0022] The input optics of a camera sensitive to radiation in the invisible wavelength bands can be equipped with a filter containing iodine to selectively reduce radiation in the visible wavelength bands, iodine being known to reduce such radiation while being transparent to certain invisible wavelength bands, such as infrared or near infrared wavelength bands.

[0023] The binocular device may comprise a processing unit, the processing unit being configured to receive signals representing a left image from the left camera and a right image from the right camera, captured by the left camera and the right camera, respectively, at approximately the same time, to compare the left and right images, and to detect specular reflections based on the results of the comparison. This is convenient for detecting specular reflections. It is observed that the processing unit does not need to be fixed to the support structure. However, the processing unit may have a communication connection (wired or wireless) for exchanging video signals with the image sensor and the display. The left and right images may be captured at approximately the same time while a light source connected to the support structure is switched on and emits radiation in the visible and invisible wavelength bands. In this way, the reflections may have a more predictable appearance.

[0024] The binocular device may further comprise a light source coupled to the support structure for generating at least infrared light and visible light, the light source being configured to emit infrared light intermittently while keeping the visible light intensity substantially constant, and the camera being configured to capture at least one image with infrared light emission and at least one image without infrared light emission. This provides an improved quality image, since the visible light image does not suffer from degradation that may be caused by the emission of radiation in the invisible wavelength range. Furthermore, flickering of the visible wavelength range may not be caused. Furthermore, this may enable an improved quality image to be generated by combining the image captured with infrared light emission with the image captured without infrared light emission.

[0025] The processing unit may be configured to calculate an enhanced infrared image based on the captured images with and without infrared light emission.

[0026] Each of the left and right cameras may be sensitive to radiation in the invisible wavelength band and radiation in the visible wavelength band, while each of the left and right input optics may be transparent to infrared wavelength bands and transparent to visible wavelength bands. Reduction In this way, radiation in the invisible wavelength band can be visualized in three dimensions.

[0027] The image sensor of the camera sensitive to radiation in the invisible wavelength band can include a sensor die sensitive to both infrared and visible radiation wavelength bands, and the sensor die can be configured to output video signals corresponding to both infrared and visible wavelength band radiation. This allows for a relatively simple and lightweight design of the binocular device. Moreover, image quality can still be high in combination with the reduction of visible light.

[0028] According to another aspect of the present invention, there is provided a method for visualizing visible and invisible radiation, the method comprising: receiving radiation along respective left and right input optical axes for the same field of view by left and right input optics coupled to a support structure and transmitting the light to a left image sensor of a left camera and a right image sensor of a right camera, respectively, the left and right cameras being coupled to the support structure; At least one of the cameras is sensitive to radiation in both the invisible radiation wavelength band and the visible light radiation wavelength band, and the input optics of the camera sensitive to the invisible radiation wavelength band are transparent to the invisible radiation wavelength band and transparent to the visible light radiation wavelength band. Reduction Steps and producing left and right video signals from the detected radiation received by the left and right cameras, respectively; presenting a left video image and a right video image formed in visible light based on a left video signal and a right video signal, respectively, by a left display and a right display, the left display and the right display being coupled to a support structure and viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display.

[0029] Those skilled in the art will appreciate that the features described above may be combined in any manner deemed useful. Furthermore, modifications and variations described with respect to the system may be applied to the method and computer program product as well, and modifications and variations described with respect to the method may be applied to the system and computer program product as well.

[0030] Aspects of the invention will now be described, by way of example, with reference to the drawings, which are schematic and may not be drawn to scale, and in which like items may be designated with the same reference numerals throughout. [Brief description of the drawings]

[0031] [Figure 1]FIG. 1 is a diagram of a system for combining visual and infrared imaging. [Figure 2A] FIG. 2 is a schematic diagram of a camera of a head-mounted device. [Figure 2B] FIG. 1 is a schematic diagram of a camera for a head-mounted device including a pair of orthogonally oriented polarizers. [Figure 3A] FIG. 1 shows a pair of orthogonally oriented polarizing filters. [Figure 3B] FIG. 1 shows a pair of apertures made from orthogonally oriented polarizing filters with an aperture. [Figure 3C] FIG. 13 is a diagram illustrating the effect of an aperture filter. [Figure 4] FIG. 1 illustrates the optical principles of stereoscopic vision using a binocular device. [Diagram 5] FIG. 2 is a diagram showing a first example of specular reflection. [Figure 6] FIG. 11 is a diagram showing a second example of specular reflection. [Figure 7] FIG. 11 is a diagram showing a third example of specular reflection. [Figure 8] A diagram of a combination camera and viewer for one eye. [Figure 9] FIG. 2 illustrates an example timing diagram for a binocular device in operation. [Figure 10] FIG. 2 is a partially machined, open view of an exemplary head-mountable binocular device, viewed from below. [Figure 11] FIG. 13 is a partially machined open view of the same head-mountable binocular device, seen from the side. [Figure 12] FIG. 1 illustrates a head-mountable device attached to a user's head. [Figure 13] FIG. 2 illustrates several wavelength bands. [Figure 14] 1 is a flow chart of a method for visualizing visible and invisible radiation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] Certain exemplary embodiments will now be described in more detail with reference to the accompanying drawings.

[0033] The matters disclosed in the specification, such as detailed structures and elements, are provided to facilitate a comprehensive understanding of the exemplary embodiments. Therefore, it is apparent that the exemplary embodiments can be implemented without the matters specified in those details. Also, well-known operations or structures are not described in detail to obscure the description with unnecessary details.

[0034] FIG. 1 shows an overview of one embodiment of a system for combining visual and infrared imaging. The system is described here in the context of infrared imaging, but a similar arrangement can be made for near infrared or ultraviolet, or any other non-visible radiation wavelength band. The system comprises a head mounted display 101 with a pair of built-in cameras. For example, the system can magnify the photo being viewed, for example using a magnification factor of 1-5. Recording can continue based on the original image while the user views the magnified image. A full HD view with a wide angle of 40 degrees horizontally can be provided. The system can be implemented to be wearable and autonomous. For example, the head mounted display 101 with camera can be provided with additional computing power by a wearable PC, which can be carried by the user and connected to the head mounted display 101 by a wired (or wireless) interface. The wearable PC and the head mounted display 101 with camera can be battery powered and can communicate with external devices via a wireless (or wired) connection 107. This provides the user with freedom of movement. The head mounted display 101 may be lightweight, e.g., less than 250 grams. The system may enable 3d recording, e.g., via direct image streaming and recording, and may enable the surgical team to view the image stream directly on a larger display screen 105. The system may further comprise an external user interface device 104 that may be operated by an assistant in parallel with the user of the head mounted device 101. The user interface device 104 may be wired or wirelessly connected to the wearable PC 102 and / or directly connected to the head mounted display 101. The system may further comprise a remote control by foot pedal 106 or tablet, which may be wired or wirelessly connected to the wearable PC 102 and / or directly connected to the head mounted display 101. The system may further be configured to provide augmented reality photos within the image overlay of pre-operative images on demand.

[0035] For example, the viewpoint follows the movement of the surgeon's head.

[0036] Existing systems are bulky, heavy, not head-mountable, do not align well with the eye, have delayed visualization of processed images, are not autonomous, and have a fixed focal length of more than two meters.

[0037] Most surgical disciplines (open or laparoscopic), such as general surgery, oncology-plastic surgery, urology, gynecology, otorhinolaryngology, thoracic surgery, and neurosurgery, have one thing in common: the need to accurately identify and prevent damage to vital anatomical structures (e.g., nerves, lymphatic tissues, and blood vessels) that need to be preserved and to identify target tissues that need to be removed or treated. This is challenging, especially when considering natural anatomical variations between individuals (the exact location of vital structures varies from one individual to another). Damaging vital structures can lead to severe surgical complications, such as vascular injury, ureteral injury, bile duct injury, and nerve injury. Such complications have a significant impact on patients and the healthcare system. Solutions that can reduce these risks are therefore of great importance to surgeons, their patients, and society as a whole.

[0038] Recognition of important non-visible anatomical structures (tissues) at working distance is desirable and can be provided by the techniques disclosed herein.

[0039] To allow surgery to be performed with optimal spatial perception of depth and body orientation for highly refined micromanipulation, real-time visualization (or, for example, visualization latency of up to 30 ms) may be provided on the head-mounted 3D display 101. Longer delays in visualization are known to cause disorientation and nausea when the head is moving freely if the visual input does not match the body proprioceptive input.

[0040] Non-contact recognition of important non-visible anatomical structures (tissues) across the normal working visual distance range (e.g. 30-50 cm) may be provided. Real-time or very low latency (up to 30 ms) simultaneous visual (VIS) and near infrared (NIR) 3D images may be provided for augmented reality visualization with optimal spatial depth perception. Computer analysis embedded algorithms (e.g. implemented in FPGA) in the wearable computing system 102 or in the head-wearable device 101 may enable (in real-time) recognition of the spectral fingerprints (characteristics) of different tissue types and human organs while generating augmented AR overlays of important tissues from data invisible to the human eye on the actual clinical field of view (FOV) by pixel alignment of the two data streams. For example, a visible light based video stream and a video stream corresponding to a non-visible wavelength range such as a near infrared stream for each eye to visualize in one 3D HD video stream. The system can be built around a compact, lightweight (e.g., less than 250 grams) head-wearable device 101 for optimal ergonomics and extended use (e.g., up to 4-6 hours) without neck strain or eye fatigue. The system can further comprise an autonomous, compact, wearable computer system 102 (e.g., battery powered over the air 107, high speed data and video transmission to an external server) for optimal freedom of movement and minimal lag.

[0041] By comparing simultaneously acquired left and right images, the system may be able to distinguish between shiny spots caused by reflectance on tissue and emission truly originating from tissue. The location of the reflected shiny spots may differ between the left and right images due to differences in the angle of reflection towards the light source. However, the location of the light spots emitted by the tissue itself (e.g., by fluorescent emission) or originating from body contrast may remain at the same position in the focused image. In particular, when weak signals from added fluorescent markers or (even weaker) signals from body autofluorescence are visualized, selective suppression of shiny spots from tissue reflections may be effective. This method of suppression of shiny reflection spots can be combined with other methods, such as cross-polarization between the light source and the camera input.

[0042] By processing locally in the FPGA, the data transfer rate limitations due to longer distances can be avoided. The head-mountable device 101 can recognize spectral signatures in real time, enabling real-time recognition of important non-visible tissues based on computational analysis embedded intrinsic algorithms (FPGA).

[0043] Dedicated image processing embedded software (e.g. implemented in FPGA) can accommodate various display technologies (e.g. OLED, FLCoS and AMLCD) displayed on a single hardware platform and convert industry standard video signals into the video formats and control signals required for the microdisplay. A 3D augmented reality contrast overlay of critical tissues can be provided on the RGB visible video stream so that surgery can be performed with optimal spatial depth perception. Dedicated software can make this possible. A head-mounted autonomous system with ergonomic advantages and freedom of movement can be provided, preferably without wires connecting it to an external source.

[0044] The head-mountable device 101 can be tethered 103 to a wearable computer 102. This can facilitate remote assistance, medical patient data recording and / or education purposes, smooth intelligent zoom capabilities from 1-5-10-20x for optimal 3D accuracy, software adjustable 3D convergence based on working distance, positioning the camera angle 0-20 degrees downwards for optimal ergonomics (minimizing neck herniation), and so that the image can be captured at the macula on the retina of the eye for optimal clarity, as well as 3D recording of procedures for patient files and educational purposes, 2D playback with real-time algorithmic calculations for convergence compensation, 2D or 3D real-time (wireless 107) streaming to nearby screens 104, 105 for improved optimal team work. The system can be used in conducting research on multiple new medical application areas for tissue structure recognition (oncology, dermatology, etc.). The system is applicable to other non-medical application areas such as forensic research. Video or information to the user (remote assistance) can be popped up in the user's field of view as needed (e.g., via voice control, gestures, or foot pedals). Optical brightness can automatically adapt to ambient lighting conditions for ideal viewing conditions, providing ambient through lens light sensing technology. An embedded IMU can register the user's head movements in three axes (e.g., yaw, pitch, and roll) in real time in a virtual (computer-generated) environment. This can enable panoramic 3D stitching of multiple stereoscopic images from multiple viewpoints. Footswitch 106 of the wireless 107 and / or voice / gesture control of various functions can be provided.

[0045] The device can operate separately controllable light sources for the visible (VIS) or UV, or near infrared (NIR) spectral range. Both bundles can be carefully aligned. Thus, dark field compensation is possible with automatic compensation for tracing of NIR light present in ambient light (similar to fluorescent tubes, for example) by switching the near infrared (NIR) light source on and off and modulating the NIR illumination to obtain a continuous NIR image. All this without the need to modulate the visual light source (which can cause unpleasant flickering or even epileptic seizures).

[0046] Illumination of anatomical structures may be performed in at least two spectral ranges, i.e. visible light and the invisible UV or near infrared range. Illumination may be performed under well-defined light conditions with respect to light temperature (K), color rendering index (CRI), angle and beam shape. However, the system may be configured to automatically adapt to different light conditions.

[0047] Two multispectral cameras, each combining a visual range with RGB video and a separate UV or near infrared spectrum visualization, may be provided. A wearable computer 102 with embedded algorithms (FPGA) for data processing, recognizing (real-time) spectral fingerprints of different tissue types and human organs may be provided. The user may choose between various tissue types for visual augmentation. Optimal depth perception may be supported by 3D software real-time correction of parallax and convergence. Optimized data transfer technology (via USB3.2 Gen2x2) for high-speed data exchange with data rates of up to 40GB / s may be provided. A single cable may be implemented to control the two NIR cameras and two microdisplays. Data processing of the image pair stream may be performed externally via a tethered 103 or wirelessly connected processor in the wearable computer 102, which may provide output images to the head-mounted display 101 within less than 30 milliseconds, for example. Enhanced AR overlays of important tissue contrasts that would otherwise be invisible to the human eye can be viewed in one 3D image over the actual clinical field of view (FOV) through pixel alignment of the two data streams. The system is compatible with a variety of display technologies (OLED, FLCoS and AMLCD) and can convert industry standard video signals into the data and control signals required for the microdisplay. The system can be controlled, for example, by foot pedal, voice control or a remote tablet touch screen.

[0048] FIG. 2A shows a schematic diagram of the camera of the head-mounted device 101. To support stereoscopic viewing, two cameras may be implemented, one for each eye. The camera may comprise a housing having a wall 208 that optically separates the optical components from external radiation, except for an aperture 209. The camera may comprise an input optics 210 and an image sensor 206. The input optics 210 may comprise a first lens 201, a notch filter 202, a visible light reduction filter 203, a visible light aperture 204, and a second lens 205, arranged in sequence from the aperture 209 towards the image sensor 206. The order of these components may vary according to various implementations, and the number and configuration of the lenses 201, 205 may also vary. One or more of the lenses 201, 205 may be movable for the purpose of focusing the image sensed by the image sensor 206. The image sensor 206 may comprise a silicon-mounted sensor having a coating to render the image sensor 206 sensitive to both at least one wavelength of visible radiation, such as the red, green, and blue wavelength ranges, and at least one wavelength range of non-visible radiation, such as the near-infrared, infrared, and / or ultraviolet wavelength ranges.

[0049] The optional notch filter 202 can at least partially suppress excitation of certain undesirable wavelength ranges.

[0050] The visible light reducing filter 203 may be adapted to suppress or reduce most of the visible light wavelengths. Additionally, the visible light reducing filter 203 may be adapted to pass or transmit light in the non-visible wavelength range to which the image sensor 206 is sensitive. For example, the visible light reducing filter 203 may be made of a dichroic material that transmits most of the light in the relevant non-visible wavelength range while reflecting or absorbing most of the light in the visible light wavelength range. The visible light reducing filter 203 is adapted to pass a portion of the visible light.

[0051] Examples of suitable materials for the visible light reduction filter 203 and / or the visible light stop 204 include H-sheet Polaroid®, which is a polyvinyl alcohol (PVA) polymer impregnated with iodine, and K-sheet Polaroid®, which contains aligned polyvinylidene chains in a PVA polymer created by dehydrating the PVA. Another exemplary material is a coatable polymer polarizer formed of a composition containing a rigid rod-like polymer capable of forming a liquid crystal phase in a solvent, which can form an achromatic polarizer, as disclosed in US Patent Application Publication No. 2016 / 0266292. Yet another exemplary material is a polarizer including a stretched stack, which is a stacked pair including a substrate layer and a hydrophilic polymer layer and is stretched, and at least a dichroic material is adsorbed to the hydrophilic polymer layer, as disclosed in US Patent Application Publication No. 8,559,105. Another example is based on materials commonly used in dielectric beam splitters. Because the wavelengths at which the radiation is split can be specified, it is possible to fabricate a dielectric coating that is transparent to invisible radiation wavelength bands, such as NIR light, and reflective to visible radiation wavelength bands. In this regard, it may be useful to fix a visible light filter 203 or visible light stop 204, consisting of a dielectric beam splitter, at an angle (e.g., at 45 degrees) to the optical axis of the camera (not shown).

[0052] The visible light diaphragm 204 is a diaphragm that can have a fixed or variable aperture. The diaphragm is made of a material that reduces (or completely blocks) light in the visible wavelength range while being transparent to relevant non-visible wavelength ranges such as near-infrared radiation. For example, the visible light diaphragm 204 can be made of a dichroic material that reflects or absorbs most of the light in the visible wavelength range while transmitting most of the light in the relevant non-visible wavelength range. For example, the visible light diaphragm can be made of the same material as the visible light reduction filter 203 or of a different material. The visible light diaphragm 204 can be completely non-transparent to light in the visible wavelength range (except for light that passes through its aperture). Alternatively, the material of the visible light diaphragm 203 can pass a (relatively small) portion of visible light.

[0053] Lens 205 cooperates with lens 201 to create a flux of incident light from aperture 209 onto image sensor 206. The two lenses may be movable relative to each other to provide a focusing function. The lenses may be transparent to both visible light and relevant non-visible wavelength ranges.

[0054] It will be noted that, while not limiting, FIG. 2A illustrates both a visible light reducing filter 203 and a visible light stop 204. In certain embodiments, any one of these components may be provided. In either case, the input optics 210 is transparent to the relevant non-visible wavelength band and transparent to the visible wavelength band. Reduction As mentioned above, the input optics do not completely block visible light, but merely reduce it to a level suitable for the image sensor 206. Furthermore, the input optics need not pass 100% of the light in the relevant non-visible wavelength range. For example, due to limitations of available materials, the input optics may also reduce some light in the relevant non-visible wavelength range.

[0055] The input optics 210 receives radiation in the visible wavelength range. Reduction For example, the radiation intensity in the visible light wavelength band may be reduced by at least 75%, preferably at least 90%, more preferably at least 95%, and even more preferably at least 98%. The input optics is transparent to radiation in the selected invisible wavelength band. For example, the input optics may transmit at least 80%, preferably at least 90%, and more preferably at least 95% of the intensity of the received radiation in the invisible radiation wavelength band. For example, the reduction in visible light is at least 75% and the transmission of invisible radiation is at least 80%. For example, the reduction in visible light is at least 95% and the transmission of invisible radiation is at least 90%.

[0056] FIG. 2B shows the camera of FIG. 2A in which the visible light reduction filter 203 is implemented as a pair of orthogonally oriented polarizing filters 203a and 203b. Additionally, the visible light aperture 204 is implemented as a pair of apertures 204a, 204b made of orthogonally oriented polarizing filter materials. For example, such materials include iodine-containing polymers. Such pairs of orthogonally oriented polarizing filters are known to transmit about 90%-95% of radiation in the near-infrared wavelength range while spectrally linearly removing about 98%-99% of visible light. It is noted that such iodine-containing polymers do not completely remove visible light. This is in keeping with the purpose of the filters and apertures to remove much, but not all, visible light.

[0057] FIG. 3A shows a pair of orthogonally oriented polarizing filters 203a and 203b in the see-through direction. The diagonal lines indicate the polarization directions. It is noted that the polarization direction of the first filter 203a is orthogonal to that of the second filter 203b. It is observed that in certain applications it may be sufficient to provide only one polarizing filter instead of two. Furthermore, when two polarizing filters are provided, the filters 203a, 203b can be rotated relative to each other, e.g. under the control of a control software, to provide a variable amount of visible light reduction.

[0058] FIG. 3B shows a pair of diaphragms 204a, 204b made of orthogonally oriented polarizing filters with an aperture 207. The diagonal lines indicate the polarization directions. It is noted that the polarization direction of the first aperture filter 204a is orthogonal to that of the second aperture filter 204b. It is observed that in certain applications it may be sufficient to provide only one polarizing filter based diaphragm instead of two. Furthermore, when two polarizing filter based diaphragms are provided, the aperture filters 204a, 204b can be rotated relative to each other, e.g. under the control of a control software, to provide a variable amount of visible light reduction.

[0059] Alternatively, certain implementations may include one polarizing filter 203a and one polarizing aperture filter 204a, and omit one or both of the second filter 203b and the second aperture filter 204b. The polarization direction of the one filter 203a may be orthogonal (or may have any other desired orientation, or may be variable) to the one aperture 204a.

[0060] FIG. 3C illustrates the effect of the aperture filter 204. The graph shown in FIG. 3C has wavelength on the vertical axis and distance from the camera on the horizontal axis (arbitrary scale). Depth of field is the distance between the nearest and furthest objects that are in acceptably sharp focus in the image, as seen by the camera. Because visible light can only pass through the aperture 207 of the aperture filter 204, visible light has a relatively large depth of field, as indicated by the large space between the horizontal arrows representing visible wavelengths (VIS). However, the non-visible light wavelength range that can pass through the material of the aperture filter 204 has a relatively small depth of field, as indicated by the smaller space between the horizontal arrows representing near-infrared wavelengths (NIR). Because non-visible light can have a much lower intensity than visible light, it can be effective to retain all non-visible light while reducing the amount of visible light through the aperture filter 204. Furthermore, due to dispersion, lenses focus differently for different wavelengths. To produce an image that is well-focused at all detected wavelengths typically requires an expensive and heavy lens system. The aperture filter of the present invention creates a large depth of field for the visible light wavelength range while creating a narrow depth of field for the non-visible light wavelength range. This principle can be exploited by optimizing the focus of the lens for the invisible light with a narrow depth of field. In this way, the invisible light image is clear and in focus. Since the visible light has a large depth of field, the visible light image is also clear and in focus. Thus, this system allows for a lightweight, high quality imaging device with relatively simple lenses and filters.

[0061] FIG. 4 illustrates the optical principles of stereoscopic vision. It shows a binocular device 400. It is observed that the diagram is simplified to illustrate the concepts disclosed herein. In many practical embodiments where the focus may be further apart, for example at a working distance of about 20 centimeters to 1 meter, the line of sight of the left eye 413 may be approximately parallel to the line of sight of the right eye 423. Thus, the left optical system 410 may be implemented at a very small angle or parallel to the right optical system 420.

[0062] The binocular device 400 comprises a support structure 404 to which the components of the binocular device 400 are attached. In certain embodiments, the entire device may be enclosed within a housing. Alternatively, the left image components may be enclosed within a first housing and the right image components may be enclosed within a second housing, with both housings attached to the support structure 404. Either way, the left optics 410 and the right optics 420 are fixed relative to one another. In certain embodiments, the left optics 410 and the right optics 420 may be movable relative to one another, for example to customize the device 400 for a particular user.

[0063] The binocular device 400 may include a left optical system 410. The left optical system 410 may include a left camera 411 and a left display 412. The right optical system 420 may include a right camera 421 and a right display 422. The left camera 411 and the right camera 421 may include a visible light reducing filter 203 and / or a visible light aperture 204, as described above. However, certain embodiments may omit such filters or apertures in one or both of the cameras.

[0064] The binocular device 400 may further comprise a light source 403. The light source 403 may be any device capable of emitting light in a desired wavelength range. Examples of suitable light sources include light emitting diodes (LEDs), incandescent lights, halogen lights, or another light source. The light source 403 may comprise two or more light sources that generate light of different spectra (e.g., a light source that generates white visible light and a light source that generates light primarily in an invisible wavelength range, such as the near-infrared wavelength range) and an optical system that combines the light generated by these light sources into a bundle. Alternatively, the light source may comprise a single emitter that emits light in both visible and invisible wavelengths. Further alternatively, the light source 403 may emit only light in a near-invisible wavelength range. For example, in certain applications, it may be envisioned that visible light is abundantly available due to ambient light generated by operating room lights or sunlight, and as a result, there is no need to generate additional visible light by the binocular device 400.

[0065] As shown, during operation, light source 403 generates light. For example, light beam 405 can impinge on object 401, such as tissue to be examined.

[0066] This is indicated by arrow 405. The tangent plane of the tissue at the point of incidence 408 of ray 405 with the tissue is indicated by dotted line 402. For example, in the case of diffuse reflection or fluorescence emission, light travels from the point of incidence 408 to both the left camera 411, as indicated by arrow 406, and to the right camera 421, as indicated by arrow 407.

[0067] 5 illustrates the binocular device 400 for specular reflection by tissue 501 at an incidence point 508 with a tangent plane 502. Light ray 505 emitted by light source 403 is primarily reflected towards the right camera 421 along ray 507 in this example. Little or no light is reflected to the left camera 411, as shown by dotted line 506.

[0068] 6 shows that, of course, there is not just one light ray, but a light beam 605 that is reflected off beam 607 into right camera 421. The geometric beam shape generated by light source 403 for the visible wavelength range may be as identical as possible to the geometric beam shape generated by light source 403 for the non-visible wavelength range.

[0069] FIG. 7 shows another example of specular reflection, where a portion 707 of a light beam 705 emitted by a light source 403 is reflected by tissue 701 towards a right camera 421, and another portion 706 of the light beam 705 is reflected towards a left camera 411.

[0070] It is undesirable that the specular reflections are misinterpreted by the user as actual diffuse reflection contrast or fluorescent emission originating from the tissue. If both cameras experience off-axis specular reflections, the output image may be misinterpreted as, for example, NIR-induced fluorescent emission from two separate spots on the tissue surface (X and Y in FIG. 7). The VIS light source may be kept on continuously (to avoid visible flicker). A dichroic iris and / or filter may suppress specular reflections at visible wavelengths, which are much stronger than those in the NIR. The NIR source may be modulated ON / OFF without causing visible flicker. Images may be captured by cameras 411 and 421 while the NIR light source is on and while the NIR light source is off. The resulting pairs of images (one pair (left / right) with the NIR light source on and one pair with the NIR light source off) may be processed to improve NIR visualization. For example, an image obtained with the NIR light source off may be subtracted from an image obtained with the NIR light source on. In this way, an image with enhanced NIR visualization is produced. The NIR image can be converted to visible color and blended with an image obtained with the NIR light source turned off, creating a combined visualization of visible and NIR light. This combined visualization can be displayed on the left display 412 and right display 422. However, it is also possible to display a visible light only image on the left display 412 and right display 422 without visualizing the NIR light.

[0071] As a special imaging mode, the visualization of the NIR light in the left and right displays 412, 422 is alternated. That is, the left NIR image is shown in the left display 412 for a certain period of time, after which the display of the left NIR image is stopped and the right NIR image is displayed in the right display 422 for that certain period of time. The time period can be, for example, long enough for an average human to perceive flicker. Longer periods are also possible. This visualization mode makes it possible to distinguish specular reflection from diffuse reflection and fluorescent emission. In the case of spots caused by diffuse reflection or fluorescent emission that are truly originating from tissue, the spots appear in the same position in both the left and right visualization, as shown in FIG. 4. However, in the case of spots caused by specular reflection, the spots appear in only one eye, as shown in FIG. 5 and FIG. 6, or in different positions in the left and right visualization, as shown in FIG. 7. Thus, if the spot does not occur in the same position in both eyes (flickers and / or "wobbles"), the user knows there is a specular reflection and moves their head, thereby moving the camera of the head-wearable device, to remove the specular reflection from view.

[0072] Another feature of specular reflection versus true tissue contrast or fluorescence is that specular reflections move their position with the observer's head orientation, whereas true contrast and / or fluorescence remain in the same location on the tissue.

[0073] The visual effect of specular reflection shown in Figures 4-7 can be used to detect specular reflection in images captured by automated image processing. Specular reflection may be undesirable as it may obscure the actual signal, especially in the non-visible wavelength range (e.g., near infrared range), and it is observed that the important signals are diffuse reflection and / or fluorescent emission. The specular reflection of the light emitted by the light source 403 makes the diffuse reflection and / or fluorescent emission invisible. After the left and right images are captured by the left camera 411 and the right camera 421 at substantially the same time with the NIR light source turned on, a specular reflection detection process may be performed. For example, a comparison of the simultaneously captured left and right images may be performed, particularly to detect bright spots. For example, a bright spot may be detected if the intensity at a particular location in the image is above a certain threshold. Furthermore, if such a bright spot is detected in a first one of the left and right images, it may be determined whether a bright spot exists at a corresponding location in the other one of the left and right images. For example, it may be determined whether a bright spot exists at a corresponding location by comparing the image intensity at the corresponding location with a predefined threshold. "Corresponding position" may mean within a certain distance from the same position as the bright spot in the first image. This distance may be determined taking into account that the corresponding positions of visualized items may differ slightly in the left and right images depending on the parallax of objects in the stereo image pair.

[0074] Alternatively, the disparity can be estimated using known algorithms and the corresponding location in the other of the left and right images can be determined based on the estimated disparity.

[0075] For example, if a bright spot is detected at a corresponding position in the left and right images according to the estimated disparity, it can be determined that this is not a specular reflection but a relatively strong diffuse reflection, as shown in FIG. 4.

[0076] For example, if a bright spot is detected in only one of the left and right images and not in the corresponding position in the other of the left and right images, this can be determined to be a specular reflection, as shown in Figures 5 and 6.

[0077] For example, if a bright spot is detected at a different position that is just offset from two corresponding positions in the left and right images according to the estimated disparity, the bright spot can be determined to be a specular reflection in both images, as shown in FIG. 7.

[0078] Preferably, specular reflection detection is performed on a non-visible channel (e.g., NIR channel) since specular reflection within the channel cannot be reduced by a visible light filter and / or a visible light aperture and may be essential to properly visualize low intensity features in non-visible wavelength bands. Alternatively, specular reflection detection may be performed separately for each color channel (red, green, blue, non-visible light). Further alternatively, the intensities of the channels may be combined to simultaneously detect reflections of all detected wavelength ranges.

[0079] When a specular reflection is detected, an alarm signal may be generated to indicate that a specular reflection has been detected. The alarm signal may include, for example, an audio signal or a visual indication. The visual indication may be shown on the display 412, 422. For example, a specular reflection detected in a non-visible wavelength range may be displayed in a different color than the rest of the non-visible wavelength image overlay. For example, non-visible wavelengths may be typically shown as a green overlay on top of a color image in the visible wavelength range. However, if a spot is discernible as a specular reflection, it may be shown as a darker green as the rest of the green overlay. This may allow the user to move the viewing position slightly to a position where there is no specular reflection.

[0080] Alternatively, the detected specular reflections can be removed by image processing. For example, ReductionBy multiplying by the factor, the image intensity at the bright spot can be locally reduced, so that it is brought to the same average level as the average image intensity around the bright spot.

[0081] In certain embodiments, neither camera is sensitive to radiation in the invisible radiation wavelength band. For example, both cameras can be sensitive to radiation in the visible radiation wavelength band (e.g., in the case of a color camera, the red, green, and blue wavelength ranges). The visible light reducing filter 203 and the visible light diaphragm 204 can be omitted as disclosed above.

[0082] The binocular device may optionally include a processing unit configured to receive signals representing a left image from the left camera and a right image from the right camera, captured approximately simultaneously by the left camera and the right camera, respectively, to compare the left and right images, and to detect specular reflection based on a result of the comparison.

[0083] In an optional specific visualization mode, the binocular device can alternately show a left image based on a left video signal generated by a left camera on a left display and a right image based on a right video signal generated by a right camera on a right display to allow the user to identify specular reflections. Specular reflections can be easily distinguished by comparing the left and right images, since specular reflections swing back and forth when the images are compared, whereas fluorescent and diffuse reflections remain static for both cameras. This visualization mode allows specular reflections to be distinguished from diffuse and / or fluorescent reflections.

[0084] The device can detect, suppress, or inform the user of specular reflections in various portions of the light spectrum.

[0085] FIG. 8 shows a diagram of a camera and viewer combination 800 for one eye (left or right optics). In use, the device 800 may be fixed between the observer's eye 810 and the object being observed, e.g., tissue 801. In a typical use, the device 800 may be held or mounted near the eye 810 at a working distance from the object being observed 801. The device 800 may comprise input optics including a notch filter 802, a dichroic aperture that may be implemented as shown as a pair of polarizing filters 803, 804, and an optics set 805 that may comprise one or more lenses and other optical elements. The order of these components may vary depending on the implementation. The device 800 further comprises an image sensor 806, e.g., a CMOS chip or another type of camera chip. For example, the chip may have a surface that converts radiation projected in both visible and invisible radiation wavelength bands into electronic signals. The image sensor 806 is electrically connected or connectable to electronics 807 for processing the image signals generated by the image sensor 806. This electronics 807 may be integrated into the binocular device 101 or may be implemented in an external processing device, such as a wearable computer 102, that has sufficient processing power. This helps to keep the head-mountable display lightweight.

[0086] The device 800 further comprises a microdisplay 808 for displaying a processed image based on the image signal output by the electronics 807 and transmitted to the microdisplay 808. The microdisplay may have a size comparable to an eye, for example. The size of the microdisplay may be arbitrary, since the device 800 further comprises output optics including an ocular display set 809 for projecting the image output by the microdisplay 808 onto the retina of the eye 810. The input optics 802, 803, 804, 805 and the image sensor 806 may be optically separated from the microdisplay 808 and the output optics 809, for example by arranging them in two separate compartments having walls that are opaque to radiation of the wavelengths of interest.

[0087] In the illustrated embodiment, the input optics, camera, microdisplay and output optics are aligned, i.e. share the same central optical axis 811, 812. In this way, the user has the impression of looking straight ahead, for example through a pair of binoculars. It is therefore easy for the user to orient himself and his hand relative to the image generated by the microdisplay 808. In alternative embodiments, there may be an inclination between the central axis 811 of the input optics and camera 806 on the one hand and the central axis 812 of the microdisplay and output optics on the other hand. For example, the input axis 811 may be inclined slightly downwards with respect to the output axis 812.

[0088] 9 shows an example of a timing diagram of the binocular device 400 in operation. The timing graph shows time on the horizontal axis and the performance of a particular activity on the vertical axis (higher positions mean that an action has been performed). As shown in graph 902, it is observed that the camera can be configured to capture images at a particular frame rate.

[0089] The light source 403 can have separately controllable visible and non-visible light generation capabilities. Alternatively, the light source can only generate non-visible light. In that case, visible light can be provided from elsewhere. However, for reasons of consistency between the images recorded, it may be preferable to have a single light source that can generate both visible and non-visible light in a single light beam. Furthermore, to prevent visible flicker, the visible light source can continue to emit radiation continuously while the non-visible light is alternately switched on and off. This prevents flicker not only in the camera image, but also in others in the room who do not have the binocular device 400.

[0090] The non-visible light may be generated by a light source in a strobe manner. This is shown in graphs 901 and 902. The light source for the non-visible light, such as NIR light, may be configured to flash slower than the frame rate, for example at half the frame rate, so that the non-visible light source is turned off during the capture of each first frame and turned on during the capture of each second frame. One image may be taken with the non-visible light source switched off. The next image may be taken with the non-visible light source switched on. Thus, as shown at 905, two successive images generate a result of imaging without non-visible light and a result of imaging with non-visible light. After such pairs of images are captured from each camera, the processing electronics 807 may calculate an output image based on the captured pair of input images, as shown at 906 in graph 903. As soon as the processing of block 906 is completed, the output image may be displayed by the microdisplay 808, as shown at 907 in graph 904. It will be understood that this is only an exemplary timing diagram. Other timings and other sequences of steps may alternatively be implemented.

[0091] For example, in processing step 906, the processing electronics 807 can subtract an image with the non-visible light source switched off from an image with the non-visible light source switched on. In this way, visible light is subtracted and non-visible light is highlighted in the subtraction image. Thus, if a pixel in the image with the non-visible light source switched off has a value X and the same pixel in the image with the non-visible light source switched on has a value Y, the same pixel in the subtraction image has a value YX. Pixels of the subtraction image that are greater than a predefined threshold can be blended with a predefined visible color on top of the image captured with the non-visible light source switched off.

[0092] Additionally, a speckle detection mode may be provided that allows the user to detect speckles by visual effects. For example, specular reflections may be indicated by an alarm or visual indication. Alternatively, in a speckle detection mode as shown by graphs 910 and 911, an overlay visualization of the non-visible light image is alternately shown only on the left display 412 during a first time interval 911, 912 and only on the right display 422 during a second time interval 913, 914. In such a case, the viewer can assess whether there is a specular reflection in the non-visible region by considering whether there is a spot that appears to oscillate between two positions. The time intervals 911, 912 and 913, 914 may be longer or shorter as desired, for example using a time interval at least as long or longer than the time interval in which the two images are captured (at 905) to ensure that the flicker is visible to the observer.

[0093] FIG. 10 shows a partially processed open view of an exemplary head-mountable binocular device, viewed from below. FIG. 11 shows a partially processed open view of the same head-mountable binocular device, viewed from the side. In this example, the optical center axis of the left viewing portion is approximately parallel to the optical center axis of the right viewing portion. Furthermore, the optical axis of the left camera portion and the optical axis of the right camera portion are slightly tilted downward toward each other. However, this alignment of the optical axes is not a limitation.

[0094] 10 and 11 show a camera portion including an optional notch filter 1001, a visible light aperture 1003, a camera lens 1005, and a camera sensor, or image sensor 1006. Also shown in FIGS. 10 and 11 is a display portion including a microdisplay 1008, output optics 1009, and an eyepiece 1011, the eyepiece 1011 comprising an annular structure suitable for holding near the eye 1010. Also shown in FIGS. 10 and 11 is a light source 1012. These items are described in more detail above, so their characteristics will not be repeated here.

[0095] 12 shows how a head-mountable device can be attached to a user's head by at least one strip 1201 connected to a support structure 1204, which can fit around the head. The eyepiece can be aligned with the user's eye as shown.

[0096] FIG. 13 shows several wavelength bands of visible and invisible light (ultraviolet and near infrared) in nanometers (nm). It is noted that blue may be centered at 445 nanometer wavelength, green may be centered at 535 nanometer wavelength, and red may be centered at 575 nanometer wavelength. Ultraviolet may be considered to be radiation having a wavelength less than about 380 nanometers. Near infrared may be considered to be radiation in the wavelength range from about 740 nanometers to about 1000 nanometers. Infrared is even greater than 1000 nanometers. It is noted that these wavelengths are provided purely as illustrative examples. The devices described herein may be designed for different wavelengths for detection, processing, and visualization.

[0097] 14 shows a flow chart of a method for visualizing visible and invisible radiation, the method includes, in step 1401, receiving radiation for the same field of view along respective left and right input optical axes by left and right input optics coupled to a support structure and transmitting the light to a left image sensor of a left camera and a right image sensor of a right camera, respectively, the left and right cameras being coupled to the support structure, at least one of the cameras being sensitive to both radiation in the invisible radiation wavelength band and radiation in the visible light radiation wavelength band, the input optics of the camera sensitive to the invisible radiation wavelength band being transparent to the invisible radiation wavelength band and transparent to the visible light radiation wavelength band. ReductionThe method further includes, at step 1402, producing left and right video signals from the detected radiation received by the left and right cameras, respectively. The method further includes, at step 1403, presenting left and right video images formed in visible light based on the left and right video signals, respectively, by a left display and a right display, the left display and the right display being coupled to the support structure and viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display.

[0098] It may be observed that features for specular reflection reduction may be implemented in a binocular device even in the absence of visible light reduction, i.e., in a device or method that has image processing capabilities to detect specular reflection, both visible light filter 202 and visible light aperture 204 may be omitted.

[0099] Certain embodiments comprise a binocular device, either a head-mountable device or designed to be held directly in front of the eyes, which has at least one camera and / or light source to observe objects from a working distance in an open space (e.g., indoor or outdoor environment). In contrast, other applications, such as endoscopy, can operate in a mainly dark cavity where the lighting can be freely controlled. This open space imposes several constraints on the lighting. First, there is the presence of ambient light caused by external light sources. Second, it may not be possible to optimize the lighting conditions purely for the binocular device's cameras, since other people and / or other camera equipment should preferably not be disturbed by the lighting caused by the binocular device. The techniques disclosed herein may help improve the usefulness of the binocular device under these circumstances. For example, the binocular device's cameras can be equipped with high dynamic range, high-quality optics, special image processing techniques, and / or polarizing filters, dichroic filters, polarizing diaphragms, and / or dichroic diaphragms, as described herein. For this reason, the light source of the binocular device may be configured to keep the emission of light in the visible wavelength band as constant as possible so as not to disturb bystanders. As explained above, the emitted light in the invisible wavelength band may be flashed in a strobe fashion, so that the visible light image can be combined with the invisible light image. Because it flashes within the invisible wavelength band, this is not disturbing to bystanders.

[0100] Although some techniques are disclosed above with respect to head-mounted binocular devices, this is not a limitation, and it is observed that the image processing techniques and / or features of polarizing filters, dichroic filters, polarizing diaphragms, and / or dichroic diaphragms may also be applied to cameras in general.

[0101] According to another aspect, a camera includes input optics and an image sensor configured to produce a video signal from detected optical radiation received from the input optics relative to a field of view along an input optical axis, at least one of the cameras being sensitive to both radiation in the invisible radiation wavelength band and radiation in the visible light radiation wavelength band, the input optics being transparent to the invisible wavelength band and transparent to the visible light wavelength band. Reduction Such cameras can be designed for many different applications, such as for example endoscopy.

[0102] Optionally, the display is configured to present a video image formed in visible light by the display based on the video signal.

[0103] Optionally, the display is positioned to be viewed by a user through an eyepiece operatively connected to the display.

[0104] The input optics of the camera can include a polarizing filter that includes at least one layer of polarizing material.

[0105] A polarizing filter may comprise at least two layers of polarizing material having mutually orthogonal polarization directions.

[0106] The device can include a light source coupled to the camera by a support structure, the light source capable of generating radiation within at least invisible and visible wavelength bands, the light source configured to generate an emission beam of visible light and an emission beam of invisible radiation aligned to be substantially identical in geometry and position.

[0107] The device can include a light source coupled to the support structure capable of generating radiation in at least an invisible wavelength band and a visible wavelength band, the light source further including a polarizing filter configured to polarize visible light in the visible wavelength band output by the light source and transmit radiation in the invisible wavelength band, the polarization direction of the polarizing filter of the light source being substantially orthogonal to the polarization direction of the polarizing filter of the input optics.

[0108] The input optics may include a stop having an aperture, the stop around the aperture being adapted to receive light in the visible wavelength range. Reduction The aperture is transparent to light in the infrared wavelength range, while the aperture is transparent to light in the infrared wavelength range.

[0109] The input optics can include a lens with an autofocus configured to focus radiation in the invisible wavelength band.

[0110] The input optical system of the camera is designed to detect light in the visible wavelength range in addition to the aperture. Reduction The filter may be of a different type.

[0111] The input optics of the camera may include a filter containing iodine to selectively reduce radiation in the visible wavelength band.

[0112] The device can include a light source coupled to the camera by a support structure for generating at least invisible light and visible light, the light source configured to intermittently emit the invisible light while keeping the visible light intensity substantially constant, and the camera configured to capture at least one image with the emission of the invisible light and at least one image without the emission of the invisible light.

[0113] The device may comprise a processing unit configured to calculate an enhanced non-visible light image based on the captured image with the non-visible light emission and the captured image without the non-visible light emission.

[0114] The image sensor can include a sensor die sensitive to both infrared and visible radiation wavelength bands, the sensor die configured to output video signals corresponding to both infrared and visible radiation wavelength band radiation.

[0115] Some or all aspects of the present invention may be suitable for implementation in software, in particular in the form of a computer program product. The computer program product may include a computer program stored on a non-transitory computer-readable medium. The computer program may also be represented by a signal, such as an optical or electromagnetic signal, carried by a transmission medium, such as an optical fiber cable or air. The computer program may have, in part or in whole, the form of a source code, an object code, or a pseudocode suitable for being executed by a computer system. For example, the code may be executable by one or more processors.

[0116] The examples and embodiments described herein serve to illustrate, rather than limit, the present invention. Those skilled in the art will be able to design alternative embodiments without departing from the spirit and scope of the present disclosure, as defined by the appended claims and their equivalents. Reference signs placed within parentheses in the claims should not be construed as limiting the scope of the claims. Items described in the claims or specification as separate entities may be implemented as a single hardware or software item combining the features of the described items.

[0117] Particular aspects are defined in the following clauses.

[0118] Clause 1. A binocular device for visualizing visible and invisible radiation, comprising: A support structure; a left camera and a right camera coupled to a support structure, the left camera comprising left optics and a left image sensor, and the right camera comprising right optics and a right image sensor; the left and right image sensors are configured to produce left and right video signals from optical radiation received and detected from the corresponding left and right input optics along corresponding left and right input optical axes and for the same field of view; At least one of the cameras is sensitive to radiation in both the invisible radiation wavelength band and the visible light radiation wavelength band, and the input optics of at least one of the cameras is transparent to the invisible wavelength band and transparent to the visible light wavelength band. Reduction A binocular device.

[0119] Clause 2. The binocular device of clause 1, further comprising a left display and a right display coupled to a support structure, the left display and the right display positioned to be viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display, and configured to present left and right video images formed in visible light by the left and right displays based on left and right video signals, respectively.

[0120] Clause 3. A binocular device as described in clause 1 or clause 2, wherein at least the input optics of the camera sensitive to radiation in the invisible wavelength band is provided with a polarizing filter comprising at least one layer of polarizing material.

[0121] Clause 4. A binocular device as described in clause 3, wherein the polarizing filter comprises at least two layers of polarizing material having polarization directions that are orthogonal to each other.

[0122] Clause 5. A binocular device as described in any of clauses 1 to 4, further comprising a light source coupled to the support structure capable of generating radiation within at least invisible wavelength bands and visible light wavelength bands, the light source being configured to generate an emission beam of visible light and an emission beam of invisible radiation aligned to be approximately identical in geometric shape and position.

[0123] Clause 6. A light source coupled to a support structure capable of generating radiation in at least an invisible wavelength band and a visible light wavelength band, the light source further comprising a polarizing filter configured to polarize visible light in the visible light wavelength band output by the light source and transmit radiation in the invisible wavelength band; 4. A binocular device as described in clause 3, wherein the polarization direction of the polarizing filter of the light source is approximately perpendicular to the polarization direction of the polarizing filter of the input optical system.

[0124] Clause 7. An input optical system corresponding to a camera sensitive to invisible radiation includes a diaphragm having an aperture, the diaphragm surrounding the aperture being sensitive to light in the visible light wavelength range. Reduction 7. A binocular device according to any one of clauses 1 to 6, wherein the aperture is transparent to light in the invisible wavelength band, while the diaphragm is transparent to light in the invisible wavelength band.

[0125] Clause 8. A binocular device as described in clause 7, wherein the input optics comprises a lens having an autofocus, the autofocus being configured to focus radiation in the invisible wavelength band.

[0126] Article 9. The input optical system of a camera sensitive to radiation in the invisible wavelength band is, in addition to the aperture, sensitive to light in the visible wavelength band. Reduction 8. The binocular device of claim 7, further comprising a filter that is transparent.

[0127] Clause 10. A binocular device as described in any one of clauses 1 to 9, wherein the input optics of the camera sensitive to radiation in the invisible wavelength band is provided with a filter comprising iodine for selectively reducing radiation in the visible wavelength band.

[0128] Clause 11. The processing unit further comprises: receiving signals representative of a left image from the left camera and a right image from the right camera, the left image and the right image being captured by the left camera and the right camera, respectively, substantially simultaneously; To compare the left and right images, and A binocular device as described in any of clauses 1 to 10, configured to detect specular reflection based on the result of the comparison.

[0129] Clause 12. A binocular device as described in any of clauses 1 to 11, further comprising a light source coupled to the support structure for generating at least radiation in the invisible wavelength band and visible light, the light source configured to intermittently emit the invisible light while keeping the visible light intensity substantially constant, and the camera configured to capture at least one image with the emitted invisible light and at least one image without the emitted invisible light.

[0130] Clause 13. A binocular device as described in clause 12, further comprising a processing unit configured to calculate an enhanced image of radiation in the invisible wavelength band based on an image captured with and an image captured without the emission of invisible light.

[0131] Clause 14. The left and right cameras are each sensitive to radiation in the invisible wavelength band and radiation in the visible wavelength band, and the left and right input optics are each transparent to the invisible wavelength band and transparent to the visible wavelength band. Reduction 14. The binocular device according to any one of clauses 1 to 13, wherein the binocular device is optically transparent.

[0132] Clause 15. A binocular device as described in any of clauses 1 to 14, wherein the image sensor of the camera sensitive to radiation in the invisible wavelength band comprises a sensor die sensitive to both the invisible radiation wavelength band and the visible radiation wavelength band, and the sensor die is configured to output video signals corresponding to both radiation in the invisible wavelength band and radiation in the visible wavelength band.

[0133] Clause 16. A binocular device according to any one of clauses 1 to 15, wherein the invisible radiation wavelength band is a near-infrared radiation wavelength band.

[0134] Clause 17. A method for visualizing visible and invisible radiation, comprising: receiving radiation along respective left and right input optical axes for the same field of view by left and right input optics coupled to a support structure and transmitting the light to a left image sensor of a left camera and a right image sensor of a right camera, respectively, the left and right cameras being coupled to the support structure; At least one of the cameras is sensitive to radiation in both the invisible radiation wavelength band and the visible light radiation wavelength band, and the input optics of the camera sensitive to the invisible radiation wavelength band are transparent to the invisible radiation wavelength band and transparent to the visible light radiation wavelength band. Reduction The method is A method for visualizing visible and invisible radiation comprising the steps of producing left and right video signals from radiation received and detected by left and right cameras, respectively.

[0135] Clause 18. The method of clause 17, further comprising the step of presenting by a left display and a right display a left video image formed in visible light based on the left video signal and the right video signal, respectively, the left display and the right display being coupled to a support structure and viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display.

Claims

1. 1. A binocular device for visualizing optical radiation, comprising: A support structure; a left camera and a right camera coupled to the support structure, the left camera comprising a left optical system and a left image sensor, and the right camera comprising a right optical system and a right image sensor, the left image sensor and the right image sensor configured to produce a left video signal and a right video signal from detected optical radiation received from corresponding left input optical systems and right input optical systems for a same field of view along corresponding left input optical axes and right input optical axes; The device further comprises a processing unit, the processing unit comprising: receiving signals representative of a left image from the left camera and a right image from the right camera simultaneously captured by the left camera and the right camera, respectively; comparing the left image with the right image; and A binocular device configured to detect whether a specular reflection is present based on a result of the comparison.

2. 2. The binocular device of claim 1, further comprising a left display and a right display coupled to the support structure, the left display and the right display positioned to be viewed by a pair of eyes of a user through a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display, and configured to present left and right video images formed in visible light by the left and right displays based on the left and right video signals, respectively.

3. 3. A binocular device according to claim 1 or 2, wherein at least one of the cameras is sensitive to optical radiation in both wavelength bands of invisible light radiation and visible light radiation.

4. A binocular device as described in claim 3, wherein the input optical system of at least one of the cameras is provided with a visible light radiation reducing filter that is transparent to a wavelength band of the invisible light radiation and is reducing to the wavelength band of the visible light radiation so as to pass only a portion of the visible light radiation.

5. 5. The binocular device of claim 4, wherein at least the input optics of the camera sensitive to light radiation in the wavelength band of invisible light radiation comprises a polarizing filter comprising at least one layer of polarizing material.

6. The binocular device of claim 5 , wherein the polarizing filter comprises at least two layers of the polarizing material, the two layers of the polarizing material having polarization directions that cross each other.

7. 7. A binocular device as claimed in any one of claims 4 to 6, further comprising a light source coupled to the support structure capable of generating light radiation within at least the wavelength bands of invisible light radiation and the wavelength bands of visible light radiation, the light source configured to generate an emission beam of visible light radiation and an emission beam of invisible light radiation that are geometrically and positionally aligned by optical elements.

8. a light source coupled to the support structure capable of generating optical radiation within at least the invisible light radiation wavelength band and the visible light radiation wavelength band, the light source further comprising a polarizing filter configured to polarize the visible light radiation within the visible light radiation wavelength band output by the light source and to transmit the optical radiation within the invisible light radiation wavelength band; The binocular device of claim 5 , wherein the binocular device is configured to implement cross-polarization between the light sources and a camera input.

9. 9. A binocular device according to any one of claims 4 to 8, wherein the input optics corresponding to the camera sensitive to invisible light radiation comprises a diaphragm having an aperture, the material of the diaphragm around the aperture being transmissive to the light radiation in the wavelength band of the invisible light radiation while being attenuative to the light radiation in the wavelength band of the visible light radiation so as to pass only a portion of the visible light radiation.

10. 10. The binocular device of claim 9, wherein the input optics comprises a lens with an autofocus, the autofocus configured to focus the optical radiation in the wavelength band of the invisible light radiation.

11. 11. A binocular device as described in claim 9 or 10, wherein the input optics of the camera sensitive to optical radiation in the wavelength band of invisible light radiation further comprises a filter, which in addition to the aperture is attenuative for the optical radiation in the wavelength band of visible light radiation so as to pass only a portion of the visible light radiation.

12. A binocular device according to any one of claims 4 to 11, wherein the visible light radiation reducing filter comprises iodine for reducing the light radiation in a wavelength band of the visible light radiation.

13. 13. The binocular device of any one of claims 3 to 12, further comprising a light source coupled to the support structure for generating invisible light radiation in at least a wavelength band of the invisible light radiation and visible light radiation in a wavelength band of the visible light radiation, the light source configured to emit the invisible light radiation intermittently while keeping a visible light radiation intensity constant, and the camera configured to capture at least one image with the emission of the invisible light radiation and at least one image without the emission of the invisible light radiation.

14. 14. The binocular device of claim 13, further comprising a processing unit configured to calculate an enhanced image of light radiation in the wavelength band of invisible light radiation based on the image captured with the emission of invisible light radiation and the image captured without the emission of invisible light radiation.

15. 15. A binocular device as claimed in any one of claims 4 to 14, wherein each of the left and right cameras is sensitive to optical radiation in the wavelength band of invisible light radiation and to optical radiation in the wavelength band of visible light radiation, and each of the left and right input optics comprises a visible light radiation reducing filter, the visible light radiation reducing filter being transmissive to the wavelength band of invisible light radiation and reducing to the wavelength band of visible light radiation so as to pass only a portion of the visible light radiation.

16. 16. A binocular device as claimed in any one of claims 3 to 15, wherein the image sensor of the camera sensitive to the optical radiation in the wavelength band of invisible light radiation comprises a sensor die sensitive to both the wavelength band of invisible light radiation and the wavelength band of visible light radiation, the sensor die configured to output the video signal corresponding to both the optical radiation in the wavelength band of invisible light radiation and the optical radiation in the wavelength band of visible light radiation.

17. A binocular device according to any one of claims 3 to 16, wherein the wavelength band of invisible light radiation is a wavelength band of near-infrared light radiation.

18. 1. A method for visualizing optical radiation in a binocular device, comprising: receiving optical radiation for a same field of view along corresponding left and right input optical axes by left and right input optics coupled to a support structure, the left and right cameras of the binocular device transmitting the optical radiation to a left image sensor of the left camera and a right image sensor of the right camera, respectively, the left and right cameras being coupled to the support structure; the left and right image sensors producing left and right video signals from the optical radiation received and detected by the left and right cameras, respectively; receiving, by a processor of the binocular device, signals representing a left image from the left camera and a right image from the right camera simultaneously captured by the left camera and the right camera, respectively; the processor comparing the left image and the right image; and wherein the processor detects whether there is a specular reflection based on a result of the comparison.

19. A method for visualizing light radiation as described in claim 18, further comprising the step of a left display and a right display coupled to the support structure presenting left and right video images formed of visible light radiation based on the left and right video signals, respectively, via a left eyepiece operably connected to the left display and a right eyepiece operably connected to the right display.

20. further comprising a left display and a right display coupled to the support structure, the left display and the right display arranged to be viewed by a pair of eyes of a user through a left eyepiece operatively connected to the left display and a right eyepiece operatively connected to the right display, and configured to present, by the left display and the right display, left and right video images formed in visible light based on the left and right video signals, respectively; 2. The binocular device of claim 1, wherein the binocular device is configured, in a particular visualization mode, to alternately show a left image based on the left video signal on the left display and a right image based on the right video signal on the right display to enable a user to identify specular reflections.

21. 20. The method of visualizing light radiation of claim 19, wherein the presenting step includes, in a particular visualization mode, alternating between showing a left image based on the left video signal on the left display and a right image based on the right video signal on the right display.

Images