Optical see-through device that can augment visual perception

Abstract

The inventive device (100) is an optical see-through device that can augment visual perception of a real-world scene comprising an input module that can capture an image of the real world, a processing and control module that can modify the image and create an overlay image, and an optical see-through output module that comprises a combiner unit and a display engine that can overlay the optical see-through view of the real world with the overlay image to create a combined view, characterized in that - the camera of the input module (200) can capture an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit (204), - where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit (204) cannot perceive, the captured image can be used by the processing and control module (201, 202) to create an overlay image that, when displayed by the display engine (203) and combined with the optical see-through view, enables the eye looking through the combiner unit (204) to perceive the dark areas of the scene, - where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit (204) can perceive, light can be added using the overlay image (created by the processing and control module (201, 202) and the display engine (203), so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

Description

[0001] Title: Optical see-through device that can augment visual perception

[0002] Description:

[0003] The invention deals with an optical see-through device that can augment visual perception of a real-world scene comprising an input module that can capture an image of the real world, a processing and control module that can modify the image and create an overlay image, and an optical see-through output module that comprises a combiner unit and a display engine that can overlay the optical see-through view of the real world with the overlay image to create a combined view.

[0004] Such a device is known from US 2020 / 183171 A1 , where an optical see-through device, using a camera as input module, can capture an image of the real world, modifies the image and overlays the modulated image onto the user’s optical see- through view of the real world.

[0005] The known device has the problem that the combined view, i.e. the optical see- through view of the real world combined with the overlayed modulated image, is not very satisfying for the user of the device; some areas in the combined view may be too bright, while others may be too dim, or artifacts may be present.

[0006] It is the aim of the invention to present an augmented view to the user of the device that does not have the above problems.

[0007] According to the invention, the optical see-through device is characterized in that the input module can capture an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit, where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit cannot perceive, the captured image can be used to create an overlay image that, when combined with the optical see-through view, enables the eye looking through the combiner unit to perceive the dark areas of the scene, where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit can perceive, light can be added using the overlay image, so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

[0008] The inventor has found that the known device does not specifically distinguish between dark and bright areas. This means that the camera used as input module in the known device will more or less have the same view as the eye of the user, especially when the camera is used with an auto exposure mode. This means that the overlay image may make already bright areas even brighter, whereas, even after the augmentation, it may still not be possible to perceive dark areas. The inventive device uses an input device that can make a low-radiance image, i.e. an image with radiance levels that the eye cannot perceive through the combiner unit. Such a low-radiance image can be used to augment the view of the user in the dark areas.

[0009] The inventor further realized that simply using the low-radiance image to augment the dark areas can create artifacts on boundaries between dark and bright areas. Moreover, this can also create a view for the user where the dark areas appear brighter than the bright areas, giving an unnatural view. The inventor found that adding light to the bright areas, i.e. those areas that the user can already see, can solve the problems, both of the artifacts and the unnatural view, by maintaining the order between radiance levels in dark and bright areas. The dark areas can be seen, but they are never brighter than the bright areas. Adding light to bright areas may seem contradictory, but it solves the problems encountered in the known device.

[0010] Besides the view provided to the user, this approach of the invented device has additional advantages as compared to the known device:

[0011] 1. It conserves power by reducing the display engine's brightness in well-lit areas, which typically consume more energy, since only a small amount of light is added in these areas.

[0012] 2. It minimizes unwanted effects like "eye glow", where light from the display engine is visible to others around the user. Additionally, due to the optical see-through architecture used in the known device, input and output modules cannot be aligned physically without obstructing the real-world view, as the optical center of the input module should not be positioned within the line of sight, right in front of the user’s eye. Thus, there is a parallax that results from the difference between the optical center of the input module that captures the image and the optical center of the real-world view through the device. When overlaying the camera image onto the real-world view, this parallax will result in distortion and double vision, which negatively affects visual perception of the scene. In the present invention, the difference in specification (e.g, field of view) and position of the input module relative to the output module are considered in the creation of the overlay image. The captured image is transformed with image processing methods such as cropping, scaling, shifting, rotating, warping, and flipping, so, when it is used to create an overlay image, there is a better alignment between regions in the overlay image and the same regions in the see-through view of the real world. At the cost of more processing capacity, alignment can be further improved by creating a monocular depth estimation using a neural network, which can be used to transform the image to adjust for the parallax.

[0013] In an advantageous embodiment of the inventive device, the overlay image can be created by specifying a fixed radiance level that will be used to add light to the bright regions. This added light pushes all “bright” radiance levels up by the specified level, creating a band of “empty” radiance levels at the border between dark and bright. This band is then used to show the dark regions: the input module observes radiance levels that are too low for the user to perceive and, using the display engine and combiner unit, light is added to these radiance levels to lift these radiance level from dark to bright. Since these levels are known, the exact amount needed for a proper redistribution (ensuring that the order of radiance levels isn’t disturbed) can be determined.

[0014] In another advantageous embodiment, the inventive device is characterized in that there is an overlap between the radiance levels that the input module can capture and the radiance levels that the eye can perceive through the combiner unit, where the overlap can be used for determining the light added to the bright areas. By using an input module that has such an overlap with the eye, the overlapping radiance levels give an indication of the amount of light the eye perceives in those areas. Since the overlap is in the lower range that the eye can perceive, the overlap can be used to determine how much light needs to be added to the bright areas to make a natural view, where the order of radiance levels is maintained. In this case, the redistribution of radiance levels that are observed by the input device can be done over the entire range of known radiance levels, meaning that it is no longer necessary to add light to regions that are too bright for the input device to observe.

[0015] Preferably, the inventive device is characterized in that the input module can capture multiple low-radiance images with different radiance levels that cannot, or only partly, be perceived by the eye through the combiner unit, where the multiple images can be used for one overlay image. This gives the possibility to adjust the input module to different low-radiance sensitivities. Thus, enabling to capture a wider range of low- radiance values. These different images then give the possibility to further augment dark areas with low radiance.

[0016] Preferably, in the inventive device, the input module can capture multiple images with at least one image for capturing radiance levels not perceivable by the eye looking through the combiner unit and at least one image capturing radiance levels perceivable by the eye looking through the combiner unit. This way, the input module in the different settings can capture low-level radiance levels in the dark areas, but also capture an image of the real-world view that the eye of the user can perceive. Knowing what the user sees, improves the capability to create a “natural” view for the user by better integrating the dark and bright areas.

[0017] In another preferred embodiment of the inventive device, the input module comprises a multi-camera system, to allow the device to capture multiple images simultaneously. This means that the device becomes faster, but also, the cameras can each be dedicated to specific radiance levels, and even different eyes of users, which would give in general better specifications for individual cameras and the overlay images. Preferably, the invented device comprises additional components, such as sensors or optical elements, that can be used to facilitate the alignment of overlay images with the optical see-through view of the real world. For example, by adding motion sensors and using a structure-from-motion approach, adding distance sensors such as lidar, or the use of stereo cameras, a depth map can be obtained, which can be used to transform the overlay image and facilitate alignment of the overlay image with the real-world view of the user. For an embodiment where the input module comprises a single camera, a depth map can also be used as an input for the creation of multiple overlay images, i.e. , a unique overlay image for each eye of the user. This procedure may include inaccuracies as some pixel values will need to be estimated. Another method for improved alignment is adding optical elements such as a semitransparent mirror at an angle in the line of sight of the user. The light of the real world is then visible to the user, while at the same time, it is reflected into the camera of the input module.

[0018] Preferably, additional optical elements are added to the output module to manipulate the optical see-through view of the real world. These can include (electronically controlled) prescription lenses, prisms and filters, positioned between the user’s eye and the combiner unit, on the outer surface of the combiner unit or on the input module. The outer surface of the combiner unit is the surface that faces the real world. These optical elements will allow the user to get an improved view of the real world through the combiner unit, by focusing the light, shifting the view or dimming the light. Dimming the light through filters can be advantageous when using the device in specific circumstances, e.g., as sunglasses. In that case, the device can incorporate an optical filter that lowers the radiance of the optical see-through view of the real world. As such, very bright areas can be observed, because of the filter and areas in the shadows, where contrast differences that would not be easily perceivable through normal sunglasses can be seen clearly with the inventive device. From the outside, it will look like the user is wearing normal sunglasses, but the user will have the added benefit that they can see clearly in scenes with a high dynamic range (i.e., being able to see the contrasts in dark shadows while in a bright, sunlit area). The light transmission of smart or liquid crystal dimmers can be adjusted by applying a voltage across the filter. Displays like e-ink can also be used to make more or less transmission through the filter possible in a controlled way. Such a use of adaptable optical filters can make the use of the device much broader, further increasing the dynamic range in which it can be applied. With little light transmission, the device can be used in bright sunlight, whereas, with full light transmission, the device can be used in low-light circumstances. Another example of such an application is as welding goggles, where almost no light is permitted through the goggles to prevent the (UV and excessive visible) light generated by the welding process from reaching the user’s eyes, but this also dims the entire environment for the welder. This is especially a problem in the time leading up to the actual welding process, as the welder wants to clearly see their workpiece. The inventive device would still keep the excessive visible light and the UV light from reaching the user’s eyes during the welding process, but would brighten the environment before, during and after the welding process, allowing the welder to clearly see their workpiece before, during and after the welding process. In a preferred embodiment, the optical filter comprises a film or display with semitransparent regions that can be individually darkened. Here, transmission of individual areas can be modified to filter out bright lights. This is especially useful in a device used, for instance, for nightly driving a car. Light from oncoming vehicles can be very bright. The processing and control module can then give signals to those regions to darken where the bright lights are visible to the user, thus preventing that a bright spot would temporarily blind the user. It is also possible for special applications, to protect the eyes of the user to accidental exposure to very bright lights that could otherwise cause blindness, assuming this very bright light doesn’t cause instantaneous blindness.

[0019] Preferably, by means of artificial intelligence, the device can improve the captured image by correcting pixel values and reducing the effects of glare, noise and non-linearity in the camera sensor’s response. The artificial intelligence comprises a neural network that is trained by providing an image and certain characteristics of the captured image, such as the camera settings, as well as radiance measurements for pixels or groups of pixels in the image from a radiance meter. Using this data, the neural network can predict pixel values that provide a more reliable representation of the radiance in the scene, for example, by reducing glare or noise. Together, the corrected pixels form a corrected image that can be used by the device for further image processing and creation of an overlay image.

[0020] Preferably, the device can, by means of image processing algorithms, apply adjustments to the combined view through additional modification of the overlay. These algorithms can be applied to the entire combined view or a region of this view. This can be done by creating an estimation of the intended combined view that can be realized by adapting the overlay image, so when it is overlayed on the optical see- through view of the real world, the intended combined view will be realized. As discussed in previous sections, the radiance of the optical see-through view is taken into consideration in this process. Further enhancing the capability to adjust the combined view are the methods described to manipulate the optical see-through view of the real world as described in previous sections. All in all, the adjustments can include operations such as contrast enhancement, edge enhancement, colour saturation, colour balance, white balance, sharpness enhancement, blur, histogram equalization and more complex adjustments, such as application of deep learningbased image enhancement models or vision-based models like Multi-Scale Retinex theory. These adjustments can, e.g., be used to visually draw the user’s attention to something in front of them or to increase contrast within the scene. The inventors found that increasing the contrast can be especially advantageous for people with visual impairments such as night blindness.

[0021] According to the invention, additional information related to the image or depth map can be added to the overlay image. Information, such as navigation information or extra information regarding objects, the scene, warnings, locations, battery life of the device, or device settings and modes, can be added. Navigation information may include written directions, arrows or paths, plotted in the environment or a virtual guide that can be followed by the user. Such navigation information includes both high-level instructions, comparable to instructions provided in a satellite navigation system in a car, as well as low-level instructions, like short paths around obstacles or road obstructions. Extra information can range from warnings of obstructions like steps, stairs and poles to touristic information. Such information can also be transferred, for instance, wirelessly via a mobile phone. The invention also relates to a method to augment visual perception of a real- world scene, using an input module that captures an image of the real world, a processing and control module that modifies the image and creates an overlay image, and an optical see-through output module that comprises a combiner unit and a display engine that overlays the optical see-through view of the real world with the overlay image to create a combined view, characterized in that the input module captures an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit, where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit cannot perceive, the captured image is used to create an overlay image that, when combined with the optical see-through view, enables the eye looking through the combiner unit to perceive the dark areas of the scene, where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit can perceive, light is added through the overlay image, so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

[0022] In the above text, “radiance” is used instead of “luminance” to express the brightness of the scene, including infra-red light, the observed light through the combiner unit and the display engine. This is done both to not confuse the reader and because the term “luminance” is primarily defined for photopic vision, while this invention mainly deals with scotopic vision. Everything in the above text is meant to include infra-red light for the lighting of the scene and the incoming light for the input module.

[0023] DESCRIPTION OF FIGURES

[0024] The invention is further explained with the help of the following drawings, in which:

[0025] Figure 1 : schematic view of the inventive device; Figure 2: example embodiment of the inventive device in the form of smart glasses;

[0026] Figure 3: a dynamic range of a scene annotated with sub-ranges related to the inventive device;

[0027] Figure 4: scene observed by the eye of the user through the combiner unit;

[0028] Figure 5: scene as captured by a camera with auto-exposure;

[0029] Figure 6: real-world scene as captured by the input module of the inventive device;

[0030] Figure 7: combined view as perceived by the user of the inventive device;

[0031] Figure 8: misalignment between the optical see-through view and the overlay image;

[0032] Figures 9a and 9b: possible adaptations by the processing and control module to align the captured image with the optical see-through view;

[0033] Figure 10: example graph of input and output radiances using a fixed added radiance level;

[0034] Figure 11 : example graph of input and output radiances using an radiance-level overlap between the camera and the eye;

[0035] Figure 12: device overview with an additional component to facilitate better alignment of the captured image with the optical see-through view;

[0036] Figure 13: device overview with additional optical elements.

[0037] Figure 1 shows a schematic view of an optical see-through device (100) that can augment visual perception of a real-world scene comprising an input module (101) that can capture an image of the real world, a processing and control module (102) that can modify the image and create an overlay image, and an optical see-through output module (103) that comprises a combiner unit and a display engine that can overlay the optical see-through view of the real world with the overlay image to create a combined view, characterized in that the input module (101) can capture an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit, where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit cannot perceive, the captured image can be used to create an overlay image that, when combined with the optical see-through view, enables the eye looking through the combiner unit to perceive the dark areas of the scene, where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit can perceive, light can be added using the overlay image, so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

[0038] This device (100) can be configured in embodiments, including, but not limited to: eye glasses, smart glasses, contact lenses, headsets, helmets, monoculars, binoculars, telescopes, head-up displays and windows, for instance of a car.

[0039] The inventive device is especially useful for users suffering from nyctalopia (night blindness), since the radiance level at which these users can no longer perceive something is significantly higher than for others.

[0040] Figure 2 shows an example of the optical see-through device (100) configured as a pair of smart glasses with a monocular camera (200) and display engine (203) setup. The glasses have an integrated camera (200) as input module (101), an onboard processing and control module (201) as processing and control module (102), a display engine (203) and a see-through combiner unit (204) as its output module (103).

[0041] The processing and control module (102) can be integrated (201 , 202) or separate from the optical see-through device (100). It may be used to manually operate the camera settings, image processing parameters, brightness of the display and dimming effects, and turn the device on, off or to sleep mode. It may consist of buttons, dials, capacitive or resistive touch elements, microphones, light sensors, eye or hand tracking cameras, or other means by which a user can convey controls. The glasses can be controlled wirelessly, via buttons or via a capacitive touch sensor that functions as input for the build-in processing and control module (102, 201). The capacitive touch sensor (202) shown in figure 2 is an example of such a control. The input module (101) comprises a camera (200) that can capture images in a large variety of light levels (i.e., extreme low light to bright sunlight) and wavelengths, such as visible light and infrared, making it useable as both a day-vision and a nightvision device.

[0042] Counterintuitively, the inventors found that the camera does not have to be a camera with high resolution, such as found in consumer devices like smartphones. A low-resolution camera suffices for use cases like improving pedestrian mobility for people suffering from night blindness. For this use case, detailed images are not needed. Moreover, it is advantageous to use a low-resolution camera, as for the same surface area of the image sensor, the pixel size is larger, which allows the camera to capture areas with low radiance levels more effectively. This can also be achieved by so-called binning pixels of a high-resolution image sensor (i.e., combining adjacent pixels into a single pixel), thereby, decreasing the resolution and increasing surface area for a pixel value. Decreasing the resolution has the added advantage that less pixel values need to be processed, allowing the device to achieve a lower latency.

[0043] An infrared illuminator light emitting diode (LED, not shown) array may be added to the device to make it useable in low-light scenes without any environmental light. The camera settings, such as the shutter speed, gain, focus and (infrared) light filter and illuminator, may be adjusted manually or automatically by the processing and control module (102) to allow for sufficient image quality and performance, based on user needs and the real world.

[0044] Figure 3 illustrates a dynamic range (300) for a low-light scene. In this figure, left is low radiance and right is high radiance. The eye looking through the combiner unit, but without the device turned on, has a perceived instantaneous dynamic range (301).

[0045] Figure 3 shows that dynamic range (302) beyond the perceivable range (301) of the eye can be captured by the input module (101), for instance a camera (200), since the camera can be set to a different sensitivity from the human eye. This dynamic range (302) is then processed by the processing and control module (102) and redistributed (303) within the perceivable dynamic range (301). Due to the processing and redistribution (303) of radiance levels within the imperceivable range (302) into the perceivable dynamic range (301), the total perceivable dynamic range of the user can be expanded to comprise range (304). In order to prevent artifacts range (301) should be modified by for example redistributing range (301) in range (305).

[0046] A technical effect can also be achieved for dynamic ranges that (partly) overlap with the perceivable range or ranges that are outside the perception range for high radiance areas by the human eye, i.e. on the right side of figure 3. For the latter, the dynamic range outside the perceivable range (301) is dimmed so that it falls again within the range (301) or the entire range (300) is dimmed, after which the radiance levels that fall under the perceivable range (302) are redistributed (303) into the perceivable range (301).

[0047] Figure 4 shows a scene as observed by the eye of the user through the combiner unit. Most of the scene is too dark for the user to observe in this instance (400), because the user has just come from outside, where it is light (401). The user can perceive (401) and cannot perceive (400).

[0048] When using a camera with auto-exposure, part of the room is visible, see Figure 5, but, due to the wide variance in radiance levels in the scene, the camera cannot capture the entire room. The left part of the scene (500) is too dark, due to the presence of the bright doorway (401) in the view of the camera. Only the bright areas (401 , 501) can be captured.

[0049] Figure 6 shows the scene as captured by the camera (200) of the input module (101) of the inventive device (100). The camera settings are modified by the processing and control module (102) to capture the part of the scene that is not perceivable by the eye (600). This results in the outside part of the scene becoming invisible to the camera (401 , 601) due to overexposure. This part could already be perceived comfortably by the user (401) and is therefore unnecessary to capture for the camera (200). Figure 7 shows the scene as perceived by the user of the inventive device (100). From the image of the camera (200), an overlay image is created, which is overlayed on the see-through view. As such, the radiance levels are redistributed enabling the user to see the entirety of the scene (700) within their own dynamic range.

[0050] In order to facilitate the redistribution, an overlay image is created in the processing and control module (102):

[0051] Entire images, select regions or pixels from the input module are analysed to measure the radiance levels in the scene and automatically control the settings and characteristics of both the input and output modules (101 , 103). As the analysis can be conducted on every individual image or combination of images within a fraction of a second, the device (100) utilizing this methodology can rapidly adapt to changes in radiance levels in the real world.

[0052] This analysis may also be used for safety measures. For example, in a low-light scene, a user may have dilated pupils. When the device (100) is started, the radiance of the display engine (203) will be set to a low level so as not to risk irritating or damaging the eyes of the user. Radiance may be gradually increased automatically or manually by the processing and control module (102), to allow the eyes to adapt to scene elements with high radiance levels, such as the headlights from a car. Due to the shrinking pupil size, some scene elements will become too dim to perceive and, through the overlayed view, the device can bring these elements closer to the radiance level of the headlights, thereby bringing them in the perceivable range.

[0053] When overlaying the optical see-through view (800) with the overlay image (801), double vision or distortion of the combined view can occur due to misalignment. Figure 8 shows an example of misalignment. To resolve or minimize the effect of misalignment the image captured by the camera (200) of the input module (101) should be adapted.

[0054] Figure 9 shows how, according to the invention, the processing and control module (102) can adapt the captured image to correct for the difference between the input module’s (101) and output module’s (103) specifications and a parallax resulting from the difference between the optical center of the input module (101) and the optical center of the eye of a user. Examples of differences in specifications between the input (101) and output (103) modules are a different field of view or a different aspect ratio.

[0055] Adaptation can be based on a controllable distance between the user and the areas observed. Here controllable means that the correction can depend on the distance of an area to the user. The adaptation can include transformations such as: warping, cropping, rotating, scaling as shown in figure 9a, in which area (900) is scaled to fit over area (901), and shifting as shown in figure 9b, where area (902) is shifted to align with area (903). Transformations may also be based on information from a depth map which can be obtained using a neural network that provides depth predictions, structure from motion, stereo cameras, or depth sensors. The depth map can then be used to adapt the image so it aligns with the optical see-through view of the real world.

[0056] Additionally, the transformations may be applied to correct for the parallax created by the difference between the position of the user’s eye relative to the position of the camera (200. These operations are another important distinction from the known device as they further decrease unwanted side effects from misaligned image overlay, such as occlusion and double vision.

[0057] Also, misalignment may result from a delay between light observed from the real-world view and the light from the same moment in time as captured by the input module (101), processed by the processing and control module (102) and displayed by the output module (103), which takes more time to reach the eye of a user. Hence, this delay should be minimized by optimizing the device so latency is at a minimum and the image-capture and display rates are as high as possible. Preferably, the latency should be lower than 20 milliseconds and the image-capture and display rates should be higher than 30 images per second. The adapted camera image is then processed using algorithms. Algorithms maybe used to filter, de-noise, sharpen, colour balance, white balance, saturate, or alter the contrast, alter the brightness or reduce glare in the image.

[0058] Next an overlay can be created using the processed image, a set of rules such as the operational mode and user settings, or additional images from the camera can be used in this process.

[0059] As an example, an input to output graph of a scene in black and white with radiance levels from 0 (i.e. , black) to 100 (i.e. , white) is provided in figure 10.

[0060] Figure 10 shows the relationship between the input radiance (horizontal axis) of the scene (1000), the overlay image (1003) and the combined view (1004) and the output radiance (vertical axis) by the input module (101) I camera (1002), the output module (103) I overlay (1004) and the user’s eye through the combiner unit (1001).

[0061] The radiance of the scene is illustrated by the double grey line (1000), input and output radiance for the scene are equal. Please be aware that line (1000) is overlapped by other lines in the figure. The scene is observed by a person suffering from night blindness looking through the combiner unit (204). To simplify, the explanation of the principles at work the combiner unit transmits 100% of the light of the scene (1000) in this example. However, in a real-world application of the inventive device (100) the processing and control module (102) should account for the light transmission rate of each area of the combiner unit (204) in controlling the input and output modules (101 , 103) and creation of the overlay image.

[0062] The eye of the user suffering from night blindness can perceive, in any one view, a certain range of radiance levels indicated by the line (1001). In this example, 51 radiance levels: 50 to 100. Any radiance levels from 50 to 100 is considered “bright”. The user will perceive any radiance level below this range as equal to the radiance level of the low end of the range, as indicated by the flat line segment of line (1001). (Any radiance level above this range would be perceived as equal to the radiance level of the high end of the range.) Any radiance level below 50 is considered “dark”. The camera of the input module (101) observes the same scene and captures an image of a certain range of radiance levels indicated by line (1002). In this example, 50 radiance levels: 0 to 49. For the camera, any radiance level above this range is registered as white as indicated by the flat line segment of line (1002). In this example figure the input to output relationships are expressed as straight lines or line segments. However, these lines do not have to be straight.

[0063] When overlaying the low-level image without considering the radiance of the scene propagating through the combiner unit, artifacts can form. For example, to make radiance levels 0 to 49 visible to the user, they should be boosted to fall within radiance levels 50 to 100. This can be done by simply adding the lower limit (50) to all radiance levels lower than 50. Radiance level 49 would in this case be boosted by 50 to 99. However, this results in a problem. Radiance level 49 is darker than radiance level 50. However, after boosting it to 99, it becomes much brighter. As such, something like a shadow can become brighter than the object casting the shadow, which looks unnatural. Moreover, when overlaying the boosted image onto the optical see-through view of the real world without considering the radiance of the scene transmitted through the combiner unit, the problem is emphasized.

[0064] The inventive device (100) resolves this problem. Radiance levels on the low- level image can be redistributed within range 50 to 100 by specifying a fixed radiance level, for example 25. This radiance level is the amount of added light to the bright regions through the overlay image as indicated by line (1003), For example, scene input radiance level 50 on line (1000) is overlayed with radiance level 25 (1003), to provide, in the combined view, radiance level 75 to the user as indicated by combined view line (1004). In the dark regions, the sum of the fixed radiance level and the lowest perceivable radiance level minus one level is the upper radiance level limit for creating the overlay image. In this example: 25 (i.e., the fixed radiance level) + 50 (i.e., the lower limit of the eye) - 1 = 74. Therefore, the low-level image of range 0 to 49 (1002) can be redistributed in range 50 to 74. This redistribution is realized through the overlay image (1003) that, together with the scene radiance (1000), results in a combined view (1004). Note that since the camera is not able to perceive differences in radiance levels 49 or higher, the device (100) cannot determine how much light should be added in those regions to redistribute the entire range.

[0065] Also note how for any input radiance level (1000), the corresponding output radiance level in the combined view (1004) is always in the same order relative to neighbouring levels; no darker region in the scene is brighter than any brighter region in the scene.

[0066] A drawback of this method is that bright areas become even brighter. In this example, bright areas with radiance levels 76 to 100 are increased to levels 101 to 125, meaning that they have been shifted beyond the upper limit of the eye. The user can no longer observe any contrast in these regions. If a substantial portion of the visual field comprises those areas, the instantaneous dynamic range of the eye may shift as well since it adapts by for example constricting the pupil (e.g., the eye may shift to radiance levels 76 to 125).

[0067] A method to resolve such issues is to use the camera (or multiple cameras) to capture radiance levels that (partially) overlap with the radiance levels perceivable by the eye through the combiner unit (204). For all areas where the camera-captured radiance levels overlap with the perceived radiance by the eye, an overlay can be created that redistributes the dark levels plus the overlapped levels.

[0068] An example input-to-output graph is provided in figure 11. The radiance of the scene is illustrated by the double grey line (1100). The eye can perceive, in any one view, a certain range of radiance levels indicated by the line (1101). In this example, 51 radiance levels: 50 to 100. The camera of the input module (101) observes the same scene and captures an image of a certain range of radiance levels indicated by line (1102). In this example, 80 radiance levels: 0 to 79. The processing and control module (102) creates an overlay image (1103) for areas with radiance level 0 to 79 so to redistribute range 0 to 79 within range 50 to 79 in the combined view (1104). Areas above range 0 to 79 remain unaffected by the process, as indicated by the overlap of the line of the combined view and scene (1100, 1104). In figure 12 solid arrows (e.g.,1200), with the exception of the indication arrows, represent light and arrows with dashed lines (e.g., 1204) represent data streams.

[0069] Figure 12 shows how, according to the invention, additional components can be used to facilitate alignment of the overlay image (1208) coming from the display engine (1207) with the optical see-through view of the real world (1202b). The overlay image (1208) combined with the optical see-through view of the real world (1202b) is the combined view (1210), which is what is perceived by the eye (1211) of the user of the device. In this example, a semi-transparent mirror (1201) is placed in the light coming from the scene (1200), so that part of the light (1202a) is reflected towards the input module (1203), while the rest of the light (1202b) coming from the scene continues through the combiner unit (1209). The light (1202) captured by the input module (1203), which sends its signal (1204) to the processing and control module (1205) to create an overlay image (1206) and presented (1208) to the eye of the user (1211), using the display engine (1207). The use of the semi-transparent mirror (1201) makes it easier to align the light of the overlay image (1208) with the light of the optical see-through view of the real world (1202b).

[0070] Figure 13 show a version of the inventive device (100) that includes additional optical elements on the inside (1311) and outside (1308) of the combiner unit (1310). The light coming from the scene (1307) is transmitted through the combiner unit (1310) to reach the user’s eye (1313) as one part of the combined image (1312). The light coming from the scene (1307) is also observed by the input module (101 , 1301) that sends its signal (1302) to the processing and control unit (1303), which makes an overlay image (1304) from the incoming signal (1302). This overlay image (1304) is sent to the display engine (1305) that displays the overlay image (1306) to the user’s eye (1313). The two light streams that reach the user’s eye (1313) (the optical see- through (1309) and the overlay image (1306)) make up the combined image (1312).

[0071] Figure 13 shows how, according to the invention, both the inner surface (1311) and the outer surface (1308) of the combiner unit (1310) of the output module (1305, 1308, 1310, 1311) comprises optical elements, e.g., a lens, prism or an optical filter, that can manipulate the optical see-through view of the real world (1309) and of the combined view (1306 + 1309). These optical elements manipulate light from the real- world view (1307), i.e. the outside world, and from the display engine (1305).

[0072] Preferably, the optical elements (1308, 1311) comprise adaptable optical elements or optical elements that comprise a film or display with semi-transparent areas that can be individually darkened.

[0073] The optical elements (1308, 1311) may include a colour filter (such as a bluelight blocker), (graduated) neutral density filter, a polarization filter, photochromatic material, or coatings such as anti-reflection, anti-fog, UV-blocker, mirror coating or similar.

[0074] In other embodiments, the optical elements (1308, 1311) may also include a smart film or display to allow for electronic control of the light transmission. With optical elements on the outside of the combiner unit (1308), light from the real world (1307) can be dimmed to provide light (1309) with a lower radiance level to the user’s eye (1313). Next, this light (1309) enters the combiner unit (1310). The combiner unit (1310) provides an optical see-through view of the real world from light (1307), while also providing an overlay image (1306), using the display engine (1305). The combiner unit (1310) may be based on a free-space combiner unit, a free-form total internal reflection prism combiner unit or a waveguide (also known as lightguide) combiner unit, or any comparable optical architecture. The display engine (1305) can consist of three main building blocks: an illumination engine (in case of non-emissive display panels), a display panel or scanner, and optics that can relay the image (1306) to the combiner unit (1310). The illumination engine may be based on a laser beam scanner (LBS) display, (active matrix) organic light-emitting diode (AMOLED / OLED) display, liquid crystal display (LCD), liquid crystal on silicon (LCOS) display, micro-light emitting diode (amu-LED / mu-LED / mu-iLED), digital light processing (DLP) display, or light emitting quantum dots or any comparable display paradigm.

[0075] The brightness of the illumination engine or the emissive display panel can be controlled automatically or manually through the processing and control module (1303). Dynamic and rapid control of the brightness of the display engine (1305) is required to permit safe and effective usage in a variety of lighting conditions and increase adaptability to changes in radiance levels of the real world (1307).

[0076] Figure 13 also shows an example of an embodiment of the output module (1305, 1308, 1310, 1311) comprising a prescription lens on the inside (1311) of the combiner unit (1310). This prescription lens will be placed after the combiner unit (1310). The lens may include a single vision lens, bifocal lens, trifocal lens, progressive lens or prism lens. The lens may work in tandem with the lens at the outer surface (1308) of the combiner unit (1310) to provide even more methods to manipulate the optical see-through view and the overlay image.

Claims

CLAIMS1. Optical see-through device that can augment visual perception of a real-world scene comprising an input module that can capture an image of the real world, a processing and control module that can modify the image and create an overlay image, and an optical see-through output module that comprises a combiner unit and a display engine that can overlay the optical see-through view of the real world with the overlay image to create a combined view, characterized in that the input module can capture an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit, where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit cannot perceive, the captured image can be used to create an overlay image that, when combined with the optical see-through view, enables the eye looking through the combiner unit to perceive the dark areas of the scene, where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit can perceive, light can be added using the overlay image, so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

2. Device according to claim 1 , characterized in that the overlay image can be created by specifying a fixed radiance level that will be used to add light to the bright regions, whereas in the dark regions the sum of the fixed radiance level and the lowest perceivable radiance level is the upper radiance level limit for creating the overlay image.

3. Device according to one or more of the preceding claims, characterized in that there is an overlap between the radiance levels that the input module can capture and the eye can perceive through the combiner unit, where the overlap can be used for determining the light added to the bright areas.

4. Device according to one or more of the preceding claims, characterized in that the input module can capture multiple images with different radiance levels that cannot be perceived by the eye through the combiner unit, where the multiple images can be used for one overlay image.

5. Device according to one or more of the preceding claims, characterized in that the input module can capture multiple images with at least one image for capturing radiance levels not perceivable by the eye looking through the combiner unit and at least one image capturing radiance levels perceivable by the eye looking through the combiner unit.

6. Device according to one of the claims 4 or 5, characterized in that the input module comprises a multi-camera system, to allow the device to capture multiple images simultaneously.

7. Device according to one or more of the preceding claims characterized in that the device comprises additional components such as sensors or optical elements that can be used to facilitate the alignment of overlay images with the optical see-through view of the real world.

8. Device according to one or more of the preceding claims characterized in that the output module comprises additional optical elements, which can be added to the output module to manipulate the optical see-through view or the view of the overlay image.

9. Device according to claim 8 characterized in that the optical elements are adaptable.

10. Device according to one or more of the preceding claims, characterized in that it can utilize a neural network to process a captured image and correct pixel values of the captured image so to provide a more accurate representation of the radiance levels in the scene.

11. Device according to one or more of the preceding claims, characterized in that, by means of image processing algorithms, an intended combined view can be estimated that can be realized by adapting the overlay image, so that when the adapted overlay image is overlayed on the optical see-through view, the intended combined view is realized.

12. Device according to one or more of the preceding claims, characterized in that additional information of interest to the user can be added to the overlay image.

13. Method to augment visual perception of a real-world scene using an input module that captures an image of the real world, a processing and control module that modifies the image and creates an overlay image, and an optical see-through output module that comprises a combiner unit and a display engine that overlays the optical see-through view of the real world with the overlay image to create a combined view, characterized in that the input module captures an image with radiance levels that are too low to be perceived by the eye looking through the combiner unit, where, in at least dark areas, i.e. in areas of the real-world scene that the eye looking through the combiner unit cannot perceive, the captured image is used to create an overlay image that, when combined with the optical see-through view, enables the eye looking through the combiner unit to perceive the dark areas of the scene, where, in bright areas, i.e. in areas of the real-world scene that an eye looking through the combiner unit can perceive, light is added using the overlay image, so that the radiance level of the bright areas in the combined image, i.e. the overlay image combined with the optical see-through view, is higher than or equal to the radiance level of the dark areas.

14. Method to provide to augment visual perception using a device according to one of the claims 1 to 13.

Images