Augmenting human vision in environments with limited visibility

The mixed-reality device enhances human vision in limited visibility environments by combining sensors to capture and display edge representations beyond human vision range, addressing misalignment issues in dynamic conditions and providing a clear, undistorted view of obscured objects.

JP2024543800AInactive Publication Date: 2024-11-26TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024524487
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-12-03
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing augmented reality solutions struggle to dynamically adapt to changing visibility conditions in environments with limited visibility, leading to misalignment between displayed virtual content and the user's actual field of view, especially in dynamic environments like dense smoke or water vapor.

Method used

A mixed-reality device that combines a visibility sensor capturing a range commensurate with human vision and an extended range sensor, processing circuitry to generate edge representations, and a display to overlay delta edges not visible to human vision, enhancing visibility by displaying edges beyond human vision range.

Benefits of technology

Augments human vision by providing a more natural view of the environment by overlaying edges of obscured objects, minimizing digitally generated content, and maintaining a clear, undistorted view of the real-world scene.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543800000001_ABST
    Figure 2024543800000001_ABST
Patent Text Reader

Abstract

A mixed reality (MR) apparatus for augmenting human vision in a limited visibility environment (e.g., due to smoke 103) is provided. The MR device includes a visibility sensor for capturing a visibility range representation of a real-world scene surrounding the MR device at a range commensurate with human vision, an extended range sensor for capturing an extended range representation of the real-world scene at an extended range greater than the range commensurate with human vision, a display, and a processing circuit. The processing circuit operates the MR device to generate a visibility range representation of edges of one or more physical objects (104) present in the real-world scene by extracting edges from the visibility range representation captured by the visibility range sensor, generate an extended range representation of edges of the one or more physical objects (104) by extracting edges from the extended range representation captured by the extended range sensor, generate a delta representation (shown in bold in FIG. 2D ) of edges of the one or more physical objects (104) that are present in the extended range representation of edges but not in the visibility range representation of edges, and display the delta representation of edges on the display.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a Mixed-Reality (MR) device for augmenting human vision in limited visibility environments, a method for augmenting human vision in limited visibility environments, a corresponding computer program, a corresponding computer-readable data carrier and a corresponding data carrier signal. [Background technology]

[0002] Augmented reality (AR) and mixed reality (MR) solutions are known to use cameras, lidar, and other sensors to detect real-world objects in a real-world scene, i.e., a physical environment, surrounding a user wearing a head-mounted display (HMD) headset, and to display such objects as an overlay on the real-world scene displayed by the user. Such HMDs typically include a see-through display through which the user can see the real-world scene, and on which virtual content is displayed as an overlay on the real-world scene.

[0003] As an example, Qwake Technologies' C-THRU helmet (https: / / www.qwake.tech / , https: / / www.behance.net / gallery / 6579685 / C-Thru-Smoke-Diving-Helmet) was developed as a tool for firefighters operating in environments with poor or limited visibility, such as heavy smoke. The C-THRU helmet uses a thermo-optical (infrared) camera to capture images of the real-world scene surrounding the user in the infrared range, which is less affected by smoke compared to the visible range. Thus, compared to human vision, which relies on visible light, an infrared camera can detect objects at an increased or extended range that would be invisible to a user relying solely on human vision, for example due to smoke.

[0004] The C-THRU helmet relies on image processing of the video or image sequence captured by the infrared camera to enhance the edges or contours (a contour is an edge that surrounds an object; if an edge defines an object, it's a contour) of objects captured in the video and generate a wireframe image of the edges. The edges are then overlaid onto the real-world scene that the user sees through the C-THRU helmet's see-through display.

[0005] Human vision is generally superior in terms of resolution, wide field of view, etc., and is often superior to digitally captured and displayed content. In situations where visibility is limited, such as heavy smoke, as well as thick water vapor, darkness, etc., solutions such as the C-THRU helmet may be advantageously used to enhance human vision, thereby extending the range at which objects can be viewed to a human user of an AR headset.

[0006] However, because visibility may change dynamically, e.g., while a user is moving through a smoky environment, or because smoke itself is not static and its density may change over time, displaying real-world objects as overlaid virtual content without regard to their actual current visibility within the user's field of view (FoV) may lead to situations where the user's natural (human visual) view of the real world is obstructed or obstructed. For example, this may be the case when real-world objects that the user can actually see are displayed as virtual content and overlaid on the user's (human visual) view of the real-world scene. Summary of the Invention

[0007] It is an object of the present invention to provide an improved alternative to the above and prior art techniques.

[0008] More specifically, it is an object of the present invention to provide an improved solution for enhancing human vision in environments with limited visibility.

[0009] These and other objects of the invention are achieved by different aspects of the invention as defined by the independent claims.Embodiments of the invention are characterized by the dependent claims.

[0010] According to a first aspect of the present invention, there is provided a mixed reality (MR) device for augmenting human vision in a limited visibility environment. The MR device comprises a visibility range sensor for capturing a visibility range representation of a real world scene surrounding the MR device. The visibility range sensor is configured to capture the visibility range representation of the real world scene at a range commensurate with the range of human vision. The MR device further comprises an extended range for capturing an extended range representation of the real world scene. The extended range sensor is configured to capture the extended range representation of the real world scene at an extended range greater than the range commensurate with human vision. The MR device further comprises a display and a processing circuit. The processing circuit operates the MR device to generate a visibility range representation of edges of one or more physical objects present in the real world scene. The visibility range representation of the edges of the one or more physical objects is generated by extracting edges from the visibility range representation captured by the visibility range sensor. The MR device is further operative to generate an extended range representation of edges of the one or more physical objects. The extended range representation of the edges of the one or more physical objects is generated by extracting edges from the extended range representation captured by the extended range sensor. The MR device is further operative to generate a delta (difference) representation of edges of one or more physical objects, the delta (difference) representation of the edges including edges that are present in the extended range representation of the edges but not in the visible range representation of the edges, and the MR device is further operative to display the delta (difference) representation of the edges on a display.

[0011] According to a second aspect of the present invention, there is provided a method for enhancing human vision in a limited visibility environment. The method is performed by an MR device and includes capturing a view range representation of a real world scene surrounding the MR device. The view range representation of the real world scene is captured at a range commensurate with human vision. The view range representation of the real world scene is captured using a view range sensor configured in the MR device. The method further includes capturing an extended range representation of the real world scene. The extended range representation of the real world scene is captured at an extended range that is greater than a range commensurate with human vision. The extended range representation of the real world scene is captured using the extended range sensor configured in the MR device. The method further includes generating a view range representation of edges of one or more physical objects present in the real world scene. The view range representation of the edges of the one or more physical objects is generated by extracting edges from the view range representation captured by the view range sensor. The method further includes generating an extended range representation of edges of the one or more physical objects. The extended range representation of edges of the one or more physical objects is generated by extracting edges from the extended range representation captured by the extended range sensor. The method further includes generating a delta representation of edges of the one or more physical objects, the delta representation of the edges of the one or more physical objects including edges that are present in the extended range representation of the edges but not in the visible range representation of the edges, and displaying the delta representation of the edges on a display of the MR device.

[0012] According to a third aspect of the invention there is provided a computer program comprising instructions which, when the computer program is executed by a MR device, cause the MR device to perform a method according to an embodiment of the second aspect of the invention.

[0013] According to a fourth aspect of the invention there is provided a computer readable data carrier having stored thereon a computer program according to the third aspect of the invention.

[0014] According to a fifth aspect of the present invention there is provided a data carrier signal, the data carrier signal carrying a computer program according to the third aspect of the present invention.

[0015] Although advantages of the present invention have sometimes been explained with reference to embodiments of the first aspect of the invention, corresponding reasons also apply to embodiments of the other aspects of the invention.

[0016] Further objects, features, and advantages of the present invention will become apparent upon review of the following detailed disclosure, drawings, and appended claims. Those skilled in the art will appreciate that different features of the present invention can be combined to create embodiments other than those described below. [Brief description of the drawings]

[0017] The above, as well as additional objects, features and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of embodiments of the present invention, taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 illustrates an MR device for augmenting human vision in limited visibility environments according to an embodiment of the present invention. [Figure 2A] FIG. 2 shows a visualization of the edge representation as well as the processed representation according to an embodiment of the present invention. [Figure 2B] FIG. 2 shows a visualization of the edge representation as well as the processed representation according to an embodiment of the present invention. [Figure 2C] FIG. 2 shows a visualization of the edge representation as well as the processed representation according to an embodiment of the present invention. [Figure 2D] FIG. 2 shows a visualization of the edge representation as well as the processed representation according to an embodiment of the present invention. [Diagram 3] FIG. 3 illustrates one embodiment of a processing circuit included in an MR device for augmenting human vision in limited visibility environments. [Figure 4] 4 is a flow chart illustrating a method for enhancing human vision in limited visibility environments according to an embodiment of the present invention. All figures are schematic, not necessarily to scale, and generally show only parts necessary to explain the present invention, other parts may be omitted or merely suggested. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention will now be described more fully herein with reference to the accompanying drawings, in which specific embodiments of the invention are shown. However, the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0019] In the following, an embodiment of a Mixed-Reality (MR) device 100 for augmenting human vision in a limited visibility environment will be described with reference to FIG.

[0020] The MR device 100 may be a handheld device, such as, for example, a mobile phone, a smartphone, a digital camera, a tablet, etc. Alternatively, the MR device 100 may be a head-mounted device (HMD) similar to that shown in Fig. 1. As a further alternative, the MR device 100 may be a helmet-mounted device, i.e., a device integrated into a helmet, such as a firefighter helmet or a rescue worker helmet, similar to the C-THRU helmet by Qwake Technologies.

[0021] In the present context, limited visibility environments are to be understood as environments, both indoor and outdoor, where human visibility is hindered, for example by smoke, steam (gaseous water) or low light conditions, resulting in low or limited visibility for humans operating in such environments, such as firefighters and rescue personnel.

[0022] Referring to FIG. 1, the MR device 100 comprises a visibility range sensor 111 for capturing a visibility range representation of a real-world scene 101 surrounding the MR device 100. The real-world scene 101 is an indoor or outdoor location where a user 102 of the MR device 100 is operating. The visibility range sensor 111 operates to capture a visibility range representation of the real-world scene 101 in a range commensurate with human vision. In other words, the visibility range representation of the real-world scene 101 captured by the visibility range sensor 111 represents what the user 102 of the MR device 100 can see depending on human vision. This can be achieved, for example, by using a visibility range sensor 111 that operates to capture a visibility range representation of the real-world scene 101 using wavelengths within the visible range. Typically, the human eye can detect wavelengths from about 380 nanometers to about 720 nanometers, i.e., in the range of the visible spectrum. The field of view (FoV) of the visibility range sensor 111 is illustrated as 114 in FIG. 1.

[0023] The MR device 100 shown in FIG. 1 further comprises an extended range sensor 112 for capturing an extended range representation of the real-world scene 101. The extended range sensor 112 operates to capture an extended range representation of the real-world scene 101 at an extended range that is wider than the range commensurate with human vision, i.e., that extends to a greater distance from the sensor 112. This may be achieved, for example, by employing an extended range sensor 112 that operates to capture an extended range representation of the real-world scene 112 using wavelengths at least partially outside the visible spectrum. The wavelength range of the extended range sensor 112 may be exclusively outside the visible spectrum, i.e., below about 380 nm or above about 720 nm, but may alternatively include at least a portion of the visible range. The wavelength range of the extended range sensor 112 may span wavelengths lower than the visible range, higher than the visible range, or both. For example, the extended range sensor 112 may be an infrared light sensor, such as an infrared light camera (also known as an infrared camera, thermal imaging camera, or simply a thermal camera), that typically operates at wavelengths greater than 1000 nm. The use of an extended range sensor 112 operating in infrared light is advantageous in that infrared light is less affected by smoke compared to sensors operating in the visible spectrum. For simplicity, in FIG. 1, the extended range sensor 112 is illustrated as having the same FoV 114 as the visible range sensor 111. In practice, different types of sensors typically have different FoVs, and alignment of the sensors 111 and 112, and alignment or transformation of the respective representations captured by the sensors 111 and 112 may be required. For purposes of describing the embodiments of the invention described herein, it is assumed here that the FoV 114 illustrated in FIG. 1 is a common FoV for the visible range sensor 111 and the extended range sensor 112.

[0024] It is understood throughout this disclosure that the representation of the real-world scene 101 is data captured by sensors such as the visible range sensor 111 or the extended range sensor 112, or data derived from captured sensor data, and may be displayed to the user 102, optionally after processing, to visualize the real-world scene 101 similar to that illustrated in FIG. 2. For example, the representation of the real-world scene 101 may be in the form of a digital image, or a sequence of images, i.e., a video. The video may be either based on a two-dimensional (2D) format or in a three-dimensional (3D) format, e.g. including depth information. As another example, the 3D representation of the real-world scene 101 may be in the form of a point cloud.

[0025] The visible range sensor 111 may be, for example, a digital camera operating in the visible spectrum. Correspondingly, the extended range sensor 112 may be a digital camera operating at least partially at wavelengths outside the visible spectrum, such as, for example, an infrared camera. If the visible / extended range sensor 111 / 112 is a digital camera, the corresponding representation of the real-world scene 101 is captured as a video, i.e., a sequence of images. The digital camera 111 / 112 may be a monocular camera operating to capture a 2D representation of the real-world scene 101, or a stereo camera operating to capture a 3D representation of the real-world scene 101.

[0026] The extended range sensor 112 may alternatively be a lidar or radar sensor. In this case, the extended range representation is a 3D representation of the real-world scene 101, for example in the form of a point cloud. Lidar sensors operate at infrared wavelengths, relying on infrared laser beams being reflected by objects. Radar sensors, on the other hand, operate at wavelengths shorter than the visible spectrum, relying on radio waves being reflected by objects. Both lidar and radar rely on time-of-flight measurements to determine the distance of an object to the sensor.

[0027] Still referring to FIG. 1, the MR device 100 further comprises a display 113. The display 113 is arranged to be viewable by the user 102 and may be used to display visual content to the user 102, including a visualization of a representation of the real-world scene 101, as illustrated in FIG. 2. The display 113 may be, for example, a see-through display, as illustrated in FIG. 1. Typically, this is the case when the MR device 100 is embodied as an HMD or helmet-mounted device (either monocular or binocular, i.e., the display 113 is arranged to be viewed by one eye or both eyes, respectively). The see-through display 113 allows the user 102 to directly view the real-world scene 101, relying on human vision. Alternatively, the display 113 may be an opaque non-see-through display. Typically, this is the case when the MR device 100 is embodied as a handheld device, such as a mobile phone, a smartphone, a digital camera, a tablet, etc.

[0028] The MR device 100 further comprises a processing circuit 120, which is described in more detail below with reference to Figure 3. The processing circuit 120 operates the MR device 100 to generate a visibility range representation of edges of one or more physical objects 104 present in the real-world scene 101. The visibility range representation of the edges of the one or more physical objects 104 may be obtained by extracting edges from the visibility range representation captured by the visibility range sensor 111.

[0029] The one or more physical objects 104 may be, for example, furniture (tables, chairs, cupboards, shelves, etc.), walls, windows, doors, stairs, human or animal bodies, debris, vehicles, trees, pipes, etc.

[0030] The processing circuit 120 further operates the MR device 100 to generate an extended range representation of edges of the one or more physical objects 104. This may be obtained by extracting edges from the extended range representation captured by the extended range sensor 112.

[0031] In this context, an "edge" refers to the boundary of a real-world object, such as object 104. An edge that surrounds or defines an object is commonly referred to as a "contour". Throughout this disclosure, "edge" and "contour" are used synonymously. Depending on the visibility and capabilities of the visible range sensor 111 or the extended range sensor 112, respectively, a portion or only a part of an edge may be captured by the sensor. A representation of an edge should be understood as a representation that includes the edge or a part thereof as seen by the sensor 111 / 112 that captured the representation of the real-world scene 101 from which the respective edge representation is derived.

[0032] In FIG. 2, an exemplary visualization 210 of a visibility range representation of edges is shown, derived from a visibility range representation of the real-world scene 101 captured by the visibility range sensor 111. As can be seen in the visualization 210, the visibility range sensor 111 captures edges, or portions thereof, of one or more physical objects 104 that are not obscured by the smoke 103. For simplicity, it is assumed here that the smoke 103 has a well-defined boundary, rather than a smooth transitional boundary where visibility gradually decreases. Also shown in FIG. 2 is an exemplary visualization 220 of an extended range representation of edges, derived from an extended range representation of the real-world scene 101 captured by the extended range sensor 112. As can be seen in the visualization 220, the extended range sensor 112 captures edges, or portions thereof, of one or more physical objects 104 at a greater range than is visible in the visualization 210 of the visibility range representation of edges, including edges, or portions thereof, that are not visible in the visualization 210 of the visibility range representation of edges. This is because the extended range sensor 112 has an extended range compared to the visible range sensor 111 and the sensor 112 is less affected by the smoke 103 .

[0033] Edges can be detected by finding boundaries of objects in an image or other type or captured representation of a real-world object, as known in the art. In the case of an image captured by a digital camera, edges can be detected, for example, as discontinuities in brightness. Digital images that have been image-processed to the extent that they represent only the edges of the captured object are also known as wireframe images and have the appearance of uncluttered line drawings. As used herein, the term "detect edges" in "extract edges" should be understood to include enhancing edges in the captured visible / extended range representation to the extent that it includes only the edges of the captured object.

[0034] As an example of generating a wireframe image, see US10,896,492B2 which discloses enhancing edges of objects in a thermal image by generating a gradient magnitude image consisting of a plurality of pixels with associated gradient magnitude values, dividing the gradient magnitude image into subregions, computing gradient magnitude statistics for each subregion, and computing mapping parameters for each subregion that equalize and smooth the dynamic range of the corresponding gradient magnitude statistics across the subregions. The calculated mapping parameters are then applied to the pixels in the subregions to generate enhanced gradient magnitude values ​​with equalized brightness and contrast, resulting in a wireframe image.

[0035] Edge detection in 3D point clouds is also known in the art. As an example, "Geometric Edge Description and Classification in Point Cloud Data with Application to 3D Object Recognition" (by TB Joergensen, AG Buch, and D. Kraft, in Proceedings of the 10th International Conference on Computer Vision Theory and Applications (VISAPP-2015), pages 333-340, SCITEPRESS, 2015) proposes the application of supervised learning techniques to shape descriptors for local point cloud features and introduces shape descriptors to capture local surface properties near edges.

[0036] It is also possible to merge 2D digital images and 3D point clouds for edge detection, as described for example in "Edge Extraction by Merging 3D Point Cloud and 2D Image Data" by Y. Wang, D. Ewert, D. Schilberg, and S. Jeschke, in 2013 10th International Conference and Expo on Emerging Technologies for a Smarter World (CEWIT), pages 1-6, IEEE, 2013. The authors describe a solution based on combining edge data from an object's point cloud with its corresponding digital image. First, edge extraction of the 2D image is performed using the Canny edge detection algorithm. A pixel data mapping mechanism is proposed to establish the correspondence between the 2D image pixels and the 3D point cloud pixels. By using the established correspondence map, the 2D edge data can be merged into the 3D point cloud, where edge extraction is performed.

[0037] The processing circuit 120 further operates the MR device 100 to generate a delta representation of edges of one or more physical objects 104. The delta representation of the edge includes edges that are present in the extended range representation of the edge (captured by the extended range sensor 112, as illustrated in visualization 220) but not present, i.e. not present, in the visible range representation of the edge (captured by the visible range sensor 111, as illustrated in visualization 210). In other words, since the visible range representation of the edge includes edges that are visible to the user 102 relying on human vision and the extended range representation of the edge includes edges that are captureable by the extended range sensor 112 that have a range that exceeds a range commensurate with human vision, the delta representation of the edge includes edges or portions thereof that are not visible when the user 102 relies only on human vision. In FIG. 2, an exemplary visualization 230 of the delta representation of the edge is shown, which corresponds to the difference between the extended range representation of the edge (visualization 220) and the visible range representation of the edge (visualization 210).

[0038] The MR device 100 may be operable to generate a delta representation of the edges of one or more physical objects 104, for example, by subtracting the visible range representation of the edges from the extended range representation of the edges. If the visible range representation of the edges and the extended range representation of the edges are digital images, the respective images may be subtracted as known in the art of image processing. Depending on the alignment and / or calibration of the visible range sensor 111 and the extended range sensor 112, it may be necessary to align the two images before performing the image subtraction so as to minimize misalignment of the sensors. This may be performed, for example, based on the edges contained in both images, typically the edges contained in the visible range representation. If the visible range representation of the edges and the extended range representation of the edges are point clouds, the delta representation of the edges of one or more physical objects 104 is generated by subtraction of the respective point clouds.

[0039] The processing circuit 120 further operates the MR device 100 to display the delta representation of the edges on the display 113. As shown in FIG. 1, if the display 113 is a see-through display, the delta representation of the edges is displayed as an overlay on the real-world scene 101 as seen by the user 102 looking at the display 113. Alternatively, if the display is an opaque non-see-through display, the MR device 100 further operates to display on the display 113 a visible range representation of the real-world scene 101 captured by the visible range sensor 111. In other words, the delta representation of the edges is displayed as an overlay on the visible range representation of the real-world scene 101. This means that both the visible range representation of the real-world scene 101, which represents what the user 102 can see relying on human vision, and the delta representation of the edges, including the edges of the objects 104 that the user 102 cannot see, are displayed. Preferably, the visible range representation of the real-world scene 101 is captured by a conventional visible light digital camera 111, providing the user 102 with a substantially undistorted view of the real-world scene 101, similar to what the user 102 would experience while viewing the real-world scene 101 through a see-through display 113.

[0040] An exemplary visualization 240 of what the user 102 can see when looking at the display 113 is shown in FIG. 2. The visualization 240 shows a delta representation of edges (visualization 230) overlaid on a view of the real-world scene 101 seen by the user 102 if the display 113 is a see-through display. Alternatively, if the display 113 is an opaque non-see-through display, the visualization 240 shows a delta representation of edges (visualization 230) overlaid on a view range representation of the real-world scene 101 (visualization 210) that represents "what the user can see". As shown in FIG. 2, the delta representation of edges (visualization 230) including edges of objects 104 that the user 102 cannot see may be displayed to emphasize and / or highlight the edges of objects 104 that the user 102 cannot see, for example, by displaying the edges as thicker lines. Alternatively, the delta representation of edges (visualization 230) may be displayed as lines of different colors.

[0041] Advantageously, by augmenting the human vision of the user 102 using the MR device 100 to view the real-world scene 101 with only the edges of objects that are not visible to the user 102 due to limitations of human vision, for example because part of the object 104 or a portion of it is obscured by smoke 103, a more natural view of the real-world scene 101 seen by the user 102 is achieved. In particular, this is the case when the display 113 is a see-through display where the user 102 can rely on human vision to view the real-world scene 101, which is often superior to digitally generated content. By overlaying only the edges of objects 104 that are not visible to the (human) eye, the amount of digitally generated or virtual content can be minimized.

[0042] Hereinafter, an embodiment of the processing circuit 120 configured in the MR device 100 will be described with reference to FIG.

[0043] The processing circuitry 120 may comprise one or more processors 121, such as a central processing unit (CPU), a microprocessor, an application processor, an application-specific processor, a graphics processing unit (GPU), and a digital signal processor (DSP), including an image processor, or a combination thereof, and a memory 122 including a computer program 123 comprising instructions. When executed by the processor 121, the instructions cause the MR device 100 to operate in accordance with the embodiments of the invention described herein. The memory 122 may be, for example, a random access memory (RAM), a read-only memory (ROM), a flash memory, etc. The computer program 123 may be downloaded to the memory 122 as a data carrier signal carrying the computer program 123 by means of a network interface circuitry 115 (not shown in FIG. 1 ) that may be included in the MR device 100. The processing circuitry 120 may alternatively or additionally comprise one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc., which operate to operate the MR device 100 in accordance with the embodiments of the invention described herein.

[0044] The network interface circuitry 115 may include one or more of a cellular modem (e.g., including GSM, UMTS, LTE, 5G, or higher generation, dedicated communications solutions for first responders, emergency personnel, military, law enforcement, etc.), a WLAN / Wi-Fi modem, a Bluetooth modem, an Ethernet interface, an optical interface, etc., for exchanging data between the MR device 100 and other MR devices, application servers, the Internet, etc.

[0045] In the following, an embodiment of a method 400 for augmenting human vision in limited visibility environments will be described with reference to FIG.

[0046] The method 400 is performed by the MR device 100 and includes capturing 401 a viewable range representation of a real-world scene 101 surrounding the MR device 100 at a range commensurate with human vision. The viewable range representation of the real-world scene 101 is captured using a viewable range sensor 111 configured on the MR device 100. The viewable range representation of the real-world scene 101 may be captured using, for example, wavelengths within the visible range. The viewable range sensor 111 may be, for example, a digital camera.

[0047] The method 400 further includes capturing 402 an extended range representation of the real-world scene 101 having an extended range greater than a range commensurate with human vision. The extended range representation of the real-world scene 101 is captured using an extended range sensor 112 included in the MR device 100. The extended range representation of the real-world scene 101 may be captured, for example, using wavelengths at least partially outside the visible range. For example, the extended range representation of the real-world scene 101 may be captured using infrared light. The extended range sensor 112 may be, for example, a digital camera. Alternatively, the extended range sensor 112 may be, for example, a lidar or radar sensor.

[0048] The method 400 further includes generating 403 a view range representation of edges of one or more physical objects 104 present in the real-world scene 101. The view range representation of the edges of the one or more physical objects 104 is generated by extracting edges from the view range representation captured by the view range sensor 111.

[0049] The method 400 further includes generating 404 an extended range representation of edges of the one or more physical objects 104. The extended range representation of the edges of the one or more physical objects 104 may be generated by extracting edges from the extended range representation captured by the extended range sensor 112.

[0050] The method 400 further includes generating 405 a delta representation of edges of the one or more physical objects 104. The delta representation of the edges includes edges that are present in the extended range representation of the edges but not in the visible range representation of the edges. The delta representation of the edges of the one or more physical objects 104 may be generated 405, for example, by subtracting the visible range representation of the edges from the extended range representation of the edges.

[0051] The method 400 further includes displaying 406 the delta representation of the edges on a display 113 included within the MR device 100. The display 113 may be a see-through display, and the delta representation of the edges may be displayed 406 as an overlay on the real-world scene 101 to be seen by a user 102 viewing the display 113. Alternatively, the display 113 may be an opaque display (i.e., a non-see-through display), and the method 400 may further include displaying 407 a viewable range representation of the real-world scene 101 on the display 113.

[0052] It will be understood that method 400 may include additional, alternative, or modified steps in accordance with what has been described throughout this disclosure.

[0053] An embodiment of the method 400 may be implemented as a computer program 123 including instructions that, when executed by a computing device such as the MR device 100, cause the MR device 100 to perform the method 400 and operate according to embodiments of the invention described herein. The computer program 123 may be stored on a computer-readable data carrier such as the memory 122. Alternatively, the computer program 123 may be transmitted by a data carrier signal, e.g., downloaded to the memory 122 via the network interface circuitry 115.

[0054] Those skilled in the art will understand that the invention is not limited to the above-described embodiments, but on the contrary, many modifications and variations are possible within the scope of the appended claims.

Claims

1. A mixed reality, MR device (100) for augmenting human vision in limited visibility environments, comprising: a visibility sensor (111) for capturing a visibility representation of a real-world scene (101) surrounding the MR device at a range commensurate with human vision; an extended range sensor (112) for capturing an extended range representation of the real-world scene (101) at a range greater than that commensurate with human vision; A display (113); A processing circuit (120) for the MR device, generating a visibility representation of edges of one or more physical objects (104) present in the real-world scene (101) by extracting edges from the visibility representation captured by the visibility sensor (111); generating an extended range representation of edges of the one or more physical objects (104) by extracting edges from the extended range representation captured by the extended range sensor (112); generating a delta representation of edges of the one or more physical objects (104), including edges that are present in the extended range representation of the edges but not in the visible range representation of the edges; displaying a delta representation of the edges on the display (113); A processing circuit (120) for operating the An MR device (100).

2. The MR device of claim 1 , operative to generate a delta representation of the edge of the one or more physical objects by subtracting a visible range representation of the edge from an extended range representation of the edge.

3. 3. The MR device (100) of claim 1 or 2, wherein the display (113) is a see-through display and the delta representation of the edge is displayed as an overlay on the real-world scene (101) to be seen by a user (102) viewing the display (113).

4. The MR device (100) of claim 1 or 2, wherein the display (113) is an opaque display, and the MR device is further operative to display the visible range representation of the real-world scene (101) on the display (113).

5. The MR device (100) of any one of claims 1 to 4, wherein the visible range sensor (111) is operative to capture the visible range representation of the real-world scene (101) using wavelengths within the visible range.

6. The MR device (100) of any one of claims 1 to 4, wherein the extended range sensor (112) operates to capture the extended range representation of the real-world scene (101) at least partially using wavelengths outside the visible range.

7. The MR device of claim 6 , wherein the extended range sensor is an infrared light sensor.

8. The MR device (100) of any one of claims 1 to 7, wherein at least one of the visible range sensor (111) and the extended range sensor (112) is a digital camera.

9. The MR device (100) of any one of claims 1 to 9, wherein the extended range sensor (112) is a lidar sensor or a radar sensor.

10. The MR device (100) according to any one of claims 1 to 9, wherein the MR device (100) is one of a handheld device, a head mounted device HMD, and a helmet mounted device.

11. A method (400) for augmenting human vision in a limited visibility environment, said method being performed by a mixed reality (MR) device, comprising: Using a visibility sensor included in the MR device, capturing (401) a visibility representation of a real-world scene (101) surrounding the MR device at a range commensurate with human vision; capturing (402) an extended range representation of the real-world scene (101) at an extended range greater than that commensurate with human vision using an extended range sensor included in the MR device; generating (403) a visibility representation of edges of one or more physical objects (104) present in the real-world scene (101) by extracting edges from the visibility representation captured by the visibility sensor; generating (404) an extended range representation of edges of the one or more physical objects (104) by extracting edges from the extended range representation captured by the extended range sensor; generating (405) a delta representation of edges of the one or more physical objects (104), including edges that are present in the extended range representation of the edges but not in the visible range representation of the edges; Displaying (406) the delta representation of the edge on a display included in the MR device; The method (400).

12. 12. The method (400) of claim 11, wherein a delta representation of the edge of the one or more physical objects (104) is generated by subtracting (405) a visible range representation of the edge from an extended range representation of the edge.

13. 13. The method (400) of claim 11 or 12, wherein the display is a see-through display and the delta representation of the edges is displayed (406) as an overlay on the real-world scene (101) to be seen by a user (102) viewing the display.

14. 13. The method (400) of claim 11 or 12, wherein the display is an opaque display, the method further comprising displaying (407) the viewable range representation of the real-world scene (101) on the display.

15. The method (400) of any one of claims 11 to 14, wherein the visible range representation of the real-world scene (101) is captured using wavelengths within the visible range.

16. The method (400) of any one of claims 11 to 14, wherein the extended range representation of the real-world scene (101) is captured at least in part using wavelengths outside the visible range.

17. 20. The method (400) of claim 16, wherein the extended range representation of the real-world scene (101) is captured using infrared light.

18. The method (400) of any one of claims 11 to 17, wherein at least one of the visible range sensor and the extended range sensor is a digital camera.

19. The method (400) of any one of claims 11 to 19, wherein the extended range sensor is a lidar sensor or a radar sensor.

20. A computer program (123) comprising instructions, which when executed by a mixed reality, MR device (100) causes the MR device (100) to perform the method according to any one of claims 11 to 19.

21. A computer readable data carrier (122) having stored thereon a computer program (123) according to claim 20.

22. A data carrier signal carrying a computer program (123) according to claim 20.

Citation Information

Patent Citations

  • Dust measurement device and dust measurement method

    JP2021156824A

  • Infrared resolution and contrast enhancement with fusion

    US20110261207A1

  • Visible light image with edge marking for enhancing IR imagery

    US20150269742A1

  • Dust measurement device

    WO2020251069A1