User-installable display system and method
The user-wearable display system addresses the inconvenience of single-depth focus by presenting virtual images at two focal planes, enhancing user experience through efficient switching between virtual and real-world objects.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- BAE SYSTEMS PLC
- Filing Date
- 2024-04-30
- Publication Date
- 2026-06-25
AI Technical Summary
Existing head-mounted displays present virtual images at a single specific depth of focus, requiring users to refocus their viewpoint when switching between virtual and real-world objects, which is inconvenient and inefficient.
A user-wearable display system with two fixed depth zones, utilizing two imaging devices and a processing unit to generate and distribute virtual images at different focal planes, allowing simultaneous presentation of images at a closer and farther depth, thereby reducing the need for refocusing.
Enables users to view virtual images at two focal planes, improving user experience by seamlessly overlaying virtual images on real-world objects without the need for continuous refocusing.
Smart Images
Figure 2026520847000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a user-wearable display system and a method of providing virtual images to a user of such a system.
Background Art
[0002] Head-mounted displays that present virtual images to a user are known. These virtual images can be presented to the user by projecting light onto a semi-reflective visor or an eyepiece such that the virtual images appear superimposed on the surrounding environment being viewed by the user.
[0003] Typically, such virtual images (also referred to as augmented images in this context) are presented at a single specific depth of focus in the surrounding scene.
[0004] A head-mounted display system monitors the surrounding scene viewed by the user, processes the associated data, and thus the associated display device can present the virtual images appropriately.
Summary of the Invention
[0005] According to one aspect of the present invention, there is provided a user-wearable display system, a display device for displaying images at a first fixed depth and a second fixed depth, the display device having a display field of view (FOV), a distance range device for generating ambient image data (AID, CD1, CD2) over a distance range field of view, a processing unit, receiving the ambient image data, generating distance range data using the ambient image data, a first zone of the surrounding scene for display of an image at the first depth, and a second zone of the surrounding scene for display of an image at the second depth To determine this, use distance range data and The process involves receiving virtual image data (VIDs) that represent at least two images for display as a virtual image, From the virtual image data (VID), determine the virtual image associated with a first zone or a first depth, and distribute such first depth image to a display device for display at the first depth. From the virtual image data (VID), determine the virtual image associated with a second zone or a second depth, and distribute such second depth images to the display device for display at the second depth. To process virtual image data (VID), A processing unit configured to perform the following: A display system is provided that includes the following features.
[0006] Such a system could allow the user to view virtual images at two focal planes, and by matching the virtual images to the appropriate focal plane for the surrounding scene (for example, by overlaying a virtual speedometer onto a real-world dashboard), it could reduce the user's need to refocus their viewpoint as a switch between virtual and real-world objects.
[0007] Furthermore, such a system provides the generation of depth maps that can be used to lay up virtual images appropriately and quickly.
[0008] A distance range device may comprise a first imaging device configured to image a first portion (ABC) of the surrounding scene, and a second imaging device configured to image a second portion (DEF) of the surrounding scene, wherein a common area (DBG) of the surrounding scene is imaged by both the first and second imaging devices. Typically, each imaging device may be a camera.
[0009] The first and second imaging devices can simultaneously generate two respective subsets of ambient image data (CD1, CD2) for use in generating distance range data.
[0010] Generating distance range data may involve using parallax determinations.
[0011] The display field of view (FOV) can be narrower than the distance-range FOV.
[0012] The processor unit may be configured to perform object recognition on ambient image data (AID), and in particular to identify a given zone or object as a first or second zone.
[0013] The processor unit may be configured to perform object recognition on ambient image data (AID) to identify objects within a common area and to determine a distance range, with a known separation between the first imaging device and the second imaging device being geometrically given.
[0014] The display device may include a first image source configured to generate optical signals carrying an image for display at a first depth, and a second image source configured to generate optical signals carrying an image for display at a second depth, wherein processing virtual image data (VID) in a processing unit includes distributing to the first image source virtual images associated with a first zone of the surrounding scene or associated with a first depth, and distributing to the second image source virtual images associated with a second zone of the surrounding scene or associated with a second depth.
[0015] The first zone can correspond to a closer depth than the second zone. In particular, the closer depth may be within a distance range of 50 cm to 200 cm. Furthermore, the second region can correspond to a focal length of infinity.
[0016] The system is in communication with a processor unit and may further include a virtual image database for generating VIDs, wherein the virtual image database comprises multiple image datasets, each image dataset comprising virtual images enumerated with respect to data identifying one or more zones suitable for overlays on the virtual image, or data identifying one or more depths of focus suitable for overlays on the image.
[0017] The user-mountable display system may be for use in a vehicle or platform having a user workstation / cockpit and associated control / instrument console, where the first zone generally corresponds to the console and the first depth generally corresponds to the distance between the user and the console.
[0018] Furthermore, the user-installable display system may be for use in a workstation / cockpit and a vehicle or platform with a canopy / windshield, and the second zone generally corresponds to the canopy / windshield.
[0019] According to a second aspect, a method for providing a virtual image to a user of a user-worn display system, comprising: a display device having a display field of view (FOV) and for displaying an image at at least first and second fixed depths; and a distance range device for generating ambient image data (AID) over a distance range field of view (FOV), This method is performed by the processor, Receiving surrounding image data, Generating distance range data using surrounding image data, A first zone of the surrounding scene for displaying the image at a first depth, and Second zone of the ambient scene for displaying the image at the second depth. To determine this, use distance range data and Receiving virtual image data (VID) representing at least two images for display as a virtual image, Determining, from the virtual image data (VID), a virtual image associated with a first zone of the surrounding scene or associated with a first depth (6), and distributing such a first depth image to a display device for display at the first depth, and Determining, from the virtual image data (VID), a virtual image associated with a second zone of the surrounding scene or associated with a second depth, and distributing such a second depth image to the display device for display at the second depth, Processing the virtual image data (VID), A method is provided that includes.
[0020] Here, embodiments of the present invention are described by way of example only with reference to the drawings.
Brief Description of the Drawings
[0021] [Figure 1] FIG. 1 shows a display device. [Figure 2] FIG. 2 shows a display system including the display device of FIG. 1. [Figure 3] FIG. 3 shows a further view of the display system. [Figure 4] FIG. 4 shows the mapping of the surrounding scene to a far field zone and a near field zone. [Figure 5] FIG. 5 shows object comparison using two camera feeds.
Embodiments for Carrying Out the Invention
[0022] Referring to FIG. 1, an exemplary display device 100 should be described.
[0023] The display device 100 includes a first image source 1 and a second image source 2.
[0024] Each of these image sources can generate and output optical signals s1 and s2 that carry images, such as virtual images. These signals are output to the combiner element 3.
[0025] The first optical output signal s1 is associated with a first, closer depth of field 6, and the second optical output signal s2 is associated with a second depth of field 7, which is different from the first depth of field. This difference can be achieved in many different ways. For example, the standoff distance between each image source and the combiner element 3 may be different. Alternatively or additionally, the focal length may be determined by the image source and the output light produced.
[0026] In this embodiment, the first and second focal lengths are fixed. The closer depth of field can be configured to be 0.5 to 2 meters in front of the user. The second depth of field can be configured to infinity.
[0027] The first image source 1 is positioned approximately perpendicular to the second image source 2.
[0028] Combiner element 3 is semi-reflective and semi-transmissive, and is configured to receive light output from the first and second image sources. Combiner 3 is tilted at 45 degrees to the output light from each of the image sources. (In alternative embodiments, other forms of combiner elements that are partially reflective and partially transmissive may be used.)
[0029] As shown, light from the first image source 1 passes through the combiner element 3, while light from the second image source 2 is reflected at a 90-degree angle. Therefore, the light signal s1 output by the first image source 1 is combined with the light signal s2 from the second image source 2 in the semi-reflective combiner element 3.
[0030] Therefore, there is an output of a combined light beam from the combiner element 3. This combined light beam is received by a relay optical system 4, which is a set of lenses arranged in series to adjust the combined beam. Proper adjustment of the beam, and therefore the proper configuration of the relay optical system 4, would be obvious to a skilled optical designer.
[0031] The relay optical system 4 outputs the combined regulated light to a second combiner element 5 that is partially reflective and partially transmissive. Typically, the combiner element 5 will be incorporated into a visor or eyepiece or a pair of eyepieces for positioning within the user's field of view.
[0032] The combiner element 5 is configured to receive the combined regulated light and reflect at least a portion of it to the user's eyes. As shown in this example, the user's boresight view is perpendicular to the output from the relay optics 4, and the combiner element is tilted at 45 degrees relative to the output from the relay optics 4 and positioned on the user's boresight view.
[0033] Therefore, the virtual image carried by the optical signal s1 output by image source 1 and the virtual image carried by the optical signal s2 output by image source 2 are presented to the user.
[0034] The second combiner 5 is partially reflective and partially transparent. Therefore, the user can see a virtual image superimposed on the view around the user.
[0035] Given the respective depths of focus of optical signals (s1, s2) and the virtual images they carry, the user perceives the virtual image at one of two depths of focus, namely, a closer depth of focus 6 and a farther depth of focus 7. In this embodiment, the first image source 1 generates a virtual image for the closer depth of focus 6, while the second image source 2 generates a virtual image for the farther depth of focus 7.
[0036] Figures 2 and 3 show an example display system 200 that utilizes the display device 100. (Not all components of this system are shown in Figure 3 for clarity.)
[0037] Furthermore, real-world objects R, T, and S, and the user's eyeball are shown.
[0038] The system 200 includes a display device 100, a virtual image database 40, a processing unit 50, and a camera device 10.
[0039] The virtual image database 40 electronically stores many virtual images (43a, b, ..., n) and associated metadata. In particular, each virtual image may be enumerated as an image dataset 42 alongside a specific target real-world object (T, R, S, ..., m) and / or a specific real-world region (302, 304, ..., p) and / or a specific depth of field (near, far, ..., q). A virtual image data signal (VID) is output from the virtual image database 40 to the processing unit 50.
[0040] The camera device 10 may consist of one or more cameras. The camera device 10 is configured to generate ambient imaging data (AID) by viewing essentially the same surrounding scene as the user.
[0041] The processing unit 50 includes an image-to-display mapping module 52 and an image processing module 56. The processing unit 50 is operably connected to a virtual image database 40 and a camera device 10, and as a result can receive an image dataset 42 and an AID, respectively. Furthermore, the processing unit 50 is operably connected to both a first image source 1 and a second image source 2, and as a result can address signals carrying appropriate images to each.
[0042] The image-to-display mapping unit 52 includes a transformation submodule 53, which can be used to apply scaling, rotation, or skewing to the virtual image.
[0043] The image processing module 56 includes an image recognition submodule 57 and a distance measurement submodule 58.
[0044] As shown in Figure 3, the camera device 10 includes a left camera 22 and a right camera 24. The left camera generates first camera data (CD1), and the right camera generates second camera data (CD2). The combined CD1 and CD2 represent AID.
[0045] The display system 200 is at least partially mounted on a head-mounted structure, such as a mount structure or frame 26 having the form of a pair of glasses or goggles. Thus, the mount tends to include arms for resting on the user's ears and a bridge for resting on the user's nose, linked by members to which eyepieces may be attached. (Other head-mounted structures are conceivable, including helmet-mounted structures).
[0046] The left and right cameras 22 and 24 are mounted on the left and right outermost parts of the mounting structure 26, separated by a dimension of 500. This mounting structure houses the second combiner 5, which is shown as a pair of eyepieces, one for each eye. In an alternative embodiment, the second combiner may be a single visor member.
[0047] The eyepiece is located on a mount 26 between the left camera 22 and the right camera 24. The left camera 22 defines a separation 501 between itself and the left eye. The right camera 24 defines a separation 502 between itself and the right eye.
[0048] The mounting structure 26 is positioned such that when it is attached to the user's head, the combiner 5 is positioned above the user's eyes.
[0049] As shown in Figure 3, the left camera 22 has a field of view ABC. The right camera 24 has a field of view DEF. The fields of view of the left and right cameras overlap at their common area BDG. The point in the common area closest to the user is point G. The system is configured to have a minimum separation between point G and the user, thereby substantially covering the user's field of view.
[0050] Furthermore, the user's left eye has a visual field IHK, and the right eye has an IHL. There is overlap in the IHJ region where the user will have binocular vision.
[0051] Figure 4 shows a view of an ambient scene in which the display system 200 may be used. In particular, Figure 4 shows an ambient scene that a user can see while sitting in a car. This scene has a clear interior zone (including the dashboard, drive wheels, rearview mirror, and windshield frame) and a clear exterior view (including the road and roadside). This ambient scene can be converted into a map 300 comprising a near-field zone 302 and a far-field zone 304. For the near-field zone 302, a near-field display of virtual images is preferred, and for the far-field zone 304, a far-field display of virtual images is preferred.
[0052] During operation, the display system 200 recognizes objects (e.g., R, T, S) or zones (304, 302) in the scene, and can then match predetermined virtual objects to each object or zone according to predetermined rules. In particular, it is provided that some virtual images should be presented at a near focal length, and other virtual images should be presented at a far focal length.
[0053] As an example of operation, the user wears a display system 200 and can view the surrounding scene. Objects T, S, and I exist in the scene.
[0054] (Alternatively, the scene can be divided into separate zones, each with a distinct focal length, that are predefined. Figure 4 shows a configuration in which the vehicle's cockpit / dashboard represents the first zone, i.e., the near-field map 302, and the external scene represents the second zone, i.e., the far-field map 304.)
[0055] When viewing the surrounding scene, the user points cameras 22 and 24 at the scene, and imaging data (AID) is generated by the cameras and sent to the processing unit 50.
[0056] The imaging data (AID) is received by the processing unit 50 and transmitted to the image processing module 56. In the image processing module 56, the AID is used by the image recognition module 57, which scans the data for objects or zones of interest. Such zones or objects are generally predefined by the intended use of the system.
[0057] As a result of such scanning, the image recognition module 57 may generate signals indicating the presence (e.g., yes or no) and direction (e.g., as bearing) of objects (or zones) in the scene.
[0058] Furthermore, the distance measuring module 58 may use AID to determine the distance to a recognized object or zone. Such distance measurement may be performed using the geometric techniques of standard rangefinders, parallax determination, or alternative methods (see the description in Figure 5 below).
[0059] Therefore, as a result of the image processing module 56 using AID, the processing unit 50 may generate a signal indicating the presence and location (e.g., orientation and distance range) of a specific object or zone.
[0060] The processor unit 50 can address this presence / location signal to the image-to-display mapping module 52. The mapping module 52 uses the presence / location signal to select any appropriate virtual image to be associated with the object / zone, referencing the virtual image database 40.
[0061] Furthermore, the mapping module 52 uses the presence / position signal to determine the depth of focus for the virtual image.
[0062] Once a desired depth of field for the virtual image is determined, taking into account the identified object or zone, the processing unit 50 may address the virtual image to the associated image source 1 or 2 as an appropriate signal.
[0063] As an example of a situation in which a user is controlling a vehicle, the system may be pre-configured such that the speedometer reading should be presented as a virtual image 43a on the dashboard at a near focal length 6, and a directional arrow (e.g., for navigation) should be presented as a virtual image 43b appearing in the center of the windshield at a far focal length 7.
[0064] Therefore, once the image processing module 56 recognizes the dashboard in AID, the presence / position signal is then used by the mapping module 52 to select a virtual image 43a of the speedometer and address this virtual image 43a to the near-depth projector 1.
[0065] Furthermore, once the image processing module 56 recognizes the windshield in AID, the presence / position signal is then used by the mapping module 52 to select a directional arrow virtual image 43b and address this directional arrow virtual image 43b to the depth projector 2.
[0066] Figure 5 illustrates the steps in a process for determining the distance range of an object, which may be used with System 200.
[0067] Box 522 represents an image captured by the left camera 22 at a given moment (thus, box 522 represents camera data CD1). Box 425 represents an image captured by the right camera 24 at the same moment (thus, box 524 represents camera data C2).
[0068] Images 522 and 524, which exist in the surrounding scene and are associated with the same time, are each object S.
[0069] However, object S is relatively close to imaging devices 22 and 24, which are separated by the separation 500. Therefore, the position of object S is different in each of images 522 and 524. Thus, an offset 530 is defined that represents the discrepancy between the cameras of the close object.
[0070] This can present a dilemma regarding wider imaging systems when deciding where to lay up a specific virtual image, which should be superimposed on object S, within the user's field of view and on the display.
[0071] However, if the offset between each image of object S is determined (for example, by superimposing images 522 and 524 and counting the intermediate pixel values), this offset can be used, for example, by using a lookup table to estimate a specific value for the distance range to object S.
[0072] Furthermore, the position of S may be taken as the average position of S between two images for the purpose of localizing any related virtual image. (This assumes that the left and right cameras are mounted at the same distance from the center of the user's field of view). As shown in Figure 3, the separation 501 between the left eye and the left camera is equal to the separation 502 between the right eye and the right camera. (Of course, if there is a difference between separation 501 and separation 502, the aggregated position of object S may be calculated by taking the corresponding weighted average of the positions for the purpose of superimposing the virtual images.)
[0073] Furthermore, a further use of offset 530 is to address a virtual image to either image source 1 or 2 without having to determine a specific distance range to the associated object. For example, if the offset 530 for object S exceeds a predetermined threshold, it may be determined that any virtual image mapped to the object should be transmitted to the first image source 1 for a near-focus depth display. Conversely, if the offset falls below a predetermined threshold, it may be determined that any virtual image mapped to that object should be transmitted to the second image source 2 for a far-focus depth display. Such further uses may find particular usefulness when a particular object or zone is likely to shift between the near and far fields.
[0074] In the example above, the camera device 10 was used as both a distance measuring device and an imaging device. In an alternative example, it may be possible to provide a distance measuring device separate from the imaging device.
[0075] Display devices and systems may be installed in the helmet. The helmet may be used for managing or controlling a vehicle, particularly an aircraft.
Claims
1. A user-installable display system, A display device for displaying an image at a first fixed depth and a second fixed depth, the display device having a display field of view (FOV), A distance range device for generating ambient image data across a distance range field of view, A processing unit, Receiving surrounding image data, Using the aforementioned surrounding image data, distance range data is generated, A first zone of the surrounding scene for displaying the image at the first depth, and The second zone of the surrounding scene for displaying the image at the second depth To determine this, the distance range data is used, The process involves receiving virtual image data (VID) representing at least two images for display as a virtual image, From the virtual image data (VID), determine the virtual image associated with the first zone or the first depth, and distribute such first depth image to the display device for display at the first depth. From the virtual image data (VID), determine the virtual image associated with the second zone or the second depth, and distribute such second depth image to the display device for display at the second depth. To process the aforementioned virtual image data (VID), A processing unit configured to perform the following: A display system equipped with these features.
2. The aforementioned distance range device is A first imaging device configured to capture a first portion of the surrounding scene, The system comprises a second imaging device configured to capture a second portion of the surrounding scene, The system according to claim 1, wherein the common area of the surrounding scene is imaged by both the first imaging device and the second imaging device.
3. The system according to claim 2, wherein the first and second imaging devices simultaneously generate two respective subsets of ambient image data for use in generating distance range data.
4. The system according to claim 3, comprising using disparity determination to generate distance range data.
5. The system according to any one of claims 1 to 4, wherein the display FOV is narrower than the distance range FOV.
6. The system according to any one of claims 1 to 5, wherein the processor unit is configured to perform object recognition on the surrounding image data in order to identify a predetermined zone or object as the first or second zone.
7. The system according to any one of claims 1 to 6, as dependent on claim 2, wherein the processor unit is configured to perform object recognition on the ambient image data (AID) to identify objects in the common area, and, given a known separation between the first imaging device and the second imaging device, to determine a geometric distance range.
8. The aforementioned display device, A first image source configured to generate an optical signal that carries an image for display at a first depth, A second image source configured to generate an optical signal that carries an image for display at a second depth, In this context, processing the virtual image data (VID) in the processing unit means that Distributing to the first image source a virtual image associated with the first zone of the surrounding scene or the first depth, The method comprises distributing to the second image source a virtual image associated with the second zone of the surrounding scene or the second depth. The system according to any one of claims 1 to 7.
9. The system according to any one of claims 1 to 8, wherein the first zone corresponds to a depth closer than the second zone.
10. The system according to any one of claims 1 to 9, wherein the first zone corresponds to a focal length of 50 cm to 200 cm.
11. The system according to any one of claims 1 to 10, wherein the second zone corresponds to a focal length of infinity.
12. The system according to any one of claims 1 to 11, further comprising a virtual image database for generating the VID, wherein the virtual image database comprises a plurality of image datasets, each image dataset comprising virtual images enumerated with respect to data identifying one or more zones suitable for overlays on the virtual image, or data identifying one or more depths of focus suitable for overlays on the image.
13. The user-mountable display system is for use in a vehicle or platform having a user workstation / cockpit and associated control / instrument console, wherein the first zone generally corresponds to the console and the first depth generally corresponds to the distance between the user and the console, according to any one of claims 1 to 12.
14. The user-installable display system is for use in a vehicle or platform having a workstation / cockpit and a canopy / windshield, and the second zone generally corresponds to the canopy / windshield, according to any one of claims 1 to 13.
15. A method for providing a virtual image to a user of a user-worn display system, comprising: a display device having a display field of view (FOV) and for displaying an image at at least first and second fixed depths; and a distance range device for generating ambient image data (AID) across a distance range field of view (FOV), The aforementioned method is performed by a processor, Receiving surrounding image data, Using the aforementioned surrounding image data, distance range data is generated, A first zone of the surrounding scene for displaying the image at the first depth, and The second zone of the surrounding scene for displaying the image at the second depth To determine this, the distance range data is used, The process involves receiving virtual image data (VID) representing at least two images for display as a virtual image, From the virtual image data (VID), determine a virtual image associated with the first zone of the surrounding scene or the first depth, and distribute such first depth images to the display device for display at the first depth, and From the virtual image data (VID), determine a virtual image associated with the second zone of the surrounding scene or the second depth, and distribute such second depth images to the display device for display at the second depth. Processing the aforementioned virtual image data (VID), A method that includes [a certain feature].