Enhanced context overlay for extended reality devices

Abstract

A viewing area (221, 321, 405) of an extended reality device (103, 203, 303, 401) is controlled. Control includes detecting (501, 701) a user gaze direction (229) toward the viewing area (221, 321, 405); using radar (505, 703) to detect respective locations (225) of, and respective distances (113) to one or more real world surfaces (109, 209, 309) in a real world environment outside of the extended reality device (103, 203, 303, 401); using (705) the detected user gaze direction (229) and the respective locations (225) of the one or more respective real world surfaces (109, 209, 309) to select one of the one or more real world surfaces (109, 209, 309); and using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

Description

[0001] ENHANCED CONTEXT OVERLAY FOR EXTENDED REALITY DEVICES

[0002] BACKGROUND

[0003] The present invention relates to adjustment of a viewing area of an extended reality (XR) device, overlaying images of virtual surfaces / objects and / or text onto a real world image in a viewed XR environment, and more particularly to technology for adjusting the viewing area and / or such overlayed images in relation to a location of a real world object in the viewed XR environment.

[0004] Some or all of the following abbreviations are used in this specification:

[0005] Abbreviation Explanation

[0006] 2D Two Dimensional

[0007] Al Artificial Intelligence

[0008] Ao A Angle of Arrival

[0009] AR Augmented Reality

[0010] MR Mixed Reality

[0011] RX Reception

[0012] TX Transmission

[0013] VR Virtual Reality

[0014] XR Extended Reality

[0015] The term “extended reality” (XR) is a generic term that encompasses augmented reality (AR), mixed reality (MR), and virtual reality (VR) technologies. In each of these, a user is able to experience immersion in a computer generated environment. The term VR generally refers to environments that are entirely computer generated, whereas in AR and MR, the user experiences a mixture of a real world environment and one that is computer generated. As an example, a user wearing an AR device while sitting at their desk may be able to see not only real world objects such as their telephone, notepad, and pen but also computer generated (i.e., virtual) digital objects such as a virtual vase with flowers and a virtual computer screen. In short, when using AR / XR devices (e.g., AR / XR headset devices), the user is able to see the surrounding real world environment while the technology adds user-perceivable additional content on top of the real environment. Such devices are typically constructed using one or more outward directed cameras to analyze the environment around the user (e.g., for object detection). The devices may also include gaze tracking capabilities to detect where in the environment the user has their attention and eye focus (e.g., by generating so-called gaze heatmaps, which statistically show how much time the user spends looking at each part of the visual environment).

[0016] The above functionalities allow additional artificial objects and / or text to be added to the end user views by, for example, superimposing an image of the object and / or text that is configured to create the illusion that it is located at a certain viewing location and viewing distance based on environmental knowledge that the device has acquired from images captured by the camera. These images are analyzed to determine how to adjust the image so that, when overlay ed on top of the user’s view of the real world, it will appear to exist at the particular location and viewing distance in that real world (e.g., on top of a flat surface).

[0017] The inventors of the herein-described technology have recognized that conventional technology, such as that which is described above, has limitations and problems. One aspect relates to the fact that there may be reasons why future XR headsets will not be equipped with outward facing cameras. Such reasons may arise from privacy concerns, where persons within proximity of XR devices will not accept being surrounded by constantly active cameras from such device. There are recorded instances of users wearing XR devices being denied entry to some commercial establishments because the devices represented a form of ubiquitous recording.

[0018] Furthermore, the European Commission has published a document called, “Proposal for a Regulation Of The European Parliament And Of The Council Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts.” In the proposal, Al systems are divided into different categories related to their societal risk, with different categories facing different levels of regulations. One category is “Unacceptable Risk” which will be severely regulated or even forbidden. That category includes real-time and remote biometric identification systems, which might include forms of facial recognition systems in public places. Examples of “High Risk” which will be strictly regulated include Surveillance systems (e.g., biometric monitoring for law enforcement, facial recognition systems). One implication of this are proposals for bans of facial recognition in public places. Overall, such regulations might put restrictions on the usage of AR glasses / devices with certain functions in public places.

[0019] To reach market attraction of XR devices, other solutions not based on outward facing cameras are therefore needed. From a technological perspective, one of the problems to be solved is how to project augmented reality material within XR devices so that the end user experiences the augmentation fitting into the real world environment, and doing so without relying on camera and image recognition / analysis for the projection to fit within the surroundings. This could be challenging when, for example, added text or objects need to fit well at suitable viewing distances and on surfaces.

[0020] Eye-tracking, using cameras that face towards the eyes, has been used for some time. This technology can determine gaze direction with an accuracy as good as 0.7 degrees. If the device has stereo-camera monitoring of both eyes, the device can perform a distance measurement based on where the gaze directions of the two eyes intersect. However, given the close distance between the eyes, the distance estimate is only fairly good up to about a meter, and at longer distances this gaze-direction accuracy is not good enough for a proper distance estimation. For cheaper systems having only a single camera, where only one eye is monitored, eye tracking fails as a method for accurately estimating distance.

[0021] In view of the foregoing, there is a need for technology that address the above-described and related problems, including allowing an XR device to accurately estimate distances from the device to a point in an XR environment without reliance on images obtained from outward facing cameras.

[0022] SUMMARY

[0023] It should be emphasized that the terms “comprises” and “comprising”, when used in this specification, are taken to specify the presence of stated features, integers, steps or components; but the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0024] Moreover, reference letters may be provided in some instances (e.g., in the claims and summary) to facilitate identification of various steps and / or elements. However, the use of reference letters is not intended to impute or suggest that the so-referenced steps and / or elements are to be performed or operated in any particular order.

[0025] In accordance with one aspect of the present invention, the foregoing and other objects are achieved in technology (e.g., methods, apparatuses, nontransitory computer readable storage media, program means) for controlling a viewing area of an extended reality device. Controlling comprises detecting a user gaze direction toward the viewing area and using radar to detect respective locations of, and respective distances to one or more real world surfaces in a real world environment outside of the extended reality device. The detected user gaze direction and the respective locations of the one or more respective real world surfaces are used to select one of the one or more real world surfaces. One or both of the detected location of the real world surface and the detected distance to the real world surface are used to adjust the viewing area of the extended reality device.

[0026] In another aspect of some but not necessarily all embodiments consistent with the invention, using one or both of the detected location of the real world surface and the detected distance to the real world surface to adjust the viewing area of the extended reality device comprises: using one or both of the detected location of the real world surface and the detected distance to the real world surface to produce an adjusted image by adjusting a first image of a digital object; and displaying the adjusted image of the digital object on the viewing area of the extended reality device.

[0027] In yet another aspect of some but not necessarily all embodiments consistent with the invention, the first image is a predefined default image of the digital object. In some but not necessarily all of these embodiments, the first image is a previously adjusted image of the digital object. In some further alternatives, adjusting the first image of the digital object comprises producing a display size of the first image of the digital object, wherein the display size is associated with the detected location of the real world surface. In some but not necessarily all such embodiments, the first image of the digital object has a default size associated with a default location of the digital object relative to the extended reality device, and producing the display size of the first image of the digital object comprises: determining a focal distance based on one or both of the detected location of the real world surface and the detected distance to the real world surface; and setting the display size of the first image to be a scaled copy of the default size of the digital object, wherein a scaling of the scaled copy is based on one or both of the detected location of the real world surface and the focal distance.

[0028] In some but not necessarily all alternative embodiments, adjusting the first image of the digital object comprises adjusting a display location of the first image of the digital object within the viewing area of the extended reality device. In some alternatives, adjusting the display location of the first image of the digital object within the viewing area of the extended reality device comprises making a display location adjustment based on the user gaze direction toward the viewing area.

[0029] In further aspects of some alternative embodiments, a display location of the adjusted image of the digital object is set by an application that is running on the extended reality device. In still further aspects of some alternative embodiments, adjusting the first image of the digital object comprises adjusting a focal plane property of the first image of the digital object.

[0030] In another aspect of some but not necessarily all embodiments consistent with the invention, using one or both of the detected location of the real world surface and the detected distance to the real world surface to adjust the viewing area of the extended reality device comprises using one or both of the detected location of the real world surface and the detected distance of the real world surface to adjust a refractive power for light from outside of the viewing area of the extended reality device.

[0031] In yet another aspect of some but not necessarily all embodiments consistent with the invention, using the detected user gaze direction and the respective locations of the one or more respective real world surfaces to select one of the one or more real world surfaces comprises using the detected user gaze direction to estimate a direction and / or a direction of an estimated location in the real world environment; and comparing the estimated location in the real world environment with each of the respective locations of the one or more respective real world surfaces and selecting one of the one or more real world surfaces based on said comparison.

[0032] In still another aspect of some but not necessarily all embodiments consistent with the invention, using radar to detect respective locations of one or more real world surfaces in the real world environment outside of the extended reality device, and respective distances to the one or more real world surfaces in the real world environment comprises: using a first spatially subsampled antenna array having a first distance between antenna elements for transmission of radar signals; and using a second spatially sub-sampled antenna array having a second distance between antenna elements for reception of radar signals, wherein the second distance is different from the first distance. In such embodiments, operating the radar comprises using different transmission and reception beam patterns.

[0033] In some alternatives of such embodiments, radar signals having a wavelength of are transmitted, wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to three quarters X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to three quarters X. In some other alternatives, the first distance between the antenna elements for transmission of radar signals is greater than or equal to four times X; and the second distance between the antenna elements for reception of radar signals is greater than or equal to one half . And in still other alternatives, the first distance between the antenna elements for transmission of radar signals is equal to one half X; and the second distance between the antenna elements for reception of radar signals is greater than or equal to four times X.

[0034] In aspects of some but not necessarily all embodiments consistent with the invention, controlling the viewing area of the extended reality device further comprises configuring the first spatially sub-sampled antenna array to produce a first beam pattern having non-uniform beam spacing; and configuring the second spatially sub-sampled antenna array to produce a second beam pattern having non-uniform beam spacing, wherein grating lobes of the first beam pattern and grating lobes of the second beam pattern are substantially different.

[0035] In some further alternative inventive embodiments, a number of antenna elements for transmission of radar signals is greater than or equal to eight, and less than or equal to sixteen.

[0036] In another aspect of some but not necessarily all embodiments consistent with the invention, using radar to detect respective locations of one or more real world surfaces in the real world environment outside of the extended reality device, and respective distances to the one or more real world surfaces in the real world environment comprises sweeping a radar signal transmission beam based on the gaze direction. In some alternatives of such embodiments, an angle of arrival of returning echoes from the radar signal transmissions is monitored. In some alternatives, this comprises determining angles of arrival of returning echoes; and determining an angular extension of a detected surface based on the determined angles of arrival.

[0037] In yet another aspect of some but not necessarily all embodiments consistent with the invention, detecting the user gaze direction toward the viewing area comprises tracking a user gaze over a period of time and constructing a gaze heat map from user gaze tracking data.

[0038] In still another aspect of some but not necessarily all embodiments consistent with the invention, the extended reality device is an extended reality headset device.

[0039] BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the drawings in which:

[0041] Figure 1 illustrates a user who is wearing an XR device configured in the form of a headset, through which the user is able to view portions of an XR environment.

[0042] Figure 2 illustrates an XR device having a viewing area 221 with stereoscopic capability for seeing at least a portion of an XR environment.

[0043] Figure 3 illustrates the monitoring over time of the positions of a user’s eyes as they look at respective left and right portions of a viewing area of an extended reality device. Figure 4 is a block diagram of a nonlimiting exemplary XR device configured to carry out actions in accordance with the invention.

[0044] Figure 5 is, in one respect, a flowchart of actions performed by an XR device in a process for detecting a user-specified location in an XR environment and, in a further aspect, placing a virtual object at that location.

[0045] Figure 6A is a graph of a transmit beam pattern that exhibits a narrow beam width but includes multiple grating lobes, suitable for use in some inventive embodiments.

[0046] Figure 6B is a graph of a receive beam pattern having a single wider beam having a low gain towards TX grating lobe directions.

[0047] Figure 7 is, in one respect, a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments.

[0048] Figure 8 is, in one respect, a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments.

[0049] Figure 9 shows an exemplary controller that may be included in a an XR device to cause any and / or all of the herein-described and illustrated actions associated with that device to be performed.

[0050] DETAILED DESCRIPTION

[0051] The various features of the invention will now be described with reference to the figures, in which like parts are identified with the same reference characters.

[0052] The various aspects of the invention will now be described in greater detail in connection with a number of exemplary embodiments. To facilitate an understanding of the invention, many aspects of the invention are described in terms of sequences of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be recognized that in each of the embodiments, the various actions could be performed by specialized circuits (e.g., analog and / or discrete logic gates interconnected to perform a specialized function), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term “circuitry configured to” perform one or more described actions is used herein to refer to any such embodiment (i.e., one or more specialized circuits alone, one or more programmed processors, or any combination of these). Moreover, the invention can additionally be considered to be embodied entirely within any form of non- transitory computer readable carrier, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Thus, the various aspects of the invention may be embodied in many different forms, and all such forms are contemplated to be within the scope of the invention. For each of the various aspects of the invention, any such form of embodiments as described above may be referred to herein as “logic configured to” perform a described action, or alternatively as “logic that” performs a described action.

[0053] Embodiments consistent with the invention address problems related to accurately estimating distances from an XR device to a point in an XR environment without reliance on images obtained from outward facing cameras.

[0054] In one aspect, an XR device is equipped with one or more radar circuits and eye-tracking components for monitoring the gaze direction of at least one of a user’s eyes and in some embodiments the gaze directions of both eyes. The radar senses the real -world environment outside of the XR device and the radar sensing results are used to detect actual distances to objects at different viewing angles, and / or the size of the objects in the real world.

[0055] By combining the information from this radar sensing with user-sensing data from other available technologies, such as gaze tracking, heat maps, and gaze angle for the eyes the technology is able to ascertain a user-specified location in the XR environment (including a distance from the XR device to the user-specified location) and based on this information to determine how to suitably configure placement of augmenting information in the XR environment, such as size, pose and / or location of a digital object or text. In this way, the user of the XR device experiences the digital object or text as being projected into, and part of the XR environment at the proper viewing distance.

[0056] While the radar technology in XR devices may have limited capability with respect to forming very sharp beams due to form factor restrictions (e.g., limiting the number of antennas that can be included in the device), the directional accuracy of the radar beam measurements would match the sharpness of the gaze direction. Even with a limited number of receive antennas having individual digital receive chains, the angle of arrival of a radar echo can be accurately estimated. In another aspect of some but not necessarily all embodiments, the link budget (i.e., the sum of power gains and losses that a radar signal experiences) can be improved by beam forming in the gaze direction in the transmission. By combining transmit beam-sweeping and measuring angle of arrival of received echoes from objects, the distances of objects the user is looking at (as determined by gaze tracking) can be determined.

[0057] In another aspect of some but not necessarily all embodiments, the viewing distance to a user-selected location in an XR environment, as ascertained by combining radar sense information with gaze tracking information, is used to adapt lenses of an XR device (e.g., for use as adaptive reading glasses).

[0058] These and further aspects of inventive embodiments are described in the following.

[0059] Figure 1 illustrates a user 101 who is wearing an XR device 103 configured in the form of a headset, through which the user is able to view portions (depending on direction and pose of the XR device 103) of an XR environment 105. The XR environment 105 includes a real -world (i.e., physical) object 107 having any number of surfaces such as the surface 109. To illustrate aspects of inventive embodiments, suppose the user 101 desires to place a digital (i.e., virtual) object 111 on top of the real -world object 107. In order to create a realistic rendering of the digital object 111 that object’s rendering should be consistent with the user’s expectations with respect to location and size of the digital object 111. In some cases, the pose of the digital object 111 (e.g., direction and tilt angle of one or more of the object’s surfaces) may also be important. In order to configure a correct rendering as just described, it is necessary to know the location and distance of the digital object 111 relative to the XR device 103. Since it is desired to render the digital object 111 on top of the real -world object 107, the location of, and distance 113 to the real -world object 107 can be used as proxies for those characteristics of the digital object 111.

[0060] In one aspect of inventive embodiments, location and distance information can both be obtained by scanning the XR environment 105 with radar signals 115 and receiving radar reflections 117. The direction of arrival (DoA) of the radar reflection 117 will indicate the direction of the real -world object 107, and the time delay between transmission of the radar signal 115 and receipt of the radar reflection 117 corresponds to the round-trip distance (i.e., twice the distance) between the XR device 103 and whichever surface of the real -world object 107 the radar signal 115 reflected off of.

[0061] It is recognized that the XR environment may include more than one real -world object 107, and some number of these may all be sensed by radar. This creates an ambiguity with respect to which real -world object the user wants the digital object 111 to be placed on. To address this technological challenge, the user’s gaze direction is also sensed. In particular, when the user instructs the headset device to create the digital object 111 (e.g., by voice command or by pressing one or more buttons on an input device) the user 101 looks at the location where the digital object 111 is to be rendered. The user’s gaze direction at that moment is sensed and recorded. Then, when multiple real -world objects are sensed by the radar scanning, knowledge of the user’s gaze direction is combined with the detected directions of radar-sensed real -world objects. The real world object having the best matching direction is then selected. Since the direction and distance information of the digital object’s location within the XR environment 105 are now known, the image of digital object can then be adjusted so that its apparent location and apparent size at that location create a realistic user experience that the digital object 111 is actually present in the XR environment 105. To compensate for subsequent movement of the XR device 103, the XR device 103 senses the movement (e.g. by means of gyroscopic or Inertial Measurement Unit (IMU) technology) and continues to adjust the rendering of the digital object 111 so that its perceived presence at the location appears to be fixed.

[0062] Further aspects that are relevant to inventive embodiments are now discussed with reference to Figure 2. In this nonlimiting example, an XR device 203 has a viewing area 221 with stereoscopic capability, so the viewing area 221 has two portions, each dedicated for viewing by a respective one of a pair of user’s eyes 201. By peering at or through the viewing area 221 the user 101 is able to see at least a portion of the XR environment 205.

[0063] In this example, the XR environment 205 includes a real -world surface 209 of an object (e.g., the surface could be a tabletop, desktop, countertop, top surface of a shelf, etc.). Further in this example, the user’s gaze 223 is directed towards a real -world location 225 on top of the surface 209. In this stereoscopic example, gaze monitoring of each eye would detect that the user’s left eye is looking at or through a display location xieft,yieft of the portion of the viewing area 221 allocated for use by the left eye, and that the user’s right eye is looking at or through a display location xright, yright of the portion of the viewing area 221 allocated for use by the right eye. In general, the coordinate pair xieft,yieft is not equal to the coordinate pair xright,yright. The gaze direction 229 in this instance can be determined by finding the intersection of the left and right gaze directions 223. In the example of Figure 2, this is shown as the real world location 225 having x,y,z coordinates (the third coordinate, z, in this case represents depth

[0064] Since digital (virtual) objects do not actually exist in the real world, they do not have real -world locations; they can be seen only in the viewing area 221 of the XR device 203. However, by creating left and right images at respective display locations in the viewing area 221 that line up with the user’s respective left and right gaze directions 223, the user 101 will perceive that the digital object 211 is located at an apparent location 227 in the XR environment 205. For convenience, the convention adopted in the herein-described examples will assign the same reference system for real-world locations and apparent locations, so that the x,y,z coordinates will be the same regardless of whichever one (“real-world” or “apparent”) is mentioned. However, this is by no means an essential aspect of all embodiments. To the contrary, different coordinate systems could be used for real-world and apparent locations. It is also noted that, to further enhance the effect that the digital object 211 will be perceived as being located at the apparent location 227 in the XR environment 205, the light of the digital object 211 should appear to come from the distance corresponding to the apparent location 227, so that the eye will focus that light on the retina when adjusting its lens to that distance, and for that reason an adjustable projection system with lenses can be used in the glasses.

[0065] It is further noted that various aspects found in inventive embodiments are not limited for use only with stereoscopic XR devices. To the contrary, inventive aspects are also present in monoscopic embodiments, as well as in embodiments employing a stereoscopic viewing area 221 but gaze monitoring of only one of the user’s eyes 201. In these latter embodiments, gaze direction 229 is based only on the gaze direction of the one monitored eye.

[0066] In a further aspect, and as mentioned earlier, radar scanning of the XR environment 205 is combined with user gaze monitoring to identify a particular real -world location 225. So, for example as shown in Figure 2, when the user is gazing at the real world location 225, a radar reflection from that point within the environment will indicate to the XR device 203 (via its direction of arrival) a radar reflection direction 231 that will be substantially the same as the detected gaze direction 229. The substantial match between those two measures confirms that the distance that is ascertainable from that same radar reflection is the distance to the real world location 225.

[0067] Various technologies for monitoring the user’s gaze are known, including technologies that detect particular points of interest within a user’s field of vision, and any such technologies may be employed in embodiments consistent with the invention. By way of example and with reference to Figure 3, the positions of a user’s eyes 301 are monitored over time as they look at respective left and right portions of a viewing area 321 of an extended reality device 303. Through the viewing area 321 the user is able to see a surface 309 upon which are placed three objects 311-1, 311-2, 311-3. The user’s various gaze directions (e.g., user gaze 323-1, 323-2, 323-3, 323-4) are detected and recorded. When a sufficient number of samples are recorded, a so-called “gaze heat map” can be created that shows, for any given point within a field of view, how often the user’s gaze was directed towards that point. Areas of higher intensity 337 in the gaze heat map indicate what the viewer spent the most time looking at. For purposes of example only, in Figure 3 three areas of higher intensity 337 correspond to the locations of the three objects 311-1, 311-2, 311-3. The above-mentioned radar object detections in combination with gaze heat map information (or equivalent) can be used to determine candidate positions for augmented reality text and object overlays. Figure 4 is a block diagram of a nonlimiting exemplary XR device 401 configured to carry out actions in accordance with the invention. The exemplary XR device 401 includes: an optical unit 403 that includes a viewing area 405 through which a user is able to see a portion of an XR environment. The optical unit is also able to superimpose computer generated digital objects within the viewing area 405.

[0068] A gaze tracker 407 that monitors the gaze directions of one or both of a user’s eyes.

[0069] - Radar circuitry 409 or equivalent radar functionality. As one example of the latter, the XR device 401 may include a modem 411 for wireless communication with, for example, a wireless communication network. Such modems 411 typically operate in frequencies that are suitable for radar operation. Accordingly, the modem 411 can be configured to operate as a radar device that is suitable for use in inventive embodiments. With a suitable radar signal bandwidth (e.g., on the order of 1GHz to give a radar range that permits resolution of two objects located 15cm distance from one another), multiple radar transceivers with a few decimeters distance in-between each other (or a radar transceiver with antenna array 419 as illustrated in Figure 4) can be used to achieve centimeter-level accuracy of distance estimation to objects surrounding the XR device 401.

[0070] An inertial measurement unit (IMU) 413 for tracking any movement of the XR device 401. This movement information can be used to stabilize images presented in the view area 405, and can also be used as a basis for determining adjustments to a rendered image of a digital object so that the digital object will appear to have a stable location as a user who is wearing the XR device 401 moves around.

[0071] A microphone array 415 that can serve a number of purposes including but not limited to receiving voiced commands and information from a user.

[0072] A controller 417 for controlling the above-described and other components of the XR device 401. The controller 417 may be configured from hardwired circuitry programmable software controlled processors / elements, or combination of both.

[0073] The following nonlimiting example illustrates the capabilities of an embodiment in accordance with aspects of the invention:

[0074] Assumptions for determination of link budget for beamforming transmit and angle of arrival receive radar:

[0075] • Frequency of operation: the 60GHz ISM band, wavelength (k)=5mm • TX array: 4x4 antenna elements, support 2D beamforming in lcm2

[0076] • RX array: 2x2 antenna elements, support 2D AoA in 5mm x 5mm

[0077] • Antenna element gain: G=3dBi

[0078] • Noise figure: NF=8dB

[0079] • Transmit power: PTx=lmW per antenna

[0080] • Integration time: Tint=lms

[0081] • SNRmin = 20dB

[0082] • Radar cross section: RCS=0.1m2

[0083] Calculated results:

[0084] RX sensitivity: PRx=-174dBm-10*log(Tint)+NF+SNRmin=-174+30+8+30dBm=-l 16dBm Maximum range: Rmax=(16*PTx*16*G*G*k2*RCS / (PRx*(4*7t)3))°-25=27m

[0085] It can be seen that, even without beamforming on the receive side, a range of 27m is obtainable. For targets in this range (closer than 27m) the TX beam can be swept while monitoring the angle of arrival of the returning echoes. The direction of small objects can then be determined with high precision. For large objects, however, only a midportion of the illuminated part is detected. It may then be difficult to determine if parts of the object are in the gaze direction of the user. But by performing a transmit beam sweep, it is possible to observe the range of angles of arrival of an object when illuminating different parts, especially the edges of an object. Such observations then make it possible to more accurately determine the angular extension of a large object.

[0086] Further aspects of some inventive embodiments will now be described with reference to Figure 5 which, in one respect, is a flowchart of actions performed by an XR device in a process for detecting a user-specified location in an XR environment and, in a further aspect, placing a virtual object at that location. In other respects, the blocks depicted in Figure 5 can also be considered to represent means 500 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0087] The illustrated actions may be initiated once a digital / virtual object is to be placed (projected) into the view of a mixed reality device such as an XR headset. The goal is to use radar to assist in identifying how to place the virtual object into the view in a natural manner (e.g., to fit in with the same focal plane setting as real -world objects, and to determine a size and location of the virtual object to fit the size, location of, and distance to a real object on which the virtual object can be projected).

[0088] As one step, the technology includes determining where, in the field of view, the virtual object is to be placed (i.e., a suitable viewing direction). In some embodiments this may be fixed by the application that the device is running, and therefore already predetermined when applying this process. In other examples placement may be more flexible and the method may include, as shown in Figure 5, gaze tracking or similar evaluation of the user’s eye directions to identify suitable directions (step 501). For example, the gaze tracking can be used to determine viewing directions that are often used by the user and therefore suitable for placement of a virtual object.

[0089] The one or more viewing directions are supplied to the radar functionality (step 503) so radar scanning can be performed at least in these directions. The radar sensing is utilized to determine one or more candidate real objects within the viewing direction (step 505). Based on time of arrival of radar reflections, the distance to a detected object is also determined (step 507). This could mean, for example determining the distance to a wall, box, table, TV, or any object within sensing range of the radar. In some instances, it is also possible to determine the size of a sensed object from the collected radar information.

[0090] By using the information from the radar sensing, an image of the virtual object can be adjusted prior to projection into the XR environment in relation to the one or more sensed real objects (step 509). Such adjustment could be to modify focal plane configuration or size adjustment, to fit the distance to the real object.

[0091] In another aspect of some but not necessarily all embodiments consistent with the invention, technology is configured to achieve full radar angular resolution with a reduced number of antennas. This can be especially important in embodiments where small size is a consideration (e.g., in XR devices configured like eyeglasses). Currently anticipated designs of XR-glasses may have size and / or power consumption constraints that limit the number of available radar antenna elements and their spacing. For example, 15 cm-wide glasses would accommodate ~60 antenna elements to fully leverage the array base dimension at A / 2-spacing at 60 GHz. Assuming that, for example, not more than 8 antenna elements may be feasible in practice, the achievable beam width would end up being ~8x wider (i.e., worse) than physically possible. Thus, directivity and spatial resolution of the radar can be quite critical in this use case.

[0092] In one class of embodiments, great er-t h an -k / 2 element spacing may be used to obtain beam widths permitted by the array base length. The resulting grating lobes are handled using the fact that the TX and RX lobes may be shaped or directed differently, especially if radar RX can be fully digital and RX beam control can be done in the baseband. Since the main lobe and grating lobe relations are known, the latter could be eliminated / ignored from one or multiple “snapshots”. The grating lobe impact is also more manageable in this use case since the augmented field of view angle, and thus the required angular range of the radar beam, may be narrower.

[0093] In an example, different TX and RX antenna element (array) setups are used to obtain desired TX and RX beam patterns. For example, in the above-mentioned exemplary use case involving XR glasses, the TX circuitry may employ 8 antenna elements spanning the full width of the glasses at approximately 4*k spacing. As shown in Figure 6A, this creates a beam pattern 601 that exhibits a narrow beam width but includes multiple grating lobes. The RX circuitry, on the other hand, may employ, for example, 8 antenna elements, placed, for example, in the middle region of the glasses, at spacing that provides a single wider main lobe while having a low gain towards TX grating lobe directions. This beam pattern 603 is illustrated in Figure 6B.

[0094] In some alternative embodiments, even fewer than 8 RX elements may be used since, for example, 6 elements provide ~20dB suppression at the location of the closest grating lobe. The TX and RX elements and their circuitry may be separate, also providing a degree of signal isolation.

[0095] Narrow-beam TX illumination can be combined with wider-beam RX reception that suppresses the TX grating lobes. A high-resolution image or ranging results may thus be collected and objects or their distances detected as if a full-base, Nyquist-spaced array were used.

[0096] The above-mentioned choice of narrow TX beams used with wider RX beams may be preferable if high resolution scanning is desired to minimize the effect of any spurious reflections from nearby objects that may reach the receiver via further scattering. The scanning may also take advantage of the fact that if multiple grating lobe directions are simultaneously illuminated, the number of TX beam steering settings in this example may be limited to 8-16 (depending on the desired spatial direction oversampling) to cover a single main-to-grating lobe angular interval. For each TX beam steering setting, multiple (e.g., up to 7) RX beam directions may be formed in baseband to probe the individual main and grating lobe directions.

[0097] If instead, continuous directional TX scanning is preferred, the array may be used for TX and the 4*k array for RX, where an RX lobe (e.g., the main lobe) is aligned with the TX lobe. In that case, up to -60-120 TX beam positions should be scanned, illuminating a wider region in each direction and relying on that the narrow RX main lobe in the TX beam direction is the only RX pattern component that picks up reflected signals. Again, the resulting reflected signal has high spatial resolution.

[0098] Other TX and RX lobe coordination approaches may also be considered in alternative embodiments to obtain effective high resolution with a limited number of antenna elements (e.g., approaches with non-uniform beam spacing where only the TX and RX beams overlap but grating lobe angles end up being offset or spatially interleaved).

[0099] Still further aspects of some but not necessarily all inventive embodiments will now be described with reference to Figure 7, which in one respect is a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments. In other respects, the blocks depicted in Figure 7 can also be considered to represent means 700 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0100] Figure 7 illustrates actions in a class of embodiments that are suitable for controlling a viewing area of an XR device. The actions include detecting a user gaze direction toward the viewing area (step 701) and also using radar to detect respective locations of, and respective distances to one or more real world surfaces in a real world environment outside of the extended reality device (step 703).

[0101] The detected user gaze direction and the respective locations of the one or more respective real world surfaces are used to select (step 705) one of the one or more real world surfaces.

[0102] One or both of the detected location of the real world surface and the detected distance to the real world surface are used to adjust (step 707) the viewing area of the extended reality device.

[0103] The adjustment of step 707 can take on different forms in different embodiments (including combinations of different forms). For example, in some instances the adjustment relates to how an image of a digital object will be rendered in the viewing area of an XR device. As described earlier, making a digital object appeared to be present at an apparent location in an XR environment can involve any one or more of:

[0104] • adjusting the size of an image of the digital object so that it will be compatible with the distance between the XR device and the desired apparent location in the XR environment

[0105] • adjusting a pose of an image of the digital object so that it will make sense to the user in light of the viewing angle from the XR device to the location in the XR environment • determining a focal distance based on one or both of the detected location of the real world surface and the detected distance to the real world surface; and adjusting the default size of the digital object based on one or both of the detected location of the real world surface and the focal distance

[0106] It is further noted that making any of the above mentioned adjustments can be performed when a digital object is initially created / rendered in the XR environment (so, for example, the adjustment is made to default size, pose, focal plane, etc.). Or, adjustment may be called into play when it is desired to move an already rendered digital object (i.e., one that is already present in the XR environment) from one location to another (so that the adjustment is being made to a previously adjusted image of the digital object).

[0107] It is also further noted that when speaking of adjusting a digital object’s apparent location within an XR environment, there is a corresponding adjustment to the display location within the viewing area of the XR device, so that the user will need to gaze in a particular direction towards the viewing area to be looking at the rendered digital object. In some applications the focal distance should also be changed.

[0108] Alternatively or in addition to the above, adjusting the viewing area of the XR device includes using one or both of the detected location of the real world surface and the detected distance of the real world surface to adjust a refractive power for light from outside of the viewing area of the extended reality device. This aspect is useful for use cases such as dynamically adjustable reading glasses.

[0109] Still further aspects of some but not necessarily all inventive embodiments will now be described with reference to Figure 8, which in one respect is a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments. In other respects, the blocks depicted in Figure 8 can also be considered to represent means 800 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0110] In some but not necessarily all inventive embodiments, using radar to detect (step 703) respective locations of one or more real world surfaces in the real world environment outside of the extended reality device, and respective distances to the one or more real world surfaces in the real world environment comprises using a first spatially sub-sampled antenna array having a first distance between antenna elements for transmission of radar signals (step 801). A second spatially sub-sampled antenna array having a second distance between antenna elements is used for reception of radar signals, wherein the second distance is different from the first distance, and wherein operating the radar comprises using different transmission and reception beam patterns (step 803).

[0111] With respect to configuring the antenna, a number of alternatives are possible. These include, but are not limited to:

[0112] • embodiments in which radar signals having a wavelength of X are transmitted, wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to one half X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to one half .

[0113] • embodiments in which the first distance between the antenna elements for transmission of radar signals is equal to four times X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to one half .

[0114] • Alternative embodiments in which the first distance between the antenna elements for transmission of radar signals is equal to one half X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to four times X.

[0115] • Embodiments in which the first spatially sub-sampled antenna array are configured to produce a first beam pattern having non-uniform beam spacing; and in which the second spatially sub-sampled antenna array is configured to produce a second beam pattern having non-uniform beam spacing, wherein grating lobes of the first beam pattern and grating lobes of the second beam pattern are substantially different. Non-limiting examples of criteria for being substantially different include embodiments in which: o the beam patterns have directional gain patterns that differ by at least X dB in at least one direction, o the beam patterns have directional gain patterns that differ by at least X dB in all directions that are not main lobe directions, o the main lobes of the beam patterns have overlapping sections at most Y dB, the main lobes of the beam patterns differ in any direction by at least in Z dB in all directions, possibly excluding the main lobe direction, where the first beam pattern exceeds the isotropic gain (0 dB) by K dB or more, the second beam pattern is below the isotropic gain by at least L dB.

[0116] Further aspects of embodiments consistent with the invention will now be described with reference to Figure 9, which shows an exemplary controller 901 that may be included in an XR device to cause any and / or all of the herein-described and illustrated actions associated with that device to be performed. In particular, the controller 901 includes circuitry configured to carry out any one or any combination of the various functions described herein. Such circuitry could, for example, be entirely hard-wired circuitry (e.g., one or more Application Specific Integrated Circuits - “ASICs”). Depicted in the exemplary embodiment of Figure 9, however, is programmable circuitry, comprising a processor 903 coupled to one or more memory devices 905 (e.g., Random Access Memory, Magnetic Disc Drives, Optical Disk Drives, Read Only Memory, etc.) and to an interface 907 that enables bidirectional communication with other elements of a device as described above. A complete list of possible other elements is beyond the scope of this description.

[0117] The memory device(s) 905 store program means 909 (e.g., a set of processor instructions) configured to cause the processor 903 to control other device elements so as to carry out any of the aspects described herein. The memory device(s) 905 may also store data (not shown) representing various constant and variable parameters as may be needed by the processor 903 and / or as may be generated when carrying out its functions such as those specified by the program means 909.

[0118] The various embodiments consistent with the invention offer advantages over conventional technology. One of these is the ability to use XR functionality that requires awareness of the contents and locations of things in an XR environment, and doing so based on technologies likely to be accepted from privacy and legal perspectives (i.e., Without reliance on outward facing cameras that might violate the privacy of bystanders).

[0119] The invention has been described with reference to particular embodiments. However, it will be readily apparent to those skilled in the art that it is possible to embody the invention in specific forms other than those of the embodiment described above.

[0120] For example, in some use cases, an application running on an XR device may have a digital object that it wishes to show to the user (e.g., virtual text), wherein the displaying of the object should be adapted to the real world environment at a location that the user is attending to. It may be desired that the object not be placed precisely where the user is looking, but rather some amount displaced from that location (e.g., in order not to disturb or obscure the view of what the user is looking at), at a similar distance so that refocusing is not needed. In such use cases, the system in some embodiments optimizes the focal distance and detailed placement of the digital object, using the location that the user is gazing at as a starting point from which adjustment is made. Thus, the described embodiments are merely illustrative and should not be considered restrictive in any way. The scope of the invention is further illustrated by the appended claims, rather than only by the preceding description, and all variations and equivalents which fall within the range of the claims are intended to be embraced therein.

Claims

CLAIMS:

1. A method of controlling a viewing area (221, 321, 405) of an extended reality device (103, 203, 303, 401), the method comprising: detecting (501, 701) a user gaze direction (229) toward the viewing area (221, 321, 405); using radar (505, 703) to detect respective locations (225) of, and respective distances (113) to one or more real world surfaces (109, 209, 309) in a real world environment outside of the extended reality device (103, 203, 303, 401); using (705) the detected user gaze direction (229) and the respective locations (225) of the one or more respective real world surfaces (109, 209, 309) to select one of the one or more real world surfaces (109, 209, 309); using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

2. The method of claim 1, wherein using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises: using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to produce an adjusted image by adjusting a first image of a digital object; and displaying the adjusted image of the digital object on the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

3. The method of claim 2, wherein the first image is a predefined default image of the digital object.

4. The method of claim 2, wherein the first image is a previously adjusted image of the digital object.

5. The method of any one of claims 2 through 4, wherein adjusting (509, 707) the first image of the digital object comprises producing a display size of the first image of the digitalobject, wherein the display size is associated with the detected location (225) of the real world surface (109, 209, 309).

6. The method of claim 5, wherein the first image of the digital object has a default size associated with a default location of the digital object relative to the extended reality device (103, 203, 303, 401), and wherein producing the display size of the first image of the digital object comprises: determining a focal distance based on one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309); and setting the display size of the first image to be a scaled copy of the default size of the digital object, wherein a scaling of the scaled copy is based on one or both of the detected location (225)of the real world surface (109, 209, 309) and the focal distance.

7. The method of any one of claims 4 through 6, wherein adjusting (509, 707) the first image of the digital object comprises adjusting a display location xieft,yieft,Xright,yrigH of the first image of the digital object within the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

8. The method of claim 7, wherein adjusting (509, 707) the display location of the first image of the digital object within the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises making a display location adjustment based on the user gaze direction (229) toward the viewing area (221, 321, 405).

9. The method of any one of claims 4 through 6, wherein a display location of the adjusted image of the digital object is set by an application that is running on the extended reality device (103, 203, 303, 401).

10. The method of any one of claims 4 through 9, wherein adjusting (509, 707) the first image of the digital object comprises adjusting a focal plane property of the first image of the digital object.

11. The method of claim 1, wherein using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises: using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) of the real world surface (109, 209, 309) to adjust a refractive power for light from outside of the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

12. The method of any one of the previous claims, wherein using the detected user gaze direction (229) and the respective locations of the one or more respective real world surfaces (109, 209, 309) to select one of the one or more real world surfaces (109, 209, 309) comprises: using the detected user gaze direction (229) to estimate a direction and / or a direction of an estimated location in the real world environment; and comparing the estimated location in the real world environment with each of the respective locations (225) of the one or more respective real world surfaces (109, 209, 309) and selecting one of the one or more real world surfaces (109, 209, 309) based on said comparison.

13. The method of any one of the previous claims, wherein using radar to detect respective locations (225) of one or more real world surfaces (109, 209, 309) in the real world environment outside of the extended reality device (103, 203, 303, 401), and respective distances (113) to the one or more real world surfaces (109, 209, 309) in the real world environment comprises: using (801) a first spatially sub-sampled antenna array having a first distance between antenna elements for transmission of radar signals; and using (803) a second spatially sub-sampled antenna array having a second distance between antenna elements for reception of radar signals, wherein the second distance is different from the first distance, wherein (803) operating the radar comprises using different transmission and reception beam patterns.

14. The method of claim 13, comprising: transmitting radar signals having a wavelength of X,wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to three quarters X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to three quarters X.

15. The method of claim 14, wherein: wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to four times X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to one half .

16. The method of claim 14, wherein: wherein the first distance between the antenna elements for transmission of radar signals is equal to one half X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to four times X.

17. The method of claim 13, comprising: configuring the first spatially sub-sampled antenna array to produce a first beam pattern having non-uniform beam spacing; configuring the second spatially sub-sampled antenna array to produce a second beam pattern having non-uniform beam spacing, wherein grating lobes of the first beam pattern and grating lobes of the second beam pattern are substantially different.

18. The method of any one of claims 13 through 17, wherein: a number of antenna elements for transmission of radar signals is greater than or equal to eight, and less than or equal to sixteen.

19. The method of any one of the previous claims, wherein using radar to detect respective locations (225) of one or more real world surfaces (109, 209, 309) in the real world environment outside of the extended reality device (103, 203, 303, 401), and respective distances (113) to the one or more real world surfaces (109, 209, 309) in the real world environment comprises:sweeping a radar signal transmission beam based on the gaze direction (229).

20. The method of claim 19, comprising: monitoring an angle of arrival of returning echoes from the radar signal transmissions.

21. The method of claim 20, comprising: determining angles of arrival of returning echoes; and determining an angular extension of a detected surface (109, 209, 309) based on the determined angles of arrival.

22. The method of any one of the previous claims, wherein detecting (501, 701) the user gaze direction (229) toward the viewing area (221, 321, 405) comprises: tracking a user gaze over a period of time and constructing a gaze heat map (337) from user gaze tracking data.

23. A computer program (909) comprising instructions that, when executed by at least one processor (903), causes the at least one processor (903) to carry out the method according to any one of the previous claims.

24. A carrier comprising the computer program (909) of claim 23, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a non-transitory computer readable storage medium (705).

25. An apparatus for controlling a viewing area (221, 321, 405) of an extended reality device (103, 203, 303, 401), wherein the apparatus is configured to perform: detecting (501, 701) a user gaze direction (229) toward the viewing area (221, 321, 405); using radar (505, 703) to detect respective locations (225) of, and respective distances (113) to one or more real world surfaces (109, 209, 309) in a real world environment outside of the extended reality device (103, 203, 303, 401); using (705) the detected user gaze direction (229) and the respective locations (225) of the one or more respective real world surfaces (109, 209, 309) to select one of the one or more real world surfaces (109, 209, 309);using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

26. The apparatus of claim 25, wherein using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises: using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to produce an adjusted image by adjusting a first image of a digital object; and displaying the adjusted image of the digital object on the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

27. The apparatus of claim 26, wherein the first image is a predefined default image of the digital object.

28. The apparatus of claim 26, wherein the first image is a previously adjusted image of the digital object.

29. The apparatus of any one of claims 26 through 28, wherein adjusting (509, 707) the first image of the digital object comprises producing a display size of the first image of the digital object, wherein the display size is associated with the detected location (225) of the real world surface (109, 209, 309).

30. The apparatus of claim 29, wherein the first image of the digital object has a default size associated with a default location of the digital object relative to the extended reality device (103, 203, 303, 401), and wherein producing the display size of the first image of the digital object comprises: determining a focal distance based on one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309); andsetting the display size of the first image to be a scaled copy of the default size of the digital object, wherein a scaling of the scaled copy is based on one or both of the detected location (225)of the real world surface (109, 209, 309) and the focal distance.

31. The apparatus of any one of claims 28 through 30, wherein adjusting (509, 707) the first image of the digital object comprises adjusting a display location xieft,yieft,Xright,yrigH of the first image of the digital object within the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

32. The apparatus of claim 31, wherein adjusting (509, 707) the display location of the first image of the digital object within the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises making a display location adjustment based on the user gaze direction (229) toward the viewing area (221, 321, 405).

33. The apparatus of any one of claims 28 through 30, wherein a display location of the adjusted image of the digital object is set by an application that is running on the extended reality device (103, 203, 303, 401).

34. The apparatus of any one of claims 28 through 33, wherein adjusting (509, 707) the first image of the digital object comprises adjusting a focal plane property of the first image of the digital object.

35. The apparatus of claim 25, wherein using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) to the real world surface (109, 209, 309) to adjust (509, 707) the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401) comprises: using one or both of the detected location (225) of the real world surface (109, 209, 309) and the detected distance (113) of the real world surface (109, 209, 309) to adjust a refractive power for light from outside of the viewing area (221, 321, 405) of the extended reality device (103, 203, 303, 401).

36. The apparatus of any one of claims 25 through 35, wherein using the detected user gaze direction (229) and the respective locations of the one or more respective real world surfaces (109, 209, 309) to select one of the one or more real world surfaces (109, 209, 309) comprises: using the detected user gaze direction (229) to estimate a direction and / or a direction of an estimated location in the real world environment; and comparing the estimated location in the real world environment with each of the respective locations (225) of the one or more respective real world surfaces (109, 209, 309) and selecting one of the one or more real world surfaces (109, 209, 309) based on said comparison.

37. The apparatus of any one of claims 25 through 36, wherein using radar to detect respective locations (225) of one or more real world surfaces (109, 209, 309) in the real world environment outside of the extended reality device (103, 203, 303, 401), and respective distances (113) to the one or more real world surfaces (109, 209, 309) in the real world environment comprises: using (801) a first spatially sub-sampled antenna array having a first distance between antenna elements for transmission of radar signals; and using (803) a second spatially sub-sampled antenna array having a second distance between antenna elements for reception of radar signals, wherein the second distance is different from the first distance, wherein (803) operating the radar comprises using different transmission and reception beam patterns.

38. The apparatus of claim 37, comprising: transmitting radar signals having a wavelength of X, wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to three quarters X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to three quarters X.

39. The apparatus of claim 38, wherein: wherein the first distance between the antenna elements for transmission of radar signals is greater than or equal to four times X; andwherein the second distance between the antenna elements for reception of radar signals is greater than or equal to one half .

40. The apparatus of claim 38, wherein: wherein the first distance between the antenna elements for transmission of radar signals is equal to one half X; and wherein the second distance between the antenna elements for reception of radar signals is greater than or equal to four times X.

41. The apparatus of claim 37, comprising: configuring the first spatially sub-sampled antenna array to produce a first beam pattern having non-uniform beam spacing; configuring the second spatially sub-sampled antenna array to produce a second beam pattern having non-uniform beam spacing, wherein grating lobes of the first beam pattern and grating lobes of the second beam pattern are substantially different.

42. The apparatus of any one of claims 37 through 41, wherein: a number of antenna elements for transmission of radar signals is greater than or equal to eight, and less than or equal to sixteen.

43. The apparatus of any one of claims 25 through 42, wherein using radar to detect respective locations (225) of one or more real world surfaces (109, 209, 309) in the real world environment outside of the extended reality device (103, 203, 303, 401), and respective distances (113) to the one or more real world surfaces (109, 209, 309) in the real world environment comprises: sweeping a radar signal transmission beam based on the gaze direction (229).

44. The apparatus of claim 43, comprising: monitoring an angle of arrival of returning echoes from the radar signal transmissions.

45. The apparatus of claim 44, comprising: determining angles of arrival of returning echoes; anddetermining an angular extension of a detected surface (109, 209, 309) based on the determined angles of arrival.

46. The apparatus of any one claims 25 through 45, wherein detecting (501, 701) the user gaze direction (229) toward the viewing area (221, 321, 405) comprises: tracking a user gaze over a period of time and constructing a gaze heat map (337) from user gaze tracking data.

47. The apparatus of any one of claims 25 through 46, wherein the extended reality device (103, 203, 303, 401) is an extended reality headset device.

Images