Location specification and virtual object placement in extended reality environments

Abstract

A user-specified location (225) in an extended reality environment (105, 205) is identified, wherein the extended reality environment (105, 205) is displayed in a viewing area (221) of an extended reality device (103, 203, 301). Identifying the user-specified location (225) involves detecting (401, 507, 601) an instance of a reference sound (115, 241) and estimating (403, 603) a direction of arrival (116, 243) of the detected instance of the reference sound (115, 241). Radar sensing is used (405, 501, 605) to detect one or more points of reflection (247) in a real world environment outside of the extended reality device (103, 203, 301), wherein each of the one or more points of reflection (247) comprises a reflection direction (231) from the extended reality device (103, 203, 301) and a reflection distance (113) from the extended reality device (103, 203, 301). The estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) and one or more reflection directions of respective ones of the one or more points of reflection are used (607) to select one of the one or more points of reflection. The selected point of reflection is used (411, 609) as a basis for determining the user-specified location (225) in the extended reality environment (105, 205).

Description

[0001] LOCATION SPECIFICATION AND VIRTUAL OBJECT PLACEMENT IN EXTENDED REALITY ENVIRONMENTS

[0002] BACKGROUND

[0003] The present invention relates to specification of locations in extended reality (XR) environments, placement of virtual objects in XR environments, user specification of locations in XR environments, and placement of a virtual object at a user-specified location in an XR environment.

[0004] Some or all of the following abbreviations are used in this specification:

[0005] Abbreviation Explanation

[0006] AR Augmented Reality

[0007] DoA Direction of Arrival

[0008] IMU Inertial Measurement Unit

[0009] MR Mixed Reality

[0010] RADAR Radio Detection and Ranging

[0011] SLAM Simultaneous Localization and Mapping

[0012] VR Virtual Reality

[0013] XR Extended Reality

[0014] The term “extended reality” (XR) is a generic term that encompasses augmented reality (AR), mixed reality (MR), and virtual reality (VR) technologies. In each of these, a user is able to experience immersion in a computer generated environment. The term VR generally refers to environments that are entirely computer generated, whereas in AR, MR, and XR, the user experiences a mixture of a real world environment and one that is computer generated. As an example, a user wearing an AR device while sitting at their desk may be able to see not only real world objects such as their telephone, notepad, and pen but also computer generated (i.e., virtual) digital objects such as a virtual vase with flowers and a virtual computer screen. In short, when using AR / XR devices (e.g., AR / XR headset devices), the user is able to see the surrounding real world environment while the technology adds user-perceivable additional content on top of the real environment. Such devices are typically constructed using one or more outward directed cameras to analyze the environment around the user (e.g., for object detection). The devices may also include gaze tracking capabilities to detect where in the environment the user has their attention and eye focus (e.g., by generating so-called gaze heatmaps, which statistically show how much time the user spends looking at each part of the visual environment).

[0015] The above functionalities allow additional artificial objects and / or text to be added to the end user views by, for example, superimposing an image of the object and / or text that is configured to create the illusion that it is located at a certain viewing location and viewing distance based on environmental knowledge that the device has acquired from images captured by the camera. These images are analyzed to determine how to adjust the image so that, when overlay ed on top of the user’s view of the real world, it will appear to exist at the particular location and viewing distance in that real world (e.g., on top of a flat surface).

[0016] Digital objects encompass a wide range of media elements, including 3D models, images, sounds, videos, and interactive assets, which can represent anything from simple static geometric shapes to complex, lifelike moving characters or objects. Digital objects play a crucial role in enriching user experiences, as they provide the building blocks for immersive environments and interactive scenarios. They can be created, manipulated, and shared by users, enabling collaborative and engaging experiences that extend beyond the constraints of the physical world. In the context of the metaverse and AR, digital objects serve as the foundation for various applications, such as gaming, education, commerce, and communication, transforming the way we interact with and perceive the digital realm.

[0017] A user may want to place a digital object within the virtual environment within proximity of themself. Commonly a user would want to place a digital object in a specific location, e.g., for inspecting it or to show the object to other people sharing the virtual space. The location in this context refers to a coordinate in the frame of reference created or used by the user. For AR- glasses this can be a coordinate in the SLAM-built map.

[0018] There are two steps involved:

[0019] 1. Interpreting the intent of the user to place a digital object.

[0020] 2. Determining the coordinate of the intended placement.

[0021] For some XR-devices, an outwards-facing camera and a microphone can be used to perform these steps, for example, using voice commands for step 1, and pointing with a finger for performing step 2. The inventors of the herein-described technology have recognized that conventional technology, such as that which is described above, has limitations and problems. One aspect relates to the fact that there may be reasons why future XR headsets may not all be equipped with outward facing cameras. Such reasons may arise from privacy concerns, where persons within proximity of XR devices will not accept being surrounded by constantly active cameras from such device. There are recorded instances of users wearing XR devices being denied entry to some commercial establishments because the devices represented a form of ubiquitous recording.

[0022] Furthermore, the European Commission has published a document called, “Proposal for a Regulation Of The European Parliament And Of The Council Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) And Amending Certain Union Legislative Acts.” In the proposal, Al systems are divided into different categories related to their societal risk, with different categories facing different levels of regulations. One category is “Unacceptable Risk” which will be severely regulated or even forbidden. That category includes real-time and remote biometric identification systems, which might include forms of facial recognition systems in public places. Examples of “High Risk” which will be strictly regulated include Surveillance systems (e.g., biometric monitoring for law enforcement, facial recognition systems). One implication of this are proposals for bans of facial recognition in public places. Overall, such regulations might put restrictions on the usage of AR glasses / devices with certain functions in public places.

[0023] To reach market attraction of XR devices, other solutions not based on outward facing cameras are therefore needed. From a technological perspective, one of the problems to be solved is how to project augmented reality material within XR devices so that the end user experiences the augmentation fitting into the real world environment, and doing so without relying on camera and image recognition / analysis for the projection to fit within the surroundings. This could be challenging when, for example, added text or objects need to fit well at suitable viewing distances and on surfaces.

[0024] Eye-tracking, using cameras that face towards the eyes, has been used for some time. This technology can determine gaze direction with an accuracy as good as 0.7 degrees. If the device has stereo-camera monitoring of both eyes, the device can perform a distance measurement based on where the gaze directions of the two eyes intersect. However, given the close distance between the eyes, the distance estimate is only fairly good up to about a meter, and at longer distances this gaze-direction accuracy is not good enough for a proper distance estimation. For cheaper systems having only a single camera, where only one eye is monitored, eye tracking fails as a method for accurately estimating distance.

[0025] In view of the foregoing, there is a need for technology that address the above-described and related problems, including allowing an XR device to accurately identify a coordinate in an XR environment without reliance on an outward-facing camera system in order to, for example, place a digital object at a user-desired location.

[0026] SUMMARY

[0027] It should be emphasized that the terms “comprises” and “comprising”, when used in this specification, are taken to specify the presence of stated features, integers, steps or components; but the use of these terms does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0028] Moreover, reference letters may be provided in some instances (e.g., in the claims and summary) to facilitate identification of various steps and / or elements. However, the use of reference letters is not intended to impute or suggest that the so-referenced steps and / or elements are to be performed or operated in any particular order.

[0029] In accordance with one aspect of the present invention, the foregoing and other objects are achieved in technology (e.g., methods, apparatuses, nontransitory computer readable storage media, program means) for identifying a user-specified location in an extended reality environment, wherein the extended reality environment is displayed in a viewing area of an extended reality device. Actions performed in this regard comprise detecting an instance of a reference sound and estimating a direction of arrival of the detected instance of the reference sound. Also, radar sensing via radio signal transmission and reception is used to detect one or more points of reflection in a real world environment outside of the extended reality device, wherein each of the one or more points of reflection comprises a reflection direction from the extended reality device and a reflection distance from the extended reality device. The estimated direction of arrival of the detected instance of the reference sound and one or more reflection directions of respective ones of the one or more points of reflection are used to select one of the one or more points of reflection. The selected point of reflection is used as a basis for determining the user-specified location in the extended reality environment.

[0030] In another aspect of some but not necessarily all embodiments consistent with the invention, the actions comprise placing a virtual object at an extended reality location in the extended reality environment that corresponds to the selected point of reflection in the real world environment.

[0031] In yet another aspect of some but not necessarily all embodiments consistent with the invention, the actions comprise displaying the virtual object in the viewing area of the extended reality device.

[0032] In still another aspect of some but not necessarily all embodiments consistent with the invention, the reference sound is one of: a finger snapping sound; a sound associated with tapping, knocking, or scratching on a surface in the real world environment; and a sound of a sound producing device located in the real world environment.

[0033] In another aspect of some but not necessarily all embodiments consistent with the invention, using radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device comprises initiating the radar sensing in response to detecting the instance of the reference sound.

[0034] In yet another aspect of some but not necessarily all embodiments consistent with the invention, using radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device comprises initiating the radar sensing in response to estimating the direction of arrival of the detected instance of the reference sound.

[0035] In some but not necessarily all alternative embodiments consistent with the invention, using radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device further comprises directing a radar beam in the estimated direction of arrival of the detected instance of the reference sound.

[0036] In still another aspect of some but not necessarily all embodiments consistent with the invention, using radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device is performed prior to detecting an instance of the reference sound.

[0037] In some but not necessarily all such embodiments, actions comprise deciding, for each one of the one or more detected points of reflection, whether said each one of the one or more detected points is a candidate source of the reference sound.

[0038] In another aspect of some but not necessarily all such embodiments, actions comprise using gesture recognition to decide, for said each one of the one or more detected points of reflection, whether said each one of the one or more detected points is the candidate source of the reference sound.

[0039] In yet another aspect of some but not necessarily all such embodiments, actions comprise detecting an instance of a first user action; and initiating the radar sensing in response to the detected instance of the first user action. In various alternatives, and without limitation, the first user action is one or more of: a reference head movement sensed by the extended reality device; and a sensed user gaze in a reference direction toward the viewing area of the extended reality device.

[0040] In still another aspect of some but not necessarily all such embodiments, actions comprise detecting a Doppler shift of radar data collected by the radar sensing; detecting that the Doppler shift of the radar data collected by the radar sensing matches a reference Doppler shift of radar data collected from snapping fingers of a user; and using the detecting that the Doppler shift of the radar data collected by the radar sensing matches the reference Doppler shift of radar data collected from snapping fingers of the user as a filter for deciding whether the instance of the reference sound is detected.

[0041] In another aspect of some but not necessarily all embodiments consistent with the invention, the actions comprise storing radar data in a buffer; and only after detecting the instance of the reference sound, processing the stored radar data to detect the one or more points of reflection in the real world environment outside of the extended reality device.

[0042] In yet another aspect of some but not necessarily all embodiments consistent with the invention, the extended reality device comprises an array of microphones, and estimating the direction of arrival of the detected instance of the reference sound comprises using sound information sensed by two or more microphones in the array of microphones.

[0043] In still another aspect of some but not necessarily all embodiments consistent with the invention, the actions comprise incorporating, into a mapping of the real world environment, the detected one or more points of reflection in the real world environment.

[0044] In some but not necessarily all alternatives of such embodiments, incorporating, into the mapping of the real world environment, the detected one or more points of reflection in the real world environment comprises adjusting the one or more points of reflection in the real world environment to compensate for a movement of the extended reality device that occurred after the radar sensing. In some but not necessarily all alternatives of such embodiments, actions comprise using an inertial measurement unit to sense the movement of the extended reality device. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the drawings in which:

[0046] Figure 1 illustrates a user who is wearing an XR device configured in the form of a headset and in accordance with some inventive embodiments, through which the user is able to view portions of an XR environment.

[0047] Figure 2 illustrates an XR device in accordance with some inventive embodiments having a viewing area with stereoscopic capability for seeing at least a portion of an XR environment.

[0048] Figure 3 is a block diagram of a nonlimiting exemplary XR device configured to carry out actions in accordance with the invention.

[0049] Figure 4 is, in one respect, a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments.

[0050] Figure 5 is, in one respect, a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments.

[0051] Figure 6 is, in one respect, a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments.

[0052] Figure 7 shows an exemplary controller that may be included in an XR device and configured to cause any and / or all of the herein-described and illustrated actions associated with that device to be performed.

[0053] DETAILED DESCRIPTION

[0054] The various features of the invention will now be described with reference to the figures, in which like parts are identified with the same reference characters.

[0055] The various aspects of the invention will now be described in greater detail in connection with a number of exemplary embodiments. To facilitate an understanding of the invention, many aspects of the invention are described in terms of sequences of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be recognized that in each of the embodiments, the various actions could be performed by specialized circuits (e.g., analog and / or discrete logic gates interconnected to perform a specialized function), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term “circuitry configured to” perform one or more described actions is used herein to refer to any such embodiment (i.e., one or more specialized circuits alone, one or more programmed processors, or any combination of these). Moreover, the invention can additionally be considered to be embodied entirely within any form of non- transitory computer readable carrier, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Thus, the various aspects of the invention may be embodied in many different forms, and all such forms are contemplated to be within the scope of the invention. For each of the various aspects of the invention, any such form of embodiments as described above may be referred to herein as “logic configured to” perform a described action, or alternatively as “logic that” performs a described action.

[0056] Embodiments consistent with the invention address problems related to allowing an XR device to accurately identify a coordinate in an XR environment without reliance on an outwardfacing camera system in order to, for example, place a digital object at a user-desired location.

[0057] In an aspect of embodiments consistent with the invention, a predetermined sound (e.g., a finger snap or a sound produced by tapping on an object) is recognized by an XR device and the direction of arrival (DoA) of the sound is detected. The radar scan of the XR environment is performed and a real world (i.e., physical) object located in the detected direction of arrival is identified. The radar scan also permits the distance between the XR device and the real world object to be identified. The direction of gaze in combination with the distance identify the user- specified location within the XR environment.

[0058] In another aspect of some but not necessarily all inventive embodiments, detection of the predetermined sound is used as a triggering event to initiate the above described process.

[0059] In an aspect of some but not necessarily all alternative embodiments, radar scanning is performed prior to an occurrence of the predetermined sound so that information about real world objects in the XR environment can be collected ahead of time.

[0060] In another aspect of some but not necessarily all inventive embodiments, sound can be recorded prior to the occurrence of the predetermined sound. This can be useful to, for example, establish a baseline reference of background sounds (typically in the frequency domain) that are to be considered noise. In some embodiments, the recording is limited to a relatively short recognition buffer time lapse.

[0061] In yet another aspect of some but not necessarily all inventive embodiments, the location of a point in 3D space within the XR environment is used for the purpose of placement of a virtual object or for subsequent processing of the localized point in the XR environment.

[0062] These and further aspects of inventive embodiments are described in the following. Figure 1 illustrates a user 101 who is wearing an XR device 103 configured in the form of a headset, through which the user is able to view portions (depending on direction and pose of the XR device 103) of an XR environment 105. The XR environment 105 includes a real -world (i.e., physical) object 107 having any number of surfaces such as the surface 109. To illustrate aspects of inventive embodiments, suppose the user 101 desires to place a digital (i.e., virtual) object 111 on top of the real -world object 107. In order to create a realistic rendering of the digital object 111 that object’s rendering should be consistent with the user’s expectations with respect to location and size of the digital object 111. In some cases, the pose of the digital object 111 (e.g., direction and tilt angle of one or more of the object’s surfaces) may also be important. In order to configure a correct rendering as just described, it is necessary to know the intended perceived location and distance 113 of the digital object 111 relative to the XR device 103. Since it is desired to render the digital object 111 on top of the real-world object 107, the location of, and distance 113 to the real -world object 107 can be used as proxies for those characteristics of the digital object 111.

[0063] In one aspect of inventive embodiments, location and distance information can both be obtained by detecting a reference sound 115 and determining the sound’s direction of arrival 116. Additionally, the XR environment 105 is scanned with radar signals 117 and its radar reflections 119 are received. A radar unit equipped with a radar signal transmitter and receiver may be located in the XR device. A radar-detected object lying in the direction of arrival of the detected reference sound 115 is taken as the source of the sound. The time delay between transmission of the radar signal 117 and receipt of the radar reflection 119 from that object corresponds to the round-trip distance (i.e., twice the distance) between the radar unit in the XR device 103 and the surface of the real -world object 107 that reflected the radar signal 117. Since the direction and distance information of the real world object’s location within the XR environment 105 are now known, the image of a digital object can then be adjusted so that its apparent location and apparent size at that location create a realistic user experience that the digital object 111 is actually present in the XR environment 105. To compensate for subsequent movement of the XR device 103, the XR device 103 senses the movement (e.g. by means of gyroscopic or Inertial Measurement Unit (IMU) technology) and continues to adjust the rendering of the digital object 111 so that the perceived placement at the location appears to be well anchored in the XR environment.

[0064] Further aspects that are relevant to inventive embodiments are now discussed with reference to Figure 2. In this nonlimiting example, an XR device 203 has a viewing area 221 with stereoscopic capability, so the viewing area 221 has two portions, each dedicated for viewing by a respective one of a pair of user’s eyes 201. By peering at or through the viewing area 221 the user 101 is able to see at least a portion of the XR environment 205.

[0065] In this example, the XR environment 205 includes a real -world surface 209 of an object (e.g., the surface could be a tabletop, desktop, countertop, top surface of a shelf, etc.). Further in this example, the user’s gaze 223 is directed towards a real -world location 225 on top of the surface 209. In this stereoscopic example, gaze monitoring of each eye would detect that the user’s left eye is looking at or through a display location xieft,yieft of the portion of the viewing area 221 allocated for use by the left eye, and that the user’s right eye is looking at or through a display location xright, yright of the portion of the viewing area 221 allocated for use by the right eye. In general, the coordinate pair xieft,yieft is not equal to the coordinate pair xright,yright. The gaze direction 229 in this instance can be determined by finding the intersection of the left and right gaze directions 223. In the example of Figure 2, this is shown as the real world location 225 having x,y,z coordinates (the third coordinate, z, in this case represents depth

[0066] Since digital (virtual) objects do not actually exist in the real world, they do not have real -world locations; they can be seen only in the viewing area 221 of the XR device 203. However, by creating left and right images at respective display locations in the viewing area 221 that line up with the user’s respective left and right gaze directions 223, the user 101 will perceive that the digital object 211 is located at an apparent location 227 in the XR environment 205 because that is where the left and right gaze directions 223 intersect. A vector from, for example, a mid-point of the XR device 203 to the apparent location 227 therefore corresponds to a gaze direction 229. For convenience, the convention adopted in the herein-described examples will assign the same reference system for real-world locations and apparent locations, so that the x,y,z coordinates will be the same regardless of whichever one (“real-world” or “apparent”) is mentioned. However, this is by no means an essential aspect of all embodiments. To the contrary, different coordinate systems could be used for real-world and apparent locations. It is also noted that, to further enhance the effect that the digital object 211 will be perceived as being located at the apparent location 227 in the XR environment 205, the light of the digital object 211 should appear to come from the distance corresponding to the apparent location 227, so that the eye will focus that light on the retina when adjusting its lens to that distance, and for that reason an adjustable projection system with lenses can be used in the glasses.

[0067] It is further noted that various aspects found in inventive embodiments are not limited for use only with stereoscopic XR devices. To the contrary, inventive aspects are also present in monoscopic embodiments, as well as in embodiments employing a stereoscopic viewing area 221 but gaze monitoring of only one of the user’s eyes 201. In these latter embodiments, gaze direction 229 is based only on the gaze direction of the one monitored eye.

[0068] In a further aspect, and as mentioned earlier, radar scanning of the XR environment 205 is combined with detection of a reference sound to identify a particular real -world location 225. So, for example as shown in Figure 2, a reference sound 241 is produced by, in a non-limiting example, having the user snap their fingers near the desired location in the XR environment 205. In another non-limiting example, the user could tap on, or scratch a surface 209 in the XR environment 205. The direction of arrival 243 of the sound waves reaching the XR device 203 is determined. A radar scan of real-world surfaces in the XR environment 205 also produces a reflection from a surface in the XR environment, including a surface of an object associated with the reference sound 241. For example, in the case of a finger snap, this can be produced anywhere within reach of the user 101, and the radar system can detect a radar reflection off of the user’s hand 245. In another non-limiting example in which the sound is a finger snap occurring close to a surface 209 in the XR environment 205, or in which the reference sound is produced by interacting with a surface 209 in the XR environment (e.g., by tapping or scratching), the radar reflection from the surface 209 can be detected by the radar system and its direction of arrival 231 determined. In each case, the direction of arrival 231 of the radar reflection is substantially the same as the direction of arrival 243 of the detected reference sound 241. Therefore, the radar system can determine the distance to the radar signal’s point of reflection 247, and this distance can be used as the distance to the location where the reference sound 241 was produced. The direction of arrival 243 of the reference sound and the determined distance thus indicate the user-specified location within the XR environment 205.

[0069] In some but not necessarily all embodiments, the user’ gaze is also tracked as described above. If the user is gazing at the real world location 225 while the reference sound 241 is being produced, then the detected gaze direction 229 will be substantially the same as the direction of arrival 243 of the reference sound, and also substantially the same as the direction of arrival 231 of the radar reflection. The gaze direction can thus further be used in some embodiments.

[0070] Figure 3 is a block diagram of a nonlimiting exemplary XR device 301 configured to carry out actions in accordance with the invention. The exemplary XR device 301 includes: an optical unit 303 that includes a viewing area 305 through which a user is able to see a portion of an XR environment. The optical unit is also able to superimpose computer generated digital objects within the viewing area 305. A gaze tracker 307 that monitors the gaze directions of one or both of a user’s eyes.

[0071] - Radar circuitry 309 or equivalent radar functionality. As one example of the latter, the XR device 301 may include a modem 311 for wireless communication with, for example, a wireless communication network. Such modems 311 typically operate in frequencies that are suitable for radar operation. Accordingly, the modem 311 can be configured to operate as a radar device that is suitable for use in inventive embodiments. With a suitable radar signal bandwidth (e.g., on the order of 1GHz to give a radar range that permits resolution of two objects located 15cm distance from one another), multiple radar transceivers with a few decimeters distance in-between each other (or a radar transceiver with antenna array 319 as illustrated in Figure 3) can be used to achieve centimeter-level accuracy of distance estimation to objects surrounding the XR device 301.

[0072] An inertial measurement unit (IMU) 313 for tracking any movement of the XR device 301. This movement information can be used to stabilize images presented in the view area 305, and can also be used as a basis for determining adjustments to a rendered image of a digital object so that the digital object will appear to have a stable location as a user who is wearing the XR device 301 moves around.

[0073] A microphone array 315 that can serve a number of purposes including but not limited to receiving voiced commands and information from a user. By including at least two microphones in the microphone array 315, and given a known separation distance between the microphones, time differences between signals picked up by respective ones of the multiple microphones can be used as an indicator of direction of arrival of sensed sounds.

[0074] A controller 317 for controlling the above-described and other components of the XR device 301. The controller 317 may be configured from hardwired circuitry programmable software controlled processors / elements, or combination of both.

[0075] It is noted that, while XR device 301 is an exemplary embodiment configured to carry out aspects of inventive embodiments, not all instances of inventive embodiments require the presence of each of the illustrated elements of XR device 301. As non-limiting examples, neither gaze tracking nor inertial measurements are required in all embodiments consistent with the invention, although each may be useful in some instances of inventive embodiments. The absence or presence of such elements will be readily apparent to persons skilled in the art based on whether the functionality provided by such elements is required to complete a particular embodiment.

[0076] Further aspects of at least some inventive embodiments will now be described with reference to Figure 4 which, in one respect, is a flowchart of actions performed by an XR device for ascertaining a desired location within an XR environment. In other respects, the blocks depicted in Figure 4 can also be considered to represent means 400 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0077] In an initializing aspect of inventive embodiments, the XR device is preconfigured (not shown) with one or more reference sound triggers that are registered within the device for monitoring by the device when carrying out the herein described actions. During performance of the process, an array of microphones is utilized for audio monitoring, and upon detecting an instance of the reference sound trigger (step 401) the signals produced by the multiple microphones are utilized to determine a direction of arrival of the sound waves (step 403).

[0078] In embodiments in which recognition of a reference sound triggers the process, it is beneficial for the microphones to be always active. Recognition of the reference sound trigger can include matching the envelope of a sensed sound with the envelope of the reference sound trigger (e.g., a sound envelope characterized by a rapid increase or decrease in sound pressure, such as with finger snapping or tapping sounds). Alternatively (or in addition), for reference sound triggers involving repetition of sound (e.g., repeated knocking or tapping), the repetition rate of the sound can be matched against an expected rate (e.g., 100-400 beats per minute). Such strategies are advantageous at least in that only low levels of power are required for processing. Once an envelope is matched, a second step can be performed to provide improved accuracy, such as converting the time-domain sound signals into the frequency domain and then checking for the presence of particular known properties (dependent on what the reference sound trigger is). In the first and / or second steps, spatial filtering (e.g., beamforming) can be applied to better separate the sound event from background noise.

[0079] Once a direction of arrival is determined, a radar transceiver is activated to sense at least a portion of real-world aspects of the XR environment. In some embodiments, a general radar scan of a portion of the XR environment can be performed. In some but not necessarily all alternative embodiments, radar scanning can be limited to a region of the XR environment in the direction of arrival of the sound waves. Regardless of how radar scanning is performed, one or more real world objects will be detected (i.e., at least the object that produced the reference sound) and by comparing the direction of arrival of the sensed sound waves with directions of arrival of radar echoes, an object associated with the reference sound is detected (step 405). The time delay between transmission of the radar signal and receipt of the radar reflection (i.e., time- of-flight of the radar signal) from the object associated with the reference sound corresponds to the round-trip distance (i.e., twice the distance) between the XR device and the surface of that object, and from this information the distance to the object is estimated (step 409). For estimating the distance to the source of the sound. Given the distance between the XR device and the radar-sensed object and also a direction from the XR device to the sound producing object, the user-specified location within the XR environment is now known (step 411). This location can be specified using the XR device as a frame of reference. A person of skill in the art will readily understand that the location can also be translated to any other frame of reference, if that is more desirable in any given application.

[0080] The location determined in this manner can thereafter be incorporated into a utilized mapping of the environment either as is or in some instances with the correction to account for user device movement (the user’s head being one such placement of device) or for a change of orientation detected by IMU sensing or similar sensors. In an optional step that is present in some but not necessarily all embodiments, a visual representation of the location (e.g., a visual highlighting) is presented within an end user interface (e.g., within the viewing area 221) to an end user (step 413). In some but not necessarily all embodiments, a purpose of the visual representation is to project the target position as a virtual object within the XR environment, enabling the user to consider the detected location and verify (by some user interaction) that the estimated position is accepted (step 415). If the user indicates that the detected location is not accepted, then the process can be restarted.

[0081] The class of exemplary embodiments illustrated by Figure 4 are premised on an instance of the predetermined sound being used as a trigger to activate radar scanning. In alternative embodiments this is not the case, and the actions illustrated in Figure 4 may be performed in a different order, and / or additional or fewer steps are implemented. To illustrate one such nonlimiting variation, reference is now made to Figure 5 which, in one respect, is a flowchart of actions performed by an XR device for ascertaining a desired location within an XR environment. In other respects, the blocks depicted in Figure 5 can also be considered to represent means 500 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0082] In an initializing aspect of inventive embodiments, the XR device is preconfigured (not shown) with one or more reference sound triggers that are registered within the device for monitoring by the device when carrying out the herein described actions. During performance of the process, the radar is operated continually to detect real world objects in the XR environment that can potentially be the source of the reference sound (step 501). In some embodiments, not all detected real world objects are considered sound sourcing candidates. To reduce subsequent processing of radar signal information that cannot possibly source the reference sound, false positives (i.e., detected objects that cannot serve as the source of the reference sound such as, for example, a distant wall or other object that the user cannot interact with if the reference sound is produced by such an interaction) are filtered out. In a non-limiting example, gesture recognition or similar classification methods that are known in the art can serve as a basis for such filtering. As this information is not used until the reference sound occurs, information about these candidate objects is stored by the XR device.

[0083] A position estimate is created for each of these candidate objects (step 503). The position can be estimated from, for example, the angle of arrival of the radar reflection (which indicates direction) and the time of flight (round trip time) from transmission of the radar signal until receipt of the radar echo (which corresponds to twice the distance between the XR device and the reflecting object).

[0084] In some embodiments, the microphone array 315 is always enabled and monitored, so that the reference sound will be detected whenever it occurs. In some alternative embodiments, the microphone array 315 is initially maintained in an inactive state (e.g., off or low-power), and in an optional step (indicated in Figure 5 by dashed lines), the radar circuitry 309 is used to detect an occurrence of what might be a sound-producing event (decision block 505). Detection can be performed by, for example, detecting a radar Doppler event that matches characteristic properties of the sound-producing event (e.g., motion associated with snapping of fingers, tapping on an object, etc.). The radar and associated processing is able to make this determination before the sound waves associated with the event can reach the microphone. Therefore, if no such event is detected (“No” path out of decision block 505), processing reverts back to step 501. But if a potential sound-producing event is detected (“Yes” path out of decision block 505), then the microphone array 315 is switched to an active state to receive the expected sound waves. Additionally, in some but not necessarily all such embodiments, the radar processing also determines a direction (relative to the XR device 301) in which the soundproducing event occurred, and this directional information is used to control the microphone array 315 processing such that it will listen for the sound waves in their expected direction of arrival. In this way, the XR device 301 avoids spending energy on trying to find the reference sound in other directions. In some further alternative embodiments, the timing of the listening activity can also be restricted to a window of time within which the sound waves are expected to be received (based on the distance of the detected Doppler-producing event from the XR device 301).

[0085] When the reference sound is not detected (“No” path out of decision block 507) the process merely repeats with radar scanning at step 501. In this aspect, it is noted that continuously repeating the radar actions enables the device to maintain up-to-date information about detected candidate objects. This information changes dynamically due to, for example, movement of the device (e.g., changing pose or location within the XR environment) of movement of a candidate sound source (e.g., a user’s hand that can potentially produce a fingersnap without needing to interact with any other object).

[0086] When the reference sound is detected (“Yes” path out of decision block 507), the sensed audio information is processed to determine which (if any) of the radar-detected candidate objects is the likely source of the reference sound (decision block 509). In some embodiments, this includes determining the angle of arrival of the sound waves (e.g., by detecting timing differences in audio signals produced by each microphone in a microphone array 315). By comparing the direction of arrival of the reference sound to the radar-detected sound source candidates, a match (in some embodiments allowing for some tolerance for difference) causes the matching radar-detected sound source candidate to be identified as the true source of the reference sound. In alternative embodiments involving step 505, the candidate object is already known, since radar processing detected the object as the source of the sound-producing event.

[0087] Accordingly, if no source of the reference sound is found from among the candidate sound sources (“No” path out of decision block 509), the process repeats with radar scanning at step 501.

[0088] When the source of the reference sound is identified (“Yes” path out of decision block 509), some or all aspects of the position estimate (i.e., distance and / or direction) of the soundsourcing candidate object is used as a basis for estimating the reference sound source’s position in a currently used map (step 511).

[0089] As with the class of embodiments illustrated with reference to Figure 4, the location determined in this manner can thereafter be incorporated into a utilized mapping of the environment either as is or in some instances with the correction to account for user device movement (the user’s head being one such placement of device) or for a change of orientation detected by IMU sensing or similar sensors. In an optional step that is present in some but not necessarily all embodiments, a visual representation of the location (e.g., a visual highlighting) is presented within an end user interface (e.g., within the viewing area 221) to an end user (step 513). In some but not necessarily all embodiments, a purpose of the visual representation is to project the target position as a virtual object within the XR environment, enabling the user to consider the detected location verify (by some user interaction) that the estimated position is accepted (step 515). If the user indicates that the detected location is not accepted, then the process can be restarted.

[0090] In further alternative embodiments, the aspects illustrated by Figure 5 are enhanced by exploiting detection of gaze direction 243 and XR device pose / posture. For example, when the XR device is embodied as a head-worn device, the radar can remain in a standby or low-power mode, and then briefly activated to perform the actions illustrated in Figure 5 when the user of the XR device turns their head and / or directs their gaze in a predetermined direction, such as looking down slightly close to the front of their body. This aspect would save energy consumption by the device and would mitigate the congestion of the frequency band being used by the radar.

[0091] In another aspect of some but not necessarily all inventive embodiments, and with further reference to Figure 5, power savings can be achieved by storing the raw radar data in a buffer rather than processing it at the time of collection. The processing of the radar data, such as with respect to step 503, can then be initiated only after an instance of the reference sound has been detected. For example, in the case where the reference sound is a finger-snapping, a buffer containing 15-20 ms of data might be a sufficient amount of storage if one assumes that a finger snap can be performed in about 10 ms and that arrival of the corresponding sound waves at the XR device occurs in about 1-4 ms.

[0092] In another aspect of some but not necessarily all inventive embodiments, and with further reference to Figure 5, another class of alternative embodiments in which the XR device is a head-worn device includes detecting when a user turns their head or gaze in a specific direction, and in response to the detection briefly activating the onboard radar and determining from received radar reflections whether a Doppler shift associated with the snapping of a user’s fingers is detected. If so, the XR device anticipates that a sound detection should be forthcoming, further establishing the intent to invoke a feature. More particularly, the nearby impression of a Doppler radar response starts a timer set to an amount of time within which the XR device ought to detect the corresponding, slower propagating sound waves. If the sound waves are not received within this time window (i.e., before a timeout event from the timer), then no further actions are taken by the XR device. A benefit of this aspect is that it avoids having other user devices detect the reference sound and respond as though the sound had been initiated by the users of those other devices. This is because, if the other users are outside the intended range (e.g., within a user’s arm length) the predetermined sound will arrive at the other users’ XR devices too late (i.e., outside the timeout period) compared to the radar detection. For purposes of illustration, the features just described could be implemented as part of decision block 507 in Figure 5 (i.e., determining whether a reference sound was detected within a window of time following detection of the Doppler shift associated with a finger snapping.

[0093] In yet another class of alternative embodiments in which it is not possible and / or desirable to detect a direction of arrival of sound waves (e.g., in the case of an XR device having only a single microphone), the above-described aspect, in which detection of a Doppler radar response associated with a finger snap instance initiates a time window within which the predetermined sound needs to be detected, can be used in place of the direction-related aspects of decision block 509 of Figure 5 because it filters out responding to sounds emanating from sources that are not within a specified range.

[0094] In still another class of alternative embodiments, gaze and pose detection may be used for the same purpose but are not able to detect the snapping itself. More particularly, information from gaze and pose detection is used to filter out from the range-angle (-Doppler) 2-dimensional (3 -dimensional) data cube only the angle and range that are consistent with the detected gaze and pose. The need for processing information from other locations is therefore eliminated.

[0095] Still further aspects of some but not necessarily all inventive embodiments will now be described with reference to Figure 6, which in one respect is a flowchart of actions performed by an XR device in accordance with some but not necessarily all inventive embodiments. In other respects, the blocks depicted in Figure 6 can also be considered to represent means 600 (e.g., hardwired or programmable circuitry or other processing means) for carrying out the described actions.

[0096] In the exemplary embodiment of Figure 6, an XR device identifies a user-specified location in an XR environment, wherein the XR environment is displayed in a viewing area of an XR device. Actions include detecting an instance of a reference sound (step 601). A direction of arrival of the detected instance of the sensed sound waves is estimated (step 603).

[0097] Also, radar sensing (i.e., via radio signal transmission and reception) is used to detect one or more points of reflection in a real world environment outside of the XR device (step 605), wherein each of the one or more points of reflection comprises a reflection direction from the XR device and a reflection distance from the XR device.

[0098] The estimated direction of arrival of the detected instance of the reference sound and one or more reflection directions of respective ones of the one or more points of reflection are used to select one of the one or more points of reflection (step 607).

[0099] The XR device then uses the selected point of reflection as a basis for determining the user-specified location in the extended reality environment (step 609). As a non-limiting example of some but not necessarily all embodiments, the XR device further perform placing a virtual object at an XR location in the XR environment that corresponds to the selected point of reflection in the real world environment (step 611). In some instances, the XR location in the XR environment is the selected point of reflection in the real world environment. But in other instances, the XR location in the XR environment may correspond to the selected point of reflection without being identical to it, such as when it is desired to further adjust a position and / or pose of the virtual object in order to have it better fit into and / or conform to its surroundings. Selection between one or the other may be application dependent.

[0100] Further aspects of embodiments consistent with the invention will now be described with reference to Figure 7, which shows an exemplary controller 701 that may be included in an XR device to cause any and / or all of the herein-described and illustrated actions associated with that device to be performed. In particular, the controller 701 includes circuitry configured to carry out any one or any combination of the various functions described herein. Such circuitry could, for example, be entirely hard-wired circuitry (e.g., one or more Application Specific Integrated Circuits - “ASICs”). Depicted in the exemplary embodiment of Figure 7, however, is programmable circuitry, comprising a processor 703 coupled to one or more memory devices 705 (e.g., Random Access Memory, Magnetic Disc Drives, Optical Disk Drives, Read Only Memory, etc.) and to an interface 707 that enables bidirectional communication with other elements of a device as described above. A complete list of possible other elements is beyond the scope of this description.

[0101] The memory device(s) 705 store program means 709 (e.g., a set of processor instructions) configured to cause the processor 703 to control other device elements so as to carry out any of the aspects described herein. The memory device(s) 705 may also store data (not shown) representing various constant and variable parameters as may be needed by the processor 703 and / or as may be generated when carrying out its functions such as those specified by the program means 709. Various embodiments consistent with the invention exhibit a number of advantages and benefits over conventional technology. For example, in some embodiments, digital (virtual) objects can be placed and detected in a virtual space without using cameras. Instead, a combination of sensors (e.g., radar, microphones) are used and this enables preservation of the privacy of users in the area. This is beneficial not only for devices that do not have cameras, but also for devices that might need to conform to privacy restricted use affecting some of the sensors. An example centers on AR-glasses (one of the types of XR devices) that are required to automatically shut down their cameras when non-consenting persons are in the vicinity, or when outside of a designated area.

[0102] Another benefit of embodiments consistent with the invention relates to privacy protections that prevent the continuous recording of sounds that could possibly breach user or bystander privacy. This concern is addressed in inventive embodiments that limit the actual recording of sound to only the brief interval required to capture the very short, reference sound, such as that which is produced by snapping fingers or tapping on an object (e.g., about 50 milliseconds) Other aspects of various embodiments, such as those that rely on radar sensing and / or gaze tracking, similarly do not call privacy concerns into play.

[0103] The invention has been described with reference to particular embodiments. However, it will be readily apparent to those skilled in the art that it is possible to embody the invention in specific forms other than those of the embodiment described above.

[0104] For example, some but not necessarily all embodiments further include accelerometer- or IMU-based processing for various purposes. For example, in one instance information about accelerometer- or IMU-sensed motion of the XR-device provides information that enables the XR-device to employ dead reckoning to compensate for any device movement that happens in the very brief interval between the moment the predetermined sound occurs and the estimation of the relative position between the sound-producing source and the XR device. In some but not necessarily all embodiments, information about sensed motion of the XR device is used after the relative position between the sound-producing source and the XR device has been determined to compensate for subsequent movement of the head. By adjusting the user-specified location based on device movement, the user is able to experience the location (e.g., including a digital object placed at that location) as being stable within the XR environment as the user moves around.

[0105] Thus, the described embodiments are merely illustrative and should not be considered restrictive in any way. The scope of the invention is further illustrated by the appended claims, rather than only by the preceding description, and all variations and equivalents which fall within the range of the claims are intended to be embraced therein.

Claims

CLAIMS:

1. A method of identifying a user-specified location (225) in an extended reality environment (105, 205), wherein the extended reality environment (105, 205) is displayed in a viewing area (221) of an extended reality device (103, 203, 301), the method comprising: detecting (401, 507, 601) an instance of a reference sound (115, 241); estimating (403, 603) a direction of arrival (116, 243) of the detected instance of the reference sound (115, 241); using (405, 501, 605) radar sensing via radio signal transmission and reception to detect one or more points of reflection (247) in a real world environment outside of the extended reality device (103, 203, 301), wherein each of the one or more points of reflection (247) comprises a reflection direction (231) from the extended reality device (103, 203, 301) and a reflection distance (113) from the extended reality device (103, 203, 301); using (607) the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) and one or more reflection directions of respective ones of the one or more points of reflection to select one of the one or more points of reflection; and using (411, 609) the selected point of reflection as a basis for determining the user- specified location (225) in the extended reality environment (105, 205).

2. The method of claim 1, comprising: placing a virtual object (111, 211) at an extended reality location in the extended reality environment (105, 205) that corresponds to the selected point of reflection in the real world environment.

3. The method of claim 2, comprising: displaying the virtual object in the viewing area (221) of the extended reality device (103, 203, 301).

4. The method of any one of the previous claims, wherein the reference sound (115, 241) is one of: a finger snapping sound; a sound associated with tapping, knocking, or scratching on a surface in the real world environment (105, 205); and a sound of a sound producing device located in the real world environment.

5. The method of any one of the previous claims, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) comprises: initiating the radar sensing in response to detecting (401, 507, 601) the instance of the reference sound (115, 241).

6. The method of any one of claims 1 through 3, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) comprises: initiating the radar sensing in response to estimating (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).

7. The method of claim 5, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) further comprises: directing a radar beam in the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).

8. The method of any one of claims 1 through 3, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) is performed prior to detecting (401, 507, 601) the instance of the reference sound (115, 241).

9. The method of claim 8, comprising: deciding, for each one of the one or more detected points of reflection, whether said each one of the one or more detected points is a candidate source of the reference sound (115, 241).

10. The method of claim 9, comprising: using gesture recognition to decide, for said each one of the one or more detected points of reflection, whether said each one of the one or more detected points is the candidate source of the reference sound (115, 241).

11. The method of any one of claims 8 through 10, comprising: detecting an instance of a first user action; and initiating the radar sensing in response to the detected instance of the first user action.

12. The method of claim 11, wherein the first user action is a reference head movement sensed by the extended reality device (103, 203, 301).

13. The method of claim 11, wherein the first user action is a sensed user gaze in a reference direction toward the viewing area (221) of the extended reality device (103, 203, 301).

14. The method of any one of claims 11 through 13, comprising: detecting a Doppler shift of radar data collected by the radar sensing; detecting that the Doppler shift of the radar data collected by the radar sensing matches a reference Doppler shift of radar data collected from snapping fingers of a user; and using the detecting that the Doppler shift of the radar data collected by the radar sensing matches the reference Doppler shift of radar data collected from snapping fingers of the user as a filter for deciding whether the instance of the reference sound (115, 241) is detected.

15. The method of any one of claims 8 through 14, comprising: storing radar data in a buffer; and only after detecting (401, 507, 601) the instance of the reference sound (115, 241), processing the stored radar data to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301).

16. The method of any one of the previous claims, wherein the extended reality device (103, 203, 301) comprises an array of microphones, and wherein estimating (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) comprises using sound information sensed by two or more microphones in the array of microphones.

17. The method of any one of the previous claims, comprising: incorporating, into a mapping of the real world environment, the detected one or more points of reflection in the real world environment.

18. The method of claim 17, wherein incorporating, into the mapping of the real world environment, the detected one or more points of reflection in the real world environment comprises: adjusting the one or more points of reflection in the real world environment to compensate for a movement of the extended reality device (103, 203, 301) that occurred after the radar sensing.

19. The method of claim 18, comprising: using an inertial measurement unit (313) to sense the movement of the extended reality device (103, 203, 301).

20. A computer program (709) comprising instructions that, when executed by at least one processor (703), causes the at least one processor (703) to carry out the method according to any one of claims 1 through 19.

21. A carrier comprising the computer program (1509) of claim 20, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a non-transitory computer readable storage medium (705).

22. An apparatus for identifying a user-specified location (225) in an extended reality environment (105, 205), wherein the extended reality environment (105, 205) is displayed in a viewing area (221) of an extended reality device (103, 203, 301), wherein the apparatus is configured to cause the extended reality device (103, 203, 3014) to perform: detecting (401, 507, 601) an instance of a reference sound (115, 241); estimating (403, 603) a direction of arrival (116, 243) of the detected instance of the reference sound (115, 241); using (405, 501, 605) radar sensing via radio signal transmission and reception to detect one or more points of reflection (247) in a real world environment outside of the extended reality device (103, 203, 301), wherein each of the one or more points of reflection (247) comprises a reflection direction (231) from the extended reality device (103, 203, 301) and a reflection distance (113) from the extended reality device (103, 203, 301);using (607) the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) and one or more reflection directions of respective ones of the one or more points of reflection to select one of the one or more points of reflection; and using (411, 609) the selected point of reflection as a basis for determining the user- specified location (225) in the extended reality environment (105, 205).

23. The apparatus of claim 22, wherein the apparatus is further configured to perform: placing a virtual object (111, 211) at an extended reality location in the extended reality environment (105, 205) that corresponds to the selected point of reflection in the real world environment.

24. The apparatus of claim 23, wherein the apparatus is further configured to perform: displaying the virtual object in the viewing area (221) of the extended reality device (103,203, 301).

25. The apparatus of any one of claims 22 through 24, wherein the reference sound (115, 241) is one of: a finger snapping sound; a sound associated with tapping, knocking, or scratching on a surface in the real world environment (105, 205); and a sound producing device located in the real world environment.

26. The apparatus of any one of claims 22 through 25, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) comprises: initiating the radar sensing in response to detecting (401, 507, 601) the instance of the reference sound (115, 241).

27. The apparatus of any one of claims 22 through 24, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) comprises: initiating the radar sensing in response to estimating (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).

28. The apparatus of claim 26, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) further comprises: directing a radar beam in the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).

29. The apparatus of any one of claims 22 through 24, wherein using (405, 501, 605) radar sensing to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301) is performed prior to detecting (401, 507, 601) the instance of the reference sound (115, 241).

30. The apparatus of claim 29, wherein the apparatus is further configured to perform: deciding, for each one of the one or more detected points of reflection, whether said each one of the one or more detected points is a candidate source of the reference sound (115, 241).

31. The apparatus of claim 30, wherein the apparatus is further configured to perform: using gesture recognition to decide, for said each one of the one or more detected points of reflection, whether said each one of the one or more detected points is the candidate source of the reference sound (115, 241).

32. The apparatus of any one of claims 29 through 31, wherein the apparatus is further configured to perform: detecting an instance of a first user action; and initiating the radar sensing in response to the detected instance of the first user action.

33. The apparatus of claim 32, wherein the first user action is a reference head movement sensed by the extended reality device (103, 203, 301).

34. The apparatus of claim 32, wherein the first user action is a sensed user gaze in a reference direction toward the viewing area (221) of the extended reality device (103, 203, 301).

35. The apparatus of any one of claims 32 through 34, wherein the apparatus is further configured to perform: detecting a Doppler shift of radar data collected by the radar sensing; detecting that the Doppler shift of the radar data collected by the radar sensing matches a reference Doppler shift of radar data collected from snapping fingers of a user; and using the detecting that the Doppler shift of the radar data collected by the radar sensing matches the reference Doppler shift of radar data collected from snapping fingers of the user as a filter for deciding whether the instance of the reference sound (115, 241) is detected.

36. The apparatus of any one of claims 29 through 35, wherein the apparatus is further configured to perform: storing radar data in a buffer; and only after detecting (401, 507, 601) the instance of the reference sound (115, 241), processing the stored radar data to detect the one or more points of reflection in the real world environment outside of the extended reality device (103, 203, 301).

37. The apparatus of any one of claims 22 through 36, wherein the extended reality device (103, 203, 301) comprises an array of microphones, and wherein estimating (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) comprises using sound information sensed by two or more microphones in the array of microphones.

38. The apparatus of any one of claims 22 through 37, wherein the apparatus is further configured to perform: incorporating, into a mapping of the real world environment, the detected one or more points of reflection in the real world environment.

39. The apparatus of claim 38, wherein incorporating, into the mapping of the real world environment, the detected one or more points of reflection in the real world environment comprises: adjusting the one or more points of reflection in the real world environment to compensate for a movement of the extended reality device (103, 203, 301) that occurred after the radar sensing.

40. The apparatus of claim 39, wherein the apparatus is further configured to perform: using an inertial measurement unit (313) to sense the movement of the extended reality device (103, 203, 301).

Images