Position designation and virtual object placement in extended reality environments
By detecting reference sound and radar sensing, combined with line-of-sight direction and distance information, the problem of coordinate recognition for XR devices in the absence of an outward-facing camera was solved, enabling the precise placement of virtual objects at the user's desired location.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)
- Filing Date
- 2023-12-11
- Publication Date
- 2026-07-21
Smart Images
Figure CN122439104A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the designation of a location in an extended reality (XR) environment, the placement of virtual objects in an XR environment, the user designation of a location in an XR environment, and the placement of virtual objects at a user-designated location in an XR environment. Background Technology
[0002] The following abbreviations are used in this specification, in part or in full:
[0003] Abbreviation Explanation
[0004] Augmented Reality (AR)
[0005] DoA Arrival Direction
[0006] IMU (Inertial Measurement Unit)
[0007] MR Mixed Reality
[0008] RADAR Radio Detection and Ranging
[0009] SLAM Synchronous Localization and Mapping
[0010] VR Virtual Reality
[0011] XR Extended Reality
[0012] The term "Extended Reality" (XR) is a general term encompassing Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) technologies. In each of these technologies, users are able to experience immersion in a computer-generated environment. While VR typically refers to a completely computer-generated environment, in AR, MR, and XR, users experience a fusion of the real-world environment and the computer-generated environment. For example, a user wearing an AR device while sitting at their desk might be able to see not only real-world objects such as a phone, notebook, and pen, but also computer-generated (i.e., virtual) digital objects such as a virtual vase with flowers and a virtual computer screen. In short, when using AR / XR devices (such as AR / XR headsets), users see their surrounding real-world environment, while the technology adds perceptible additional content on top of that reality.
[0013] Such devices typically use one or more outward-facing cameras to analyze the user's surroundings (e.g., for object detection). The device may also include eye-tracking capabilities to detect where the user's attention and eyes are focused in the environment (e.g., by generating a so-called gaze heatmap, which statistically shows the duration the user spends looking at each part of the visual environment).
[0014] By overlaying images and / or text of objects configured to create an illusion of a particular viewing position and distance, based, for example, on environmental knowledge already acquired by the device from images captured by a camera, the aforementioned functionality allows additional artificial objects and / or text to be added to the end-user's view. These images are analyzed to determine how to adjust the image so that, when overlaid on the user's real-world view, it will appear to exist at a specific location and viewing distance in the real world (e.g., above a flat surface).
[0015] Digital objects encompass a wide range of media elements, including 3D models, images, sound, video, and interactive assets, representing anything from simple static geometry to complex, realistic, dynamic characters or objects. Digital objects play a crucial role in enriching user experiences because they provide building blocks for immersive environments and interactive scenarios. Users can create, manipulate, and share digital objects, enabling collaborative and participatory experiences that transcend the limitations of the physical world. In the context of the metaverse and AR, digital objects serve as the foundation for a variety of applications such as gaming, education, business, and communication, transforming how we interact with and perceive the digital realm.
[0016] Users might want to place digital objects near themselves within a virtual environment. Typically, users will want to place digital objects in specific locations, for example, to view the object or to show it to other people sharing the virtual space. In this context, location refers to coordinates in a reference coordinate system created or used by the user. For AR glasses, this location could be coordinates in a SLAM-built map.
[0017] There are two steps involved:
[0018] 1. Interpret the user's intent in placing digital objects.
[0019] 2. Determine the coordinates of the intended placement.
[0020] For some XR devices, outward-facing cameras and microphones can be used to perform these steps, for example, using voice commands for step 1 and pointing with a finger to perform step 2.
[0021] The inventors of the technology described herein have recognized that conventional technologies, such as those mentioned above, have limitations and problems. One reason is related to the fact that, for various reasons, not all future XR headsets may be equipped with outward-facing cameras. This reason may stem from privacy concerns, where individuals near XR devices would not accept being surrounded by continuously moving cameras from such devices. There are documented instances where users wearing XR devices have been denied entry to some commercial establishments because the device represents a form of uninterrupted recording.
[0022] Furthermore, the European Commission has published a document entitled "Regulations by the European Parliament and the Council on Unified Rules on Artificial Intelligence (AI Act) and Amendments to Certain Legislative Acts of the Union." This proposal categorizes AI systems into different classes related to their social risks, with each class facing different levels of regulation. One class, "unacceptable risks," will be subject to strict regulation or even prohibition. This category includes real-time and remote biometric systems, which can include facial recognition systems in public places. Examples of "high-risk" systems, which will be subject to strict regulation, include surveillance systems (such as biometric surveillance for law enforcement, facial recognition systems). One consequence of this is a proposal to ban facial recognition in public places. Overall, such regulations may restrict the use of AR glasses / devices with specific functions in public places.
[0023] To gain market appeal for XR devices, alternative solutions not based on outward-facing cameras are needed. From a technical perspective, one key challenge is how to project augmented reality content within an XR device so that the end-user experience adapts the augmented content to the real-world environment, without relying on cameras and image recognition / analysis for projection to fit within the surroundings. This can be challenging when, for example, added text or objects need to be perfectly aligned with the appropriate viewing distance and surface.
[0024] Eye tracking using an eye-facing camera has been in use for some time. This technology can determine the direction of gaze with an accuracy of up to 0.7 degrees. If the device has a stereo camera that monitors both eyes, it can perform distance measurements based on the intersection of the gaze directions of the eyes. However, given the close distance between the eyes, this distance estimation is only acceptable within a meter, and at greater distances, this gaze direction accuracy is insufficient for accurate distance estimation. For low-cost systems with only a single camera (monitoring only one eye), eye tracking is not a viable method for accurate distance estimation.
[0025] In view of the above, there is a need for technologies that address the aforementioned and related problems, including technologies that allow XR devices to accurately identify coordinates in an XR environment without relying on an outward-facing camera system, so as to place digital objects at locations desired by the user. Summary of the Invention
[0026] It should be emphasized that, when used in this specification, the terms "comprising" and "including" are used to specify the presence of the said feature, integer, step, or component, but do not exclude the presence or addition of one or more other features, integers, steps, components, or combinations thereof.
[0027] In addition, reference marks may be provided in some instances (such as in the claims and the summary of the invention) to facilitate the identification of individual steps and / or elements. However, the use of reference marks is not intended to presume or suggest that the marked steps and / or elements shall be performed or operated in any particular order.
[0028] According to one aspect of the invention, the above and other objectives are achieved in techniques (such as methods, apparatus, non-transitory computer-readable storage media, program components) for identifying a user-specified location in an extended reality environment, wherein the extended reality environment is displayed in the viewing area of an extended reality device. The actions performed therein include: detecting instances of reference sound and estimating the direction of arrival of the detected instances of reference sound. Furthermore, radar sensing via radio signal transmission and reception is used to detect one or more reflection points in the real-world environment outside the extended reality device, wherein each of the one or more reflection points includes a reflection direction from the extended reality device and a reflection distance from the extended reality device. The estimated direction of arrival of the detected instances of reference sound and the one or more reflection directions of the corresponding reflection points of the one or more reflection points are used to select a reflection point from the one or more reflection points. The selected reflection point is used as the basis for determining the user-specified location in the extended reality environment.
[0029] In another aspect of embodiments consistent with the present invention, but not necessarily all, the action includes: placing the virtual object in the extended reality environment at an extended reality location corresponding to a selected reflection point in the real-world environment.
[0030] In another aspect of embodiments consistent with, but not necessarily all, of the present invention, the action includes: displaying a virtual object in the viewing area of the extended reality device.
[0031] In yet another aspect of the invention, which is consistent with some, but not all, embodiments, reference sound is one of the following:
[0032] • The sound of snapping fingers;
[0033] • Sounds associated with tapping, striking, or scratching on surfaces in the real world; and
[0034] • The sound of sound-generating devices located in the real-world environment.
[0035] In another aspect of embodiments consistent with the present invention, but not necessarily all, using radar sensing to detect one or more reflective points in a real-world environment outside an extended reality device includes: activating radar sensing in response to the detection of an instance of a reference sound.
[0036] In yet another aspect of embodiments consistent with the present invention, but not necessarily all, using radar sensing to detect one or more reflective points in a real-world environment outside an extended reality device includes: initiating radar sensing in response to estimating the direction of arrival of the detected instance of a reference sound.
[0037] In some, but not all, embodiments consistent with the present invention, using radar sensing to detect one or more reflection points in a real-world environment outside an extended reality device includes: directing a radar beam in the estimated direction of arrival of the detected instance of a reference sound.
[0038] In another aspect of embodiments consistent with, but not necessarily all, of the present invention, radar sensing is used to detect one or more reflection points in the real-world environment outside the extended reality device before an instance of a reference sound is detected.
[0039] In some, but not all, of these embodiments, the action includes: for each of one or more detected reflection points, determining whether each of the one or more detected points is a candidate source of the reference sound.
[0040] In another aspect of some, but not all, of such embodiments, the action includes: using gesture recognition to determine, for each of one or more detected reflection points, whether each of the one or more detected points is a candidate source of the reference sound.
[0041] In another aspect of some, but not all, of such embodiments, the action includes: detecting an instance of a first user action; and in response to the detected instance of the first user action, initiating radar sensing. In a variety of alternatives and without limitation, the first user action is one or more of the following: a reference head movement sensed by the extended reality device; and a sensed user gaze in a reference direction toward the viewing area of the extended reality device.
[0042] In another aspect of some, but not all, of such embodiments, the action includes: detecting a Doppler shift in radar data collected by radar sensing; detecting a match between the Doppler shift in the radar data collected by radar sensing and a reference Doppler shift in radar data collected from a user's snap of the fingers; and using the match between the detected Doppler shift in the radar data collected by radar sensing and the reference Doppler shift in radar data collected from a user's snap of the fingers as a filter for determining whether an instance of a reference sound has been detected.
[0043] In another aspect of embodiments consistent with the present invention, but not necessarily all, the action includes: storing radar data in a buffer; and processing the stored radar data only after an instance of a reference sound is detected to detect one or more reflection points in a real-world environment outside the extended reality device.
[0044] In another aspect of embodiments consistent with, but not necessarily all, of the present invention, the extended reality device includes a microphone array, and estimating the direction of arrival of a detected instance of reference sound includes using sound information sensed by two or more microphones in the microphone array.
[0045] In another aspect of embodiments consistent with the present invention, but not necessarily all, the action includes incorporating one or more detected reflection points in the real-world environment into a mapping of the real-world environment.
[0046] In some, but not all, alternative embodiments of this type, incorporating one or more detected reflection points in the real-world environment into the mapping of the real-world environment includes: adjusting one or more reflection points in the real-world environment to compensate for movement of the real-world device that occurs after radar sensing. In some, but not all, alternative embodiments of this type, action includes: using an inertial measurement unit to detect movement of the extended real-world device. Attached Figure Description
[0047] The objects and advantages of the invention will be understood by reading the following detailed description in conjunction with the accompanying drawings, in which:
[0048] Figure 1 The illustration shows a user wearing an XR device configured as a head-mounted display, and according to some embodiments of the invention, the user is able to view multiple parts of an XR environment through the XR device.
[0049] Figure 2 An XR device according to some embodiments of the present invention is shown, the XR device having a viewing area with stereoscopic capability for viewing at least a portion of an XR environment.
[0050] Figure 3 This is a block diagram of a non-limiting exemplary XR device configured to perform the actions of the present invention.
[0051] Figure 4 This is a flowchart of actions performed by an XR device according to some, but not necessarily all, embodiments of the present invention in one aspect.
[0052] Figure 5 This is a flowchart of actions performed by an XR device according to some, but not necessarily all, embodiments of the present invention in one aspect.
[0053] Figure 6 This is a flowchart of actions performed by an XR device according to some, but not necessarily all, embodiments of the present invention in one aspect.
[0054] Figure 7 An exemplary controller that may be included in an XR device is shown, and the controller is configured to perform any and / or all of the actions described and illustrated herein that are associated with the device. Detailed Implementation
[0055] The various features of the invention will now be described with reference to the accompanying drawings, in which the same parts are labeled with the same reference numerals.
[0056] Various aspects of the invention will now be described in more detail with reference to several exemplary embodiments. For ease of understanding, the various aspects of the invention are described as sequences of actions performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be appreciated that in each embodiment, the actions may be performed by dedicated circuitry (e.g., analog and / or discrete logic gates interconnected to perform dedicated functions), by one or more processors programmed with a suitable instruction set, or by a combination of both. Throughout this document, the term “circuit configured to perform one or more of the described actions” is used to refer to any such embodiment (i.e., one or more dedicated circuits alone, one or more programmable processors, or any combination thereof). Furthermore, the invention can also be considered as being embodied entirely in any form of non-transitory computer-readable carrier, such as a solid-state memory, disk, or optical disk containing a suitable computer instruction set that will cause a processor to perform the techniques described herein. Therefore, various aspects of the invention can be embodied in a variety of different forms, and all such forms are contemplated as being within the scope of the invention. For each aspect of the invention, embodiments of any such form as described above may be referred to herein as “logic configured to perform the actions,” or alternatively as “logic performing the actions.”
[0057] Embodiments consistent with the present invention address the related problem of allowing XR devices to accurately identify coordinates in an XR environment without relying on an outward-facing camera system, in order to, for example, place digital objects at a location desired by the user.
[0058] In one aspect of an embodiment consistent with the present invention, a predetermined sound (e.g., a snap of fingers or a sound produced by tapping on an object) is identified by the XR device, and the direction of arrival (DoA) of the sound is detected. A radar scan of the XR environment is performed, and real-world (i.e., physical) objects located within the detected direction of arrival are identified. The radar scan also allows for the identification of the distance between the XR device and real-world objects. The line-of-sight direction combined with this distance identifies a user-specified location within the XR environment.
[0059] In another aspect of some, but not all, embodiments of the invention, the detection of a predetermined sound is used as a trigger event to initiate the above-described process.
[0060] In one aspect of some, but not all, alternative embodiments, a radar scan is performed prior to the occurrence of a predetermined sound, thereby allowing information about real-world objects in the XR environment to be collected in advance.
[0061] In another aspect of embodiments of the invention, but not necessarily all, sound may be recorded before a predetermined sound occurs. For example, this can help establish a baseline reference for background noise (typically in the frequency domain), which is considered noise. In some embodiments, recording is limited to a short recognition buffer time interval.
[0062] In another aspect of some, but not all, embodiments of the present invention, the position of a point in three-dimensional space within an XR environment is used for the purpose of placing virtual objects or for subsequent processing of positioning points within the XR environment.
[0063] These and further aspects of the embodiments of the present invention are described below.
[0064] Figure 1 A user 101 is shown wearing an XR device 103 configured as a head-mounted display, through which the user can view a portion of an XR environment 105 (depending on the orientation and pose of the XR device 103). The XR environment 105 contains real-world (i.e., physical) objects 107 having any number of surfaces, such as surface 109. To illustrate various aspects of this embodiment, it is assumed that the user 101 wishes to place a digital (i.e., virtual) object 111 above the real-world object 107. To create a realistic rendering of the digital object 111, the rendering of the object should be consistent with the user's expectations regarding the position and size of the digital object 111. In some cases, the pose of the digital object 111 (e.g., the orientation and tilt angle of one or more surfaces of the object) may also be important. To configure correct rendering as described above, the expected perceived position and distance 113 of the digital object 111 relative to the XR device 103 need to be known. Since it is expected that the digital object 111 will be rendered above the real-world object 107, the position of the real-world object 107 and the distance 113 to the real-world object 107 can be used as alternative representations of those features of the digital object 111.
[0065] In one aspect of the invention, both location and distance information can be obtained by detecting a reference sound 115 and determining its direction of arrival 116. Furthermore, the XR environment 105 is scanned using a radar signal 117, and its radar reflection signal 119 is received. A radar unit equipped with a radar signal transmitter and receiver may be located in the XR device. An object detected by the radar in the direction of arrival of the detected reference sound 115 is considered the source of the sound. The time delay between the transmission of the radar signal 117 and the reception of the radar reflection signal 119 from the object corresponds to the round-trip distance (i.e., twice the distance) between the radar unit in the XR device 103 and the surface of the real-world object 107 reflecting the radar signal 117. Since the orientation and distance information of the real-world object's location within the XR environment 105 are now known, the image of the digital object can be adjusted such that its apparent position and apparent size at that location create a realistic user experience where the digital object 111 actually exists within the XR environment 105. To compensate for subsequent movement of the XR device 103, the XR device 103 senses the movement (e.g., via gyroscope or inertial measurement unit (IMU) technology) and continuously adjusts the rendering of the digital object 111 so that the perceived placement at that location appears well anchored in the XR environment.
[0066] Now for reference Figure 2 Further aspects related to embodiments of the invention are discussed. In this non-limiting example, the XR device 203 has a viewing area 221 with stereoscopic capabilities, and thus the viewing area 221 has two parts, each dedicated to being viewed by a corresponding eye of a pair of user eyes 201. By looking at or through the viewing area 221, the user 101 can see at least a portion of the XR environment 205.
[0067] In this example, the XR environment 205 includes a real-world surface 209 of the object (e.g., the surface could be a desktop, tabletop, countertop, shelf top, etc.). Furthermore, in this example, the user's gaze 223 is directed towards a real-world location 225 above surface 209. In this stereoscopic example, gaze monitoring for each eye will detect the display location (x, y) that the user's left eye is viewing within the portion of the viewing area 221 allocated to the left eye. 左 y 左 ) part or through the display location (x) 左 y 左 The user's right eye is viewing the display position (x) within the viewing area 221 allocated to the right eye. 右 y 右 ) or via the display location (x) 右 y 右 Viewing. Typically, coordinate pairs (x...) 左y 左 ) and (x 右 y 右 They are not equal. In this example, the line of sight 229 can be determined by finding the intersection of the left and right line of sight 223. Figure 2 In the example, this is shown as a real-world location 225 with x, y, z coordinates (in this case, the third coordinate z represents depth).
[0068] Since digital (virtual) objects do not actually exist in the real world, they have no real-world location; they can only be seen within the viewing area 221 of the XR device 203. However, by creating left and right images at corresponding display positions within the viewing area 221 aligned with the user's corresponding left and right viewing directions 223, the user 101 will perceive the digital object 211 as being located at an apparent position 227 in the XR environment 205, because that is where the left and right viewing directions 223 intersect. For example, the vector from the midpoint of the XR device 203 to the apparent position 227 thus corresponds to the viewing direction 229. For convenience, the convention adopted in the examples described herein will assign the same reference coordinate system to both the real-world and apparent positions so that the x, y, z coordinates will be the same regardless of which position is referred to (“real-world” or “apparent”). However, this is not a necessary aspect of all embodiments. Instead, different coordinate systems can be used for both the real-world and apparent positions. Furthermore, it should be noted that, in order to further enhance the effect that the digital object 211 will be perceived as being located at the apparent position 227 in the XR environment 205, the light from the digital object 211 should appear to come from a distance corresponding to the apparent position 227, so that when its lens is adjusted to that distance, the eye will focus the light on the retina, and for that reason, an adjustable projection system with lenses can be used in the glasses.
[0069] It should also be noted that the aspects found in the embodiments of the present invention are not limited to stereoscopic XR devices. Rather, aspects of the invention also exist in monocular embodiments, and in embodiments utilizing the stereoscopic viewing area 221 but monitoring the gaze of only one eye of the user. In these later embodiments, the gaze direction 229 is based solely on the gaze direction of the monitored eye.
[0070] In a further aspect, and as previously mentioned, the radar scan of the XR environment 205 is combined with reference sound detection to identify a specific real-world location 225. Thus, for example, as... Figure 2As shown, reference sound 241 is generated by a user snapping their fingers near a desired location in XR environment 205 (in a non-limiting example). In another non-limiting example, a user may tap or scratch surface 209 in XR environment 205. The direction of arrival 243 of the sound waves reaching XR device 203 is determined. Radar scanning of real-world surfaces in XR environment 205 also produces reflections from surfaces in the XR environment, including the surfaces of objects associated with reference sound 241. For example, in the case of a finger snap, this could be generated at any location within reach of user 101, and a radar system could detect radar reflections from user's hand 245. In another non-limiting example, where the sound is a finger snap occurring near surface 209 in XR environment 205, or where the reference sound is generated by interaction with surface 209 in XR environment 205 (e.g., by tapping or scratching), a radar reflection from surface 209 can be detected by a radar system, and its direction of arrival 231 is determined. In each case, the direction of arrival 231 of the radar reflected signal is substantially the same as the direction of arrival 243 of the detected reference sound 241. Therefore, the radar system can determine the distance to the radar signal reflection point 247, and this distance can be used as the distance to the location where the reference sound 241 was generated. The direction of arrival 243 of the reference sound and the determined distance thus indicate a user-specified location within the XR environment 205.
[0071] In some, but not all, embodiments, the user's gaze can also be tracked as described above. If the user is looking at real-world location 225 while reference sound 241 is being generated, the detected gaze direction 229 will be substantially the same as the direction of arrival of the reference sound 243 and also substantially the same as the direction of arrival of the radar reflection signal 231. Therefore, gaze direction can be further used in some embodiments.
[0072] Figure 3 This is a block diagram of a non-limiting exemplary XR device 301 configured to perform actions according to the invention. The exemplary XR device 301 includes:
[0073] - Optical unit 303: It includes a viewing area 305 through which the user can see a portion of the XR environment. The optical unit can also overlay computer-generated digital objects within the viewing area 305.
[0074] - Eye Tracker 307: It monitors the direction of the user's gaze in one or both eyes.
[0075] - Radar circuitry 309 or equivalent radar functionality. As an example of the latter, XR device 301 may include modem 311 for wireless communication with, for example, a wireless communication network. Such modem 311 typically operates at frequencies suitable for radar operation. Therefore, modem 311 can be configured to operate as a radar device suitable for use in embodiments of the invention. Using a suitable radar signal bandwidth (e.g., approximately 1 GHz to give a radar range that allows resolution of two objects spaced 15 cm apart), multiple radar transceivers (or...) spaced several decimeters apart... Figure 3 The radar transceiver shown with antenna array 319 can be used to achieve centimeter-level distance estimation of objects around the XR device 301.
[0076] - Inertial Measurement Unit (IMU) 313: Used to track any movement of the XR device 301. This movement information can be used to stabilize the image presented in the viewing area 305, and can also be used as a basis for determining adjustments to the rendered image of digital objects so that the digital objects will appear to have a stable position when the user wearing the XR device 301 moves around.
[0077] - Microphone array 315: It can be used for a variety of purposes, including but not limited to receiving voice commands and information from a user. By including at least two microphones in the microphone array 315, and given a known spacing between the microphones, the time difference between signals collected by the respective microphones of the plurality of microphones can be used as an indicator of the direction of arrival of the sensed sound.
[0078] - Controller 317: Used to control the above-mentioned and other components of the XR device 301. Controller 317 may consist of hard-wired circuitry, a processor / component controlled by programmable software, or a combination of both.
[0079] It should be noted that while XR device 301 is configured to perform an exemplary embodiment of an embodiment of the present invention, not all instances of the invention require the presence of every shown element of XR device 301. As a non-limiting example, neither eye-tracking nor inertial measurement is required in all embodiments consistent with the invention, although each may be useful in some instances of the invention. The absence or presence of such elements will be apparent to those skilled in the art based on whether the functionality provided by such elements is required to complete a particular embodiment.
[0080] Reference Figure 4 Further aspects of at least some embodiments of the present invention are described, in one aspect, Figure 4 This is a flowchart of the actions performed by the XR device to determine the desired location within the XR environment. In other aspects, Figure 4The boxes depicted can also be considered as representing components 400 (e.g., hardwired or programmable circuitry or other processing components) used to perform the actions.
[0081] In the initialization aspect of this embodiment of the invention, the XR device is pre-configured (not shown) with one or more reference sound triggers, which are registered within the device for use by the device to monitor when the actions described herein are performed. During the execution of this process, a microphone array is used for audio monitoring, and once an instance of a reference sound trigger is detected (step 401), signals generated by the multiple microphones are used to determine the direction of arrival of the sound waves (step 403).
[0082] In embodiments where the process is triggered by the identification of a reference sound, it is advantageous that the microphone is always active. Identification of a reference sound trigger may include matching the sensed sound envelope with the envelope of a reference sound trigger (e.g., a sound envelope characterized by rapid rises and falls in sound pressure levels, such as a snap or tap). Alternatively (or additionally), for reference sound triggers involving repetitive sounds (e.g., repeated taps or knocks), the sound repetition rate may be matched with a desired rate (e.g., 100 to 400 times per minute). Such a strategy is advantageous, at least in that the processing requires only a low power level. Once the envelope is matched, a second step may be performed to provide improved accuracy, such as converting the time-domain sound signal to the frequency domain and then detecting the presence of a specific known feature (depending on what the reference sound trigger is). In the first and / or second steps, spatial filtering (e.g., beamforming) may be applied to better separate the sound event from background noise.
[0083] Once the direction of arrival is determined, the radar transceiver is activated to sense at least a portion of the real world within the XR environment. In some embodiments, a general radar scan may be performed on a portion of the XR environment. In some, but not necessarily all, alternative embodiments, the radar scan may be limited to an area of the XR environment in the direction of arrival of the sound waves. Regardless of the type of radar scan performed, one or more real-world objects (i.e., at least the object that generated the reference sound) will be detected, and the object associated with the reference sound will be detected by comparing the direction of arrival of the sensed sound waves with the direction of arrival of the radar echo (step 405). The time delay between the transmission of the radar signal and the reception of the radar reflection signal from the object associated with the reference sound (i.e., the time of flight of the radar signal) corresponds to the round-trip distance between the XR device and the surface of the object (i.e., twice the actual distance), and from this information, the distance to the object is estimated (step 409). This is used to estimate the distance to the source of the sound. Given the distance between the XR device and the radar-sensed object, and also given the direction from the XR device to the sound-generating object, a user-specified location within the XR environment is now known (step 411). This location may be specified using the XR device as a reference coordinate system. It will be readily understood by those skilled in the art that this position can also be transformed to any other reference coordinate system, which would be more desirable in any given application.
[0084] The location determined in this way can then be directly or in some instances corrected to incorporate into the adopted environment mapping, the correction being used to compensate for user equipment movement (the user's head is one such device arrangement), or the correction compensating for changes in orientation detected by IMU sensing or similar sensors. In an optional step, present in some, but not all, embodiments, a visual representation (e.g., visual highlighting) of the location is presented to the end user within the terminal user interface (e.g., within viewing area 221) (step 413). In some, but not all, embodiments, the purpose of the visual representation is to project the target orientation as a virtual object into the XR environment, allowing the user to consider the detected location and (through several user interactions) verify acceptance of the estimated location (step 415). If the user indicates rejection of the detected location, the process can be restarted.
[0085] Depend on Figure 4 The exemplary embodiment shown illustrates a category based on instances of predetermined sounds being used as triggers to activate radar scanning. In alternative embodiments, this is not the case, and... Figure 4 The actions shown may be performed in different orders, and / or additional or fewer steps may be implemented. To illustrate one such non-limiting variation, reference is now made to... Figure 5 , Figure 5One aspect is a flowchart of an action performed by an XR device to determine a desired location within the XR environment. In other aspects, Figure 5 The depicted box can also be regarded as a component 500 (e.g., a hardwired or programmable circuit or other processing component) performing the action.
[0086] In the initialization aspect of this embodiment, the XR device is pre-configured (not shown) with one or more reference sound triggers, which are registered within the device for monitoring when the action described herein is performed. During the execution of this process, radar continuously operates to detect real-world objects in the XR environment that may potentially be sources of the reference sound (step 501). In some embodiments, not all detected real-world objects are considered sound source candidates. To reduce subsequent processing of radar signal information that cannot be sourced from reference sounds, false positives (i.e., detected objects that cannot be used as sources of reference sounds, such as, for example, distant walls or other objects that the user cannot interact with if the reference sound is generated by such an interaction) are filtered out. In a non-limiting example, gesture recognition or similar classification methods known in the art can be used as the basis for such filtering. Since this information is not used until the reference sound appears, the XR device stores information about these candidate objects.
[0087] A position estimate is created for each of these candidate objects (step 503). The position can be estimated from, for example, the angle of arrival of the radar reflected signal (which indicates the direction) and the flight time (round-trip time) from the transmission of the radar signal to the reception of the radar echo (which corresponds to twice the distance between the XR device and the reflecting object).
[0088] In some embodiments, microphone array 315 is always enabled and monitored so that it will be detected whenever a reference sound occurs. In some alternative embodiments, microphone array 315 is initially kept in an inactive state (e.g., off or low power), and in optional steps ( Figure 5In the dashed line (indicated in the middle), radar circuitry 309 is used to detect the possible occurrence of a sound event (decision box 505). Detection can be achieved, for example, by detecting radar Doppler events that match characteristic attributes of a sound event (e.g., a movement associated with a snap of the fingers or tapping on an object). The radar and associated processing can make this determination before the sound wave associated with the event reaches the microphone. Therefore, if no such event is detected (the "No" path of decision box 505), the processing returns to step 501. However, if a potential sound event is detected (the "Yes" path of decision box 505), microphone array 315 is switched to an active state to receive the expected sound wave. Furthermore, in some, but not all, such embodiments, the radar processing also determines the direction in which the sound event occurs (relative to XR device 301), and this direction information is used to control microphone array 315 processing so that it listens for the sound wave in the expected direction of arrival. In this way, XR device 301 avoids expending energy trying to find a reference sound in other directions. In some further alternative embodiments, the duration of the listening activity may also be limited to a time window in which sound waves are expected to be received (based on the distance between the detected Doppler-generating event and the XR device 301).
[0089] When no reference sound is detected (the "No" path in decision box 507), the process simply repeats the radar scan of step 501. In this respect, it should be noted that the continuous repetition of radar actions allows the device to maintain up-to-date information about the detected candidates. This information changes dynamically due to factors such as device movement (e.g., changes in pose or position within the XR environment) and movement of candidate sound sources (e.g., a user's hand that could potentially produce a snap without interaction with any other object).
[0090] When a reference sound is detected (the "Yes" path in decision box 507), the sensed audio information is processed to determine which of the radar-detected candidate objects (if any) is a possible source of the reference sound (decision box 509). In some embodiments, this step includes determining the angle of arrival of the sound wave (e.g., by detecting the time difference in the audio signals generated by each microphone in microphone array 315). By comparing the direction of arrival of the reference sound with the radar-detected candidate sound sources, a match (in some embodiments, allowing a certain tolerance for deviation) results in the matched radar-detected candidate sound source being identified as the true source of the reference sound. In an alternative embodiment involving step 505, the candidate object is already known because the radar processing detected that object as the source of the sound event.
[0091] Therefore, if no source of the reference sound is found from the candidate sound sources (the "No" path in decision box 509), the process repeats the radar scan at step 501.
[0092] When the source of the reference sound is identified (the “Yes” path in decision box 509), some or all aspects of the location estimate of the sound source candidate (i.e., distance and / or direction) are used as the basis for estimating the location of the reference sound source in the currently used map (step 511).
[0093] As a reference Figure 4 The categories of embodiments described, where the location is determined in this way, may then be directly or in some instances modified to incorporate into the adopted environment mapping, the modification being used to compensate for user equipment movement (the user's head being one such device arrangement), or the modification compensating for changes in orientation detected by IMU sensing or similar sensors. In an optional step, present in some, but not all, embodiments, a visual representation (e.g., visual highlighting) of the location is presented to the end user within the terminal user interface (e.g., within viewing area 221) (step 513). In some, but not all, embodiments, the purpose of the visual representation is to project the target orientation as a virtual object into the XR environment, allowing the user to consider the detected location and (through several user interactions) verify acceptance of the estimated location (step 515). If the user indicates rejection of the detected location, the process can be restarted.
[0094] In a further alternative embodiment, the enhancement of the XR device pose / posture detection is achieved by using gaze direction 243 and XR device pose / posture detection. Figure 5 The aspects described. For example, when an XR device is instantiated as a head-mounted device, the radar may remain in standby or low-power mode, and then be briefly activated to perform [operations] when the user of the XR device turns their head and / or directs their gaze in a predetermined direction (e.g., looking slightly down, close to the front of their body). Figure 5 The action shown is as described. This aspect will save the device's energy consumption and will alleviate congestion on the frequency band used by the radar.
[0095] In another aspect of some, but not necessarily all, embodiments of the invention, and further reference is made to... Figure 5 Energy efficiency can be achieved by storing the raw radar data in a buffer instead of processing it during collection. Radar data processing can only begin after an instance of the reference sound has been detected, as in step 503. For example, in the case of a finger snap as the reference sound, if it is assumed that the snap can be performed in approximately 10 milliseconds and the corresponding sound wave arrives at the XR device in approximately 1 to 4 milliseconds, then a buffer storing 15 to 20 milliseconds of data might be sufficient.
[0096] In another aspect of some, but not necessarily all, embodiments of the invention, and further reference is made to... Figure 5One alternative embodiment of the XR device for head-mounted devices includes: detecting when a user turns their head or gazes in a specific direction, and in response to this detection, briefly activating an onboard radar, and determining from the received radar reflection signal whether a Doppler shift associated with the user's snap has been detected. If detected, the XR device anticipates that a reference sound should be imminent, further establishing intent to invoke the feature. More specifically, a near-field impression of the Doppler radar response initiates a timer, which is set for a duration within which the XR device should detect the corresponding slower-propagating sound wave. If no sound wave is received within this time window (i.e., before a timeout event from the timer), the XR device does not perform any further action. The advantage of this aspect is that it avoids other user devices detecting and responding to the reference sound as if the sound had already been triggered by users of those other devices. This is because if other users are outside the range of intent (e.g., within the user's arm's length), the predetermined sound will arrive at the other users' XR devices too late (i.e., outside the timeout period) compared to radar detection. For illustrative purposes, the features just described can be used as... Figure 5 This is achieved as part of decision box 507 (i.e., determining whether a reference sound is detected within a time window after the Doppler shift associated with the snap is detected).
[0097] In yet another alternative embodiment, where it is not possible and / or not desirable to detect the direction of arrival of the sound waves (e.g., in the case where the XR device has only a single microphone), the aforementioned aspect (wherein the detection initiation time window of the Doppler radar response associated with the snapping example, within which a predetermined sound needs to be detected) can be used instead. Figure 5 The direction-related aspect of decision box 509 is filtered out because it filters out responses to sounds emitted by sound sources that are not within the specified range.
[0098] In another alternative embodiment, gaze and pose detection can be used for the same purpose, but without detecting the snap itself. More specifically, information from gaze and pose detection is used to filter only the angles and distances consistent with the detected gaze and pose from a distance-angle (-Doppler) two-dimensional (three-dimensional) data cube. Therefore, there is no need to process information from other locations.
[0099] Reference Figure 6 Further aspects of some, but not all, of the embodiments of the present invention are described. Figure 6 One aspect illustrates a flowchart of actions performed by an XR device according to some, but not all, embodiments of the invention. In other aspects, Figure 6 The box shown can also be interpreted as representing a component 600 (e.g., hardwired or programmable circuitry, other processing component) for performing the action.
[0100] exist Figure 6 In an exemplary embodiment, the XR device identifies a user-specified location in the XR environment, wherein the XR environment is displayed in the viewing area of the XR device. The actions include: detecting an instance of a reference sound (step 601); estimating the direction of arrival of the detected instance of the sensed sound wave (step 603).
[0101] In addition, radar sensing (i.e., via radio signal transmission and reception) is used to detect one or more reflection points in the real-world environment outside the XR device (step 605), wherein each of the one or more reflection points includes a reflection direction from the XR device and a reflection distance from the XR device.
[0102] The estimated direction of arrival of the detected instance of the reference sound and one or more reflection directions of the corresponding reflection point among one or more reflection points are used to select a reflection point from one or more reflection points (step 607).
[0103] The XR device then uses the selected reflection point as the basis for determining the user-specified location in the extended reality environment (step 609). As a non-limiting example of some, but not all, embodiments of the invention, the XR device further performs the following: placing the virtual object in the XR environment at an XR location corresponding to the selected reflection point in the real-world environment (step 611). In some instances, the XR location in the XR environment is the selected reflection point in the real-world environment. However, in other instances, the XR location in the XR environment may correspond to but is not equivalent to the selected reflection point, for example, when it is desirable to further adjust the orientation and / or pose of the virtual object to better integrate it into and / or conform to its surroundings. The choice between the two can depend on the application.
[0104] Reference Figure 7 Further aspects of embodiments consistent with the present invention are described. Figure 7 An exemplary controller 701 that may be included in an XR device is shown, which causes any and / or all of the actions described and illustrated herein to be performed in connection with the device. Specifically, controller 701 includes circuitry configured to perform any one or any combination of the various functions described herein. Such circuitry may, for example, be entirely hardwired circuitry (e.g., one or more application-specific integrated circuits – “ASICs”). However, Figure 7The exemplary embodiments depicted herein are programmable circuits including a processor 703 coupled to one or more storage devices 705 (e.g., random access memory, disk drive, optical disk drive, read-only memory, etc.) and coupled to an interface 707 that enables bidirectional communication with other elements of the devices as described above. A complete list of possible other elements is beyond the scope of this specification.
[0105] One or more storage devices 705 store program components 709 (e.g., processor instruction sets), which are configured to cause processor 703 to control other device elements to perform any of the aspects described herein. One or more storage devices 705 may also store data (not shown) representing various constants and variable parameters that may be required by processor 703 and that may be generated when processor 703 performs its functions, such as those specified by the program component 709.
[0106] Compared to conventional techniques, various embodiments consistent with this invention exhibit several advantages and benefits. For example, in some embodiments, digital (virtual) objects can be placed and detected in virtual space without the use of a camera. Instead, a combination of sensors (e.g., radar, microphone) is used, which enables the protection of user privacy in the area. This is beneficial not only for devices without cameras but also for devices that need to comply with privacy restrictions affecting the use of some sensors. An example centered on AR glasses (one type of XR device) is that these glasses are instructed to automatically turn off their cameras when non-consenting persons are nearby or when they are outside a designated area.
[0107] Another advantage of embodiments consistent with this invention relates to privacy protection, which avoids continuous recording of sounds that could potentially infringe on the privacy of the user or bystanders. This problem is addressed in embodiments of the invention, which limit the recording of sound to only the short time interval required to capture a very short reference sound, such as a reference sound produced by a snap of the fingers or tapping on an object (e.g., about 50 milliseconds). Other aspects of embodiments of the invention (e.g., embodiments relying on radar sensing and / or eye tracking) also do not raise privacy concerns.
[0108] The present invention has been described with reference to specific embodiments. However, those skilled in the art will readily understand that the invention may be embodied in other specific forms besides those described above.
[0109] For example, some, but not all, embodiments further include accelerometer- or IMU-based processing for various purposes. For instance, in one example, information about the motion sensed by the XR device based on the accelerometer or IMU is provided, enabling the XR device to utilize dead reckoning to compensate for any device movement occurring in the very brief interval between a predetermined sound occurrence and an estimate of the relative orientation between the sound source and the XR device. In some, but not all, embodiments, the information about the sensed motion of the XR device is used to compensate for subsequent head movement after the relative orientation between the sound source and the XR device has been determined. By adjusting a user-specified position based on device movement, the user can experience that the position (including digital objects placed at that position) as they move around within the XR environment.
[0110] Therefore, the embodiments described are merely illustrative and should not be construed as limiting in any way. The scope of the invention is defined by the appended claims, not just by the foregoing description, and all variations and equivalents falling within the scope of the claims are intended to be included therein.
Claims
1. A method for identifying a user-specified location (225) in an extended reality environment (105, 205), wherein, The extended reality environment (105, 205) is displayed in the viewing area (221) of the extended reality device (103, 203, 301), the method comprising: Detection of instances of reference sounds (115, 241) (401, 507, 601); Estimate (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241). Radar sensing via radio signal transmission and reception (405, 501, 605) is used to detect one or more reflection points (247) in the real-world environment outside the extended reality devices (103, 203, 301), wherein each of the one or more reflection points (247) includes a reflection direction (231) from the extended reality devices (103, 203, 301) and a reflection distance (113) from the extended reality devices (103, 203, 301). Using the estimated arrival direction (116, 243) of the detected instance of the reference sound (115, 241) as described in (607) and one or more reflection directions of the corresponding reflection point among the one or more reflection points, select one of the one or more reflection points; and The reflection point selected at (411, 609) is used as the basis for determining the user-specified location (225) in the extended reality environment (105, 205).
2. The method according to claim 1, comprising: Place the virtual objects (111, 211) in the extended reality environment (105, 205) at the extended reality location corresponding to the selected reflection point in the real world environment.
3. The method according to claim 2, comprising: The virtual object is displayed in the viewing area (221) of the extended reality device (103, 203, 301).
4. The method according to any one of the preceding claims, wherein, The reference sound (115, 241) is one of the following: The sound of snapping fingers; Sounds associated with tapping, striking, or scratching on surfaces in the real-world environment (105, 205); and The sound of a sound-emitting device located in the real-world environment.
5. The method according to any one of the preceding claims, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) includes: In response to the instance detected (401, 507, 601) to the reference sound (115, 241), the radar sensing is initiated.
6. The method according to any one of claims 1 to 3, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) includes: The radar sensing is initiated in response to the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) estimated (403, 603).
7. The method according to claim 5, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) further includes: The radar beam is directed toward the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).
8. The method according to any one of claims 1 to 3, wherein, The use of radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) was performed before the detection of the instance of the reference sound (115, 241) (401, 507, 601).
9. The method of claim 8, comprising: For each of the one or more detected reflection points, determine whether each of the one or more detected reflection points is a candidate source of the reference sound (115, 241).
10. The method of claim 9, comprising: Gesture recognition is used to determine, for each of the one or more detected reflection points, whether each of the one or more detected reflection points is a candidate source of the reference sound (115, 241).
11. The method according to any one of claims 8 to 10, comprising: Detect the instance of the first user action; as well as In response to the detected instance of the first user action, the radar sensing is initiated.
12. The method according to claim 11, wherein, The first user action is a reference head movement sensed by the extended reality devices (103, 203, 301).
13. The method according to claim 11, wherein, The first user action is a sensed user gaze in a reference direction toward the viewing area (221) of the extended reality device (103, 203, 301).
14. The method according to any one of claims 11 to 13, comprising: Detect the Doppler frequency shift of the radar data collected by the radar sensor; The Doppler frequency shift of the radar data collected by the radar sensing is matched with the reference Doppler frequency shift of the radar data collected from the user's snap of the fingers; as well as The Doppler frequency shift of the radar data collected by the radar sensing is matched with the reference Doppler frequency shift of the radar data collected from the user's snap of the fingers as a filter for determining whether the instance of the reference sound (115, 241) has been detected.
15. The method according to any one of claims 8 to 14, comprising: Store radar data in a buffer; Only after detecting (401, 507, 601) an instance of the reference sound (115, 241) is the stored radar data processed to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301).
16. The method according to any one of the preceding claims, wherein, The extended reality device (103, 203, 301) includes a microphone array, and wherein estimating (403, 603) the direction of arrival (116, 243) of a detected instance of the reference sound (115, 241) includes using sound information sensed by two or more microphones in the microphone array.
17. The method according to any one of the preceding claims, comprising: Incorporate one or more detected reflection points in the real-world environment into a mapping of the real-world environment.
18. The method according to claim 17, wherein, Incorporating one or more detected reflection points in the real-world environment into the mapping of the real-world environment includes: Adjust one or more reflection points in the real-world environment to compensate for the movement of the extended reality device (103, 203, 301) that occurs after the radar sensing.
19. The method of claim 18, comprising: An inertial measurement unit (313) is used to sense the movement of the extended reality device (103, 203, 301).
20. A computer program (709) comprising instructions that, when executed by at least one processor (703), cause the at least one processor (703) to perform the method according to any one of claims 1 to 19.
21. A carrier comprising the computer program (1509) of claim 20, wherein, The carrier is one of electronic signals, optical signals, radio signals, and non-transitory computer-readable storage media (705).
22. An apparatus for identifying a user-specified location (225) in an extended reality environment (105, 205), said extended reality environment (105, 205) being displayed in a viewing area (221) of an extended reality device (103, 203, 301), wherein, The device is configured to cause the extended reality device (103, 203, 3014) to perform: Detection of instances of reference sounds (115, 241) (401, 507, 601); Estimate (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241). Radar sensing via radio signal transmission and reception (405, 501, 605) is used to detect one or more reflection points (247) in the real-world environment outside the extended reality devices (103, 203, 301), wherein each of the one or more reflection points (247) includes a reflection direction (231) from the extended reality devices (103, 203, 301) and a reflection distance (113) from the extended reality devices (103, 203, 301). Using the estimated arrival direction (116, 243) of the detected instance of the reference sound (115, 241) as described in (607) and one or more reflection directions of the corresponding reflection point among the one or more reflection points, select one of the one or more reflection points; and The reflection point selected at (411, 609) is used as the basis for determining the user-specified location (225) in the extended reality environment (105, 205).
23. The apparatus according to claim 22, wherein, The device is further configured to perform: Place the virtual objects (111, 211) in the extended reality environment (105, 205) at the extended reality location corresponding to the selected reflection point in the real world environment.
24. The apparatus of claim 23, further configured to perform: The virtual object is displayed in the viewing area (221) of the extended reality device (103, 203, 301).
25. The apparatus according to any one of claims 22 to 24, wherein, The reference sound (115, 241) is one of the following: The sound of snapping fingers; Sounds associated with tapping, striking, or scratching on surfaces in the real-world environment (105, 205); and A sound-generating device located in the real-world environment.
26. The apparatus according to any one of claims 22 to 25, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) includes: In response to the instance detected (401, 507, 601) to the reference sound (115, 241), the radar sensing is initiated.
27. The apparatus according to any one of claims 22 to 24, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) includes: The radar sensing is initiated in response to the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) estimated (403, 603).
28. The apparatus according to claim 26, wherein, Using radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) further includes: The radar beam is directed toward the estimated direction of arrival (116, 243) of the detected instance of the reference sound (115, 241).
29. The apparatus according to any one of claims 22 to 24, wherein, The use of radar sensing (405, 501, 605) to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301) was performed before the detection of the instance of the reference sound (115, 241) (401, 507, 601).
30. The apparatus according to claim 29, wherein, The device is further configured to perform: For each of the one or more detected reflection points, determine whether each of the one or more detected reflection points is a candidate source of the reference sound (115, 241).
31. The apparatus according to claim 30, wherein, The device is further configured to perform: Gesture recognition is used to determine, for each of the one or more detected reflection points, whether each of the one or more detected reflection points is a candidate source of the reference sound (115, 241).
32. The apparatus according to any one of claims 29 to 31, wherein, The device is further configured to perform: Detecting the first instance of user action; and In response to the detected instance of the first user action, the radar sensing is initiated.
33. The apparatus according to claim 32, wherein, The first user action is a reference head movement sensed by the extended reality devices (103, 203, 301).
34. The apparatus according to claim 32, wherein, The first user action is a sensed user gaze in a reference direction toward the viewing area (221) of the extended reality device (103, 203, 301).
35. The apparatus according to any one of claims 32 to 34, wherein, The device is further configured to perform: Detect the Doppler frequency shift of the radar data collected by the radar sensor; The Doppler frequency shift of the radar data collected by the radar sensing is matched with the reference Doppler frequency shift of the radar data collected from the user's snap of the fingers; as well as The Doppler frequency shift of the radar data collected by the radar sensing is matched with the reference Doppler frequency shift of the radar data collected from the user's snap of the fingers as a filter for determining whether the instance of the reference sound (115, 241) has been detected.
36. The apparatus according to any one of claims 29 to 35, wherein, The device is further configured to perform: Store radar data in a buffer; Only after detecting (401, 507, 601) an instance of the reference sound (115, 241) is the stored radar data processed to detect one or more reflection points in the real-world environment outside the extended reality device (103, 203, 301).
37. The apparatus according to any one of claims 22 to 36, wherein, The extended reality device (103, 203, 301) includes a microphone array, and wherein estimating (403, 603) the direction of arrival (116, 243) of the detected instance of the reference sound (115, 241) includes: Use sound information sensed by two or more microphones in the microphone array.
38. The apparatus according to any one of claims 22 to 37, wherein, The device is further configured to perform: Incorporate one or more detected reflection points in the real-world environment into a mapping of the real-world environment.
39. The apparatus according to claim 38, wherein, Incorporating one or more detected reflection points in the real-world environment into the mapping of the real-world environment includes: Adjust one or more reflection points in the real-world environment to compensate for the movement of the extended reality device (103, 203, 301) that occurs after the radar sensing.
40. The apparatus according to claim 39, wherein, The device is further configured to perform: An inertial measurement unit (313) is used to sense the movement of the extended reality device (103, 203, 301).