User indication of location information in shared extended reality environments

Abstract

Identify a user-specified location (225) in an extended reality environment (105, 205), wherein the extended reality environment is displayed in the viewing area (221) of an extended reality device (103, 203, 401). Identification includes: obtaining a self-position (501, 601, 701), wherein the self-position is a location in a shared set of one or more locations of a corresponding known real-world object in the extended reality environment; detecting (505, 703) the user's gaze direction (231) toward the viewing area; and using radar (703) to detect the corresponding distance (113) to one or more sensed real-world objects (107) in the real-world environment. Use (707) the self-position of the extended reality device, the detected user gaze direction (231), and the corresponding distance of one or more sensed real-world objects to select one of one or more known real-world objects. Use (709) the shared set of one or more locations of the corresponding known real-world objects in the extended reality environment to determine the selected location (225) of the selected known real-world object, and use the selected location as the user-specified location in the extended reality environment.

Description

Background Technology

[0001] The present invention relates to location in extended reality (XR) environments, more particularly to location in shared XR environments, and even more particularly to user designation of shared XR environment location information for multiple users in a shared XR environment.

[0002] Some or all of the following abbreviations are used in this specification.

[0003] abbreviation illustrate

[0004] AoA Angle of Arrival

[0005] AR Augmented Reality

[0006] DoA (Direction of Arrival)

[0007] IMU (Inertial Measurement Unit)

[0008] Mixed Reality (MR)

[0009] RADAR Radio Detection and Ranging

[0010] SLAM (Simultaneous Localization and Mapping)

[0011] SNR (Signal-to-Noise Ratio)

[0012] VR Virtual Reality

[0013] XR Extended Reality

[0014] The term "Extended Reality" (XR) is a general term encompassing Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR) technologies. In each of these technologies, users are able to experience immersion in a computer-generated environment. The term VR typically refers to a completely computer-generated environment, while in AR and MR, users experience a blend of real-world and computer-generated environments. As an example, a user sitting at a desk wearing an AR device can see not only real-world objects, such as their phone, notebook, and pen, but also computer-generated (i.e., virtual) digital objects, such as a virtual vase with flowers and a virtual computer screen. In short, when using AR / XR devices (e.g., AR / XR headsets), users can see their surrounding real-world environment while the technology adds perceptible additional content on top of that real environment.

[0015] Such devices typically use one or more outward-facing cameras to analyze the user's surroundings (e.g., for object detection). The device may also include gaze tracking capabilities to detect where the user's attention and eyes are focused in the environment (e.g., by generating a so-called gaze heatmap, which statistically shows how much time the user spends looking at each part of the visual environment).

[0016] The aforementioned functionality allows additional artificial objects and / or text to be added to the end-user view, for example, by overlaying images of the objects and / or text. These images are configured to create the illusion that the object or text is located at a specific viewing position and distance, based on environmental knowledge acquired by the device from images captured by a camera. These images are analyzed to determine how to adjust them so that when overlaid on the user's real-world view, they will appear to exist at a specific location and viewing distance in that real world (e.g., on a flat surface).

[0017] Digital objects encompass a wide range of media elements, including 3D models, images, sounds, videos, and interactive assets, representing anything from simple static geometry to complex, realistic moving characters or objects. Digital objects play a crucial role in enriching user experiences because they provide building blocks for immersive environments and interactive scenarios. They can be created, manipulated, and shared by users, enabling collaborative and participatory experiences that extend beyond the limitations of the physical world. In the context of the metaverse and AR, digital objects serve as the foundation for various applications such as gaming, education, business, and communication, transforming how we interact with and perceive the digital realm.

[0018] Users might want to place digital objects within a virtual environment near themselves. Typically, users will want to place digital objects in specific locations, for example, to inspect them or to display them to others sharing the virtual space. In this context, location refers to coordinates within a frame of reference created or used by the user. For AR glasses, this would be coordinates within a map built using SLAM.

[0019] It involves two steps:

[0020] 1. Explain the user's intent in placing the digital objects.

[0021] 2. Determine the coordinates of the intended placement.

[0022] For some XR devices, these steps can be performed using an outward-facing camera and microphone; for example, a voice command can be used for step 1, and a finger can be used to indicate for step 2.

[0023] The inventors of the technologies described in this article have recognized that conventional technologies, such as those described above, have limitations and problems. One aspect involves the fact that future XR headsets may not be equipped with outward-facing cameras. This reason may stem from privacy concerns, where people near XR devices would not accept being surrounded by constantly moving cameras from such devices. There are documented instances where users wearing XR devices have been denied entry to some commercial establishments because the device represents a form of ubiquitous recording. Similarly, and by analogy, there are instances where photographs taken by drones are not permitted in many locations without the explicit consent or agreement of the person being photographed or the owner of the animal or property being photographed.

[0024] Furthermore, the European Commission has published a document entitled "Proposal for a Regulation Of The European Parliament and Of The Council Laying Down Harmonised Rules On Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts." In this proposal, AI systems are categorized into different groups related to their social risks, with each group facing varying degrees of regulation. One category is "unacceptable risk," which will be subject to strict regulation or even prohibition. This category includes real-time and remote biometric identification systems, which may include facial recognition systems in public places. Examples of "high-risk" systems that will be subject to strict regulation include surveillance systems (e.g., biometric surveillance for law enforcement, facial recognition systems). One indication of this is a proposal to ban facial recognition in public places. Overall, such regulations may impose restrictions on the use of AR glasses / devices with certain functions in public places.

[0025] The above and other examples illustrate the growing concern for privacy.

[0026] To achieve market appeal for XR devices, alternative solutions not based on outward-facing cameras are needed. From a technical perspective, one of the challenges is how to project augmented reality materials within an XR device to provide an end-user experience that is adapted to the real-world environment, without relying on cameras and image recognition / analysis for adapting the projection to the surroundings. This can be challenging, for example, when added text or objects need to fit well at a suitable viewing distance and on a surface.

[0027] Eye tracking using eye-facing cameras has been in use for some time. This technique can determine the gaze direction with an accuracy of 0.7 degrees. If the device has stereo camera monitoring of both eyes, it can perform distance measurements based on the intersection of the gaze directions of the two eyes. However, given close distances between the eyes, the distance estimation is only about one meter at most, which is quite good, while at greater distances, this gaze direction accuracy is not good enough for a proper distance estimate. For less expensive systems with only a single camera, where only one eye is monitored, eye tracking fails as a method for accurate distance estimation.

[0028] In view of the foregoing, there is a need for a technology that addresses the related problems described above, including allowing XR devices to accurately identify coordinates in a shared XR environment without relying on an outward-facing camera system, in order to, for example, place digital objects at a location desired by the user and provide other devices sharing the XR environment with useful information about the user's desired location. Summary of the Invention

[0029] It should be emphasized that the terms “comprising” and “including” are used in this specification to specify the presence of the stated features, integrals, steps, or components; however, the use of these terms does not exclude the presence or addition of one or more other features, integrals, steps, components, or combinations thereof.

[0030] Furthermore, in some instances (e.g., in the claims and the summary of the invention), reference letters may be provided to facilitate the identification of various steps and / or elements. However, the use of reference letters is not intended to infer or imply that the referenced steps and / or elements will be performed or operated in any particular order.

[0031] According to one aspect of the invention, the above and other objectives are achieved in a technique (e.g., method, apparatus, non-transitory computer-readable storage medium, program apparatus) for identifying a user-specified location in an extended reality environment, wherein the extended reality environment is displayed in a viewing area of ​​an extended reality device. Identifying the user-specified location involves: obtaining a self-position, wherein the self-position is a position in a shared set of one or more locations of a corresponding known real-world object in the extended reality environment; detecting a user gaze direction toward the viewing area; using radar to detect corresponding distances to one or more sensed real-world objects (107) in the real-world environment; selecting one of one or more known real-world objects using the self-position of the extended reality device, the detected user gaze direction, and the corresponding distances to one or more corresponding sensed real-world objects; determining a selected location of the selected known real-world object using a shared set of one or more locations of the corresponding known real-world objects in the extended reality environment, and using the selected location as the user-specified location in the extended reality environment.

[0032] In some, but not necessarily all, of the embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: determining the pose of a selected known real-world object using a shared set of one or more locations of corresponding known real-world objects in the extended reality environment, and using the pose of the selected known real-world object in the extended reality environment.

[0033] In another aspect of some, but not necessarily all, embodiments of the invention, self-position includes the posture of the extended reality device.

[0034] In another aspect of some, but not necessarily all, embodiments of the invention, the selected known real-world object has a corresponding sensed real-world object from one or more sensed real-world objects, and the actions involved in identifying a user-specified location in the extended reality environment further include: determining the pose of the extended reality device using the distance of the corresponding sensed real-world object from one or more sensed real-world objects, referencing a shared set of one or more locations of the corresponding known real-world object in the extended reality environment. In some, but not necessarily all, embodiments of this invention, the actions involved in identifying a user-specified location in the extended reality environment further include: determining an updated self-position of the extended reality device using the distance of the corresponding sensed real-world object from one or more sensed real-world objects, referencing a shared set of one or more locations of the corresponding known real-world object in the extended reality environment.

[0035] In another aspect of some, but not all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: presenting an indication of the selected location in the viewing area of ​​the extended reality device. In some, but not all, alternatives, this embodiment includes: receiving user input indicating rejection of the selected location; and adjusting the selected location based on further user input in response to rejection. In some, but not all, other alternatives, this embodiment includes: receiving user input indicating rejection of the selected location; and repeating the location identification steps in response to rejection.

[0036] In another aspect of some, but not all, embodiments of the invention, the actions involved in identifying a user-specified location in the extended reality environment further include: positioning a virtual object at the user-specified location in the extended reality environment. In some, but not all, alternatives to this embodiment, the virtual object has a front portion, and the virtual object is placed at the user-specified location in the extended reality environment such that the pose of the virtual object includes the front portion of the virtual object facing the extended reality device in the extended reality environment. In some, but not all, other alternatives to this embodiment, the virtual object is placed at the user-specified location in the extended reality environment such that the pose of the virtual object includes receiving user input indicating rejection of the selected location, and, in response to rejection of the selected location, adjusting one or more of the pose and size of the virtual object based on further user input.

[0037] In another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: publishing the pose of a virtual object to a second extended reality device that uses a shared set of corresponding known real-world objects at one or more locations in the extended reality environment.

[0038] In yet another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: publishing information about the virtual object to at least one other extended reality device.

[0039] In another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: receiving a location, posture, gaze direction, and estimated distance to a real-world object in the real-world environment from each of one or more other extended reality devices. Further, using the extended reality device's ego position, the detected user gaze direction, the corresponding distance to one or more corresponding sensed real-world objects, the received location, the received posture, the received gaze direction, and the received estimated distance, a common point occupied by a selected known real-world object from one or more known real-world objects is identified. In some, but not necessarily all, alternative embodiments, the actions involved in identifying a user-specified location in an extended reality environment further include: publishing the common point occupied by the selected known real-world object from one or more known real-world objects to one or more other extended reality devices. And in some, but not necessarily all, other alternative embodiments, the actions involved in identifying a user-specified location in an extended reality environment further include: publishing the common point occupied by the selected known real-world object from one or more known real-world objects to one or more additional extended reality devices that do not include one or more other extended reality devices.

[0040] In yet another aspect of some, but not necessarily all, embodiments of the present invention, the actions involved in identifying a user-specified location in an extended reality environment further include: initiating the execution of the location identification step in response to detecting a user command, wherein the user command is one or more of the following:

[0041] Voice commands;

[0042] Button or switch activated;

[0043] Reference gestures;

[0044] Reference sound;

[0045] Reference movement of one or more eyelids; and

[0046] A stable user gaze that persists in the reference direction for at least the reference time period.

[0047] In another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: receiving a tag; and associating the tag with the user-specified location. In some, but not necessarily all, alternative embodiments, receiving a tag includes: perceiving one or more spoken words, and using one or more spoken words as a tag.

[0048] In yet another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: repeating the location identification step for each of a plurality of locations in the extended reality environment; and using the plurality of locations in the extended reality environment to define the corners of the projection area in the extended reality environment.

[0049] In another aspect of some, but not necessarily all, embodiments of the invention, the actions involved in identifying a user-specified location in an extended reality environment further include: accessing a 3D occupancy grid map of the extended reality environment; and adjusting the location in the 3D occupancy grid map using the user-specified location in the extended reality environment to prevent the location from being within an occupied voxel.

[0050] In another aspect of some, but not necessarily all, embodiments of the invention, detecting the user's gaze direction toward the viewing area includes: monitoring the user's gaze direction over time, and detecting when the monitored user gaze has stabilized for at least a predetermined amount of time.

[0051] In another aspect of some, but not necessarily all, embodiments of the invention, a shared set of corresponding known real-world objects in one or more locations in an extended reality environment is a shared map of corresponding known real-world objects in one or more locations in an extended reality environment. Attached Figure Description

[0052] The object and advantages of the present invention will be understood by reading the following detailed description in conjunction with the accompanying drawings, wherein:

[0053] Figure 1 The image shows a user wearing an XR device configured as a headset, through which the user can view portions of the XR environment.

[0054] Figure 2 An XR device with a viewing area is shown, which has stereoscopic capability for viewing at least a portion of the XR environment.

[0055] Figure 3 It is a diagram of the user's gaze on the left and right parts of the viewing area of ​​the extended reality device, and the corresponding gaze heatmap indicating the parts of the viewing area that are viewed more than other parts.

[0056] Figure 4 This is a block diagram of a non-limiting exemplary XR device configured to perform actions according to the present invention.

[0057] Figure 5One aspect is a flowchart of an action performed by a device and / or system for specifying a point in a 3D XR environment, according to some exemplary embodiments of the invention.

[0058] Figure 6 One aspect is a flowchart of actions performed by a device and / or system for specifying points in a 3D XR environment, according to some exemplary alternative embodiments of the invention.

[0059] Figure 7 One aspect is a flowchart of actions performed by an XR device according to some, but not necessarily all, embodiments of the invention.

[0060] Figure 8 An exemplary controller is shown that can be included in an XR device to enable the execution of any and / or all actions associated with the device as described and illustrated herein. Detailed Implementation

[0061] Various features of the invention will now be described with reference to the accompanying drawings, wherein like parts are identified by like reference numerals.

[0062] Various aspects of the invention will now be described in more detail with reference to several exemplary embodiments. To facilitate understanding of the invention, many aspects are described as a sequence of actions to be performed by elements of a computer system or other hardware capable of executing programmed instructions. It will be appreciated that in each embodiment, various actions may be performed by dedicated circuitry (e.g., analog and / or discrete logic gates interconnected to perform dedicated functions), by one or more processors programmed with a suitable set of instructions, or by a combination of both. The term “circuit configured to” perform one or more of the described actions is used herein to refer to any such embodiment (i.e., one or more dedicated circuits alone, one or more programmed processors, or any combination thereof). Furthermore, the invention can be additionally considered to be fully embodied in any form of non-transitory computer-readable medium, such as solid-state memory, disk, or optical disk, containing a suitable set of computer instructions that will cause a processor to perform the techniques described herein. Therefore, aspects of the invention can be embodied in many different forms, and all such forms are considered to be within the scope of the invention. For each aspect of the invention, any embodiment of any such form as described above may be referred to herein as “logic configured to” perform the described actions, or alternatively as “logic” performing the described actions.

[0063] Embodiments of the invention, including various aspects thereof, involve designating 3D points in areas of an XR environment viewable by means of an XR device (e.g., but not limited to AR glasses), placing one or more virtual objects at designated locations, and sharing these points with others using corresponding other XR devices, so that multiple users virtually located in the same area can see the virtual objects from a consistent location. In some, but not necessarily all, embodiments, information regarding the orientation of a corresponding virtual object placed by one user is shared with other users, thus these users experience the virtual objects aligned within the area, as these users would expect based on the corresponding position and orientation of the virtual objects in the shared area.

[0064] In another aspect of at least some embodiments, the above is accomplished with the lowest possible overhead required by the user (e.g., to avoid requiring the user to perform a complex process to learn 3D points in the surrounding space).

[0065] To provide an overview of the technical aspects discovered in the various embodiments of the invention, the inventors of the technology described herein have observed that wireless communication devices are becoming increasingly advanced. The evolution of radio access protocol functionality over time has enabled wireless communication to utilize large bandwidths, and wireless devices typically support several different radio frequency bands, sometimes even operating some of them simultaneously. As an emerging feature, wireless communication devices can also leverage their advanced capabilities in radio signal transmission, reception, and signal processing to support radar sensing capabilities. For example, International Application PCT / EP2021 / 08358 describes a technology in which a mobile communication device, equipped with, for example, a radar-enhanced 5G modem, can determine its own location coordinates with the support of information about the device's surrounding environment obtained from radar, and this ability to do so has very high positioning accuracy without relying on a camera.

[0066] The embodiments of the invention described herein are based on the combined use of a communication connection between devices with radar functionality (e.g., but not limited to, 5G cellular connections) and an eye-tracking sensor in an XR device (e.g., but not limited to, AR glasses). In some, but not necessarily all, embodiments, information from an IMU is also used.

[0067] Among known technologies that enable mobile devices to determine their own location information, this is done by associating radar images of the area around the device with previously acquired radar images of the same area (similar to how SLAM algorithms are used with vision-based sensors) and determining how relevant objects relate to a detailed, stored map of objects in that environment. (If a detailed map does not yet exist, it can also be created through this process.) This also enables the device to determine which direction it is facing.

[0068] Movement of the device during the positioning process can lead to inaccuracies. To counteract this effect, intermittent movements of the user's position and head posture can be estimated using dead reckoning, since such glasses typically contain an IMU or accelerometer, and at certain times, the position can be re-estimated based on the same method just described, thereby improving accuracy or reducing power consumption and requiring less perception overall.

[0069] In various embodiments of the invention, after obtaining his / her own position in 3D, the user can gaze at an object and trigger a "designate point" command. The system then estimates which point in space the user is looking at and designates that location as the designated point for any subsequent actions in response.

[0070] In some, but not necessarily all, embodiments of the invention, the device is able to access a 3D map of the surrounding environment and make a specified point and the relevant direction of the first user's gaze known to other users in that area who also have access to the 3D map. In this way, the location selected by the user can be shared with others, and the virtual environments of these other users can be adapted accordingly so that all users agree on the content of the virtual environment and the location of that content.

[0071] In another aspect of some, but not necessarily all, embodiments of the invention, where a virtual object is placed at a point selected by the user, it is assumed that the orientation of the virtual object is also shared with any other user or stored in a scene descriptor for that area.

[0072] In another aspect of some, but not necessarily all, embodiments of the invention, when the map is incomplete or the user is viewing a dynamic object that is not present on the map or has moved from its position on the map, radar measurements can be used to determine the distance to the object. The combination of distance and detected gaze direction provides the user with a relative position. If the user has a known position and orientation on the map, the object's position can be shared with other users.

[0073] To further elaborate on the above discussion, in embodiments of the invention, this technology enables users of XR devices to designate points in 3D space using radar and eye tracking. Eye tracking is used to estimate the gaze direction at the time of triggering a command. The estimated gaze direction is used to designate the point (and any environmental metadata associated with the trigger). Radar scanning is used to detect the presence of physical objects in the XR environment. The device or associated system then estimates the accurate point relative to a 3D map of the area (which may be stored in the device or obtained from an external source) based on the user's position and gaze direction, as confirmed and aligned using radar signals used for environmental perception. In some, but not necessarily all, embodiments, the user receives confirmation of the identified point via some form of indication in the AR glasses. The user can respond by accepting, rejecting, or modifying the location. This allows the user to improve the calibration and accuracy of the technology.

[0074] In addition to 3D points, the pose / orientation of virtual objects placed at specified locations can be set by default, so that, for example, the user will see the front of the virtual object projected into the XR environment. This can be specified, for example, by adding the user's 3D position (at the specified location) to the information about the specified point or by adding a vector to the 3D point to point to the user's 3D position (at the specified location).

[0075] In another aspect of some, but not necessarily all, embodiments of the invention, if triggered by a user, the calibration process can further utilize radar perception to interpret gestures to refine 3D points and orientations. In a non-limiting example, this can be accomplished by interpreting the extension of the user's hand as indicating an increase in distance. Similarly, changes in arm pitch and yaw or hand rotation can be interpreted as commands to change the orientation of virtual objects.

[0076] A specified point in 3D space (along with gaze direction and orientation) confirmed by the user can be shared with multiple users in the same area. These users can then use this information to project the same information, which is accurately oriented and blended into the environment in a manner best suited to the application. The definition of "suitable" is application-dependent. Therefore, a full description of this is beyond the scope of this invention.

[0077] For multiple users, the combination and coordinated designation of 3D points can be enhanced by triangulation of the gaze direction of multiple users relative to a shared map and by radar-based distance measurement of potential objects in the gaze direction by each device.

[0078] Any other user's device can determine the use of a specified point, gaze direction, and orientation of the shared object by displaying virtual objects on a display (e.g., AR glasses). Users may be able to adjust the specified point by attaching a trigger command and incorporating the new gaze at the time of that command, thus making the scene truly interactive and updated in real time, as virtual objects can be placed and moved by any user sharing the XR environment.

[0079] In another aspect of some, but not necessarily all, embodiments of the invention, a specified location can be determined more accurately by having multiple users look at the same point when they give a trigger command. Their respective gazes can be used to triangulate the location relative to a public map of the area.

[0080] In another aspect of some, but not necessarily all, embodiments of the invention, to support stable gaze tracking functionality based on commands, a timer-based triggering criterion is applied to the “specify point” command. In some examples, this can be used to restrict activation of a new point only to those times during which the user’s gaze is stable and unchanging, for a period of time longer than a defined time value. In this case, the defined time value is application-dependent.

[0081] These and other aspects of the embodiments of the invention are described below.

[0082] Figure 1 A user 101 wearing an XR device 103 configured as a headset is shown, through which the user can view portions of an XR environment 105 (depending on the orientation and pose of the XR device 103). The XR environment 105 includes real-world (i.e., physical) objects 107 having any number of surfaces, such as surface 109. To illustrate aspects of the embodiment of the invention, it is assumed that the user 101 desires to place a digital (i.e., virtual) object 111 on top of the real-world object 107. To create a realistic rendering of the digital object 111, the rendering of the object should be consistent with the user's expectations regarding the position and size of the digital object 111. In some cases, the pose of the digital object 111 (e.g., the orientation and tilt angle of one or more surfaces of the object) can also be important. To configure correct rendering as described above, the position and distance of the digital object 111 relative to the XR device 103 need to be known. Since it is expected that the digital object 111 will be rendered on top of the real-world object 107, the position of the real-world object 107 and the distance 113 from the real-world object 107 can be used as proxies for these properties of the digital object 111.

[0083] In one aspect of the invention, both location and distance information can be obtained by scanning the XR environment 105 with radar signal 115 and receiving radar reflection 117. The direction of arrival (DoA) of radar reflection 117 will indicate the orientation of the real-world object 107, and the time delay between the transmission of radar signal 115 and the reception of radar reflection 117 corresponds to the round-trip distance (i.e., twice the distance) between the XR device 103 and any surface of the real-world object 107 from which the radar signal 115 is reflected.

[0084] It is recognized that an XR environment may include more than one real-world object 107, and some of these objects may be entirely sensed by radar. This creates ambiguity regarding which real-world object the user wants the digital object 111 to be placed on. To address this technical challenge, the user's gaze direction is also sensed. Specifically, when the user instructs a headset device to create the digital object 111 (e.g., via voice command or by pressing one or more buttons on the input device), the user 101 looks at the location where the digital object 111 will be rendered. The user's gaze direction at that moment is sensed and recorded. Then, when multiple real-world objects are sensed via radar scanning, the knowledge of the user's gaze direction is combined with the detected directions of the radar-sensed real-world objects. The real-world object with the best matching direction is then selected. Since the orientation and distance information of the digital object's location within the XR environment 105 are now known, the image of the digital object can be adjusted so that its apparent location and apparent size at that location create a realistic user experience of the digital object 111 actually existing in the XR environment 104. To compensate for subsequent movement of the XR device 103, the XR device 103 senses the movement (e.g., by means of gyroscope or inertial measurement unit (IMU) technology) and continues to adjust the rendering of the digital object 111 so that its perceived presence at that location appears fixed.

[0085] Now for reference Figure 2 Other aspects related to the embodiments of the invention are discussed. In this non-limiting example, the XR device 203 has a viewing area 221 with stereoscopic capability, and thus the viewing area 221 has two portions, each dedicated to viewing by a corresponding eye of the user's pair of eyes 201. By gazing at the viewing area 221, the user 101 is able to see at least a portion of the XR environment 205.

[0086] In this example, the XR environment 205 includes a real-world surface 209 of an object (e.g., the surface could be a countertop, desktop, workbench, the top of a shelf, etc.). Further, in this example, the user's gaze 223 is directed to a real-world location 225 above surface 209. In this stereoscopic example, gaze monitoring for each eye will detect the display position x of the portion of the viewing area 221 allocated to the left eye that the user's left eye is looking at or viewing. left y left And the user's right eye is looking at or viewing the display position x of the portion of viewing area 221 allocated for right eye use. right y right Typically, coordinates are x... left y left Not equal to coordinates x right y right In this example, gaze direction 229 can be determined by finding the intersection of the left and right gaze directions 223. Figure 2 In the example, this is shown as a real-world location 225 with x, y, z coordinates (in this case, the third coordinate z represents depth).

[0087] Since digital (virtual) objects do not actually exist in the real world, they have no real-world location; they can only be seen within the viewing area 221 of the XR device 203. However, by creating left and right images at corresponding display locations in the viewing area 221 that correspond to the user's respective left and right gaze directions 223, the user 101 will perceive the digital object 211 as being located at a view location 227 in the XR environment 205. For convenience, the convention adopted in the examples described herein will assign the same reference frame to both the real-world location and the view location, such that the x, y, z coordinates are the same regardless of which is referred to ("real-world location" or "view location"). However, this is by no means a fundamental aspect of all embodiments. Instead, different coordinate systems may be used for the real-world location and the view location. It should also be noted that, in order to further enhance the effect that the digital object 211 will be perceived as being located at viewing position 227 in the XR environment 205, the light from the digital object 211 should appear to come from a distance corresponding to viewing position 227, so that the eye focuses the light onto the retina when it adjusts its lens to that distance, and for this reason, an adjustable projection system with lenses can be used in eyeglasses.

[0088] It should be further noted that the aspects found in the embodiments of the invention are not limited to use only with stereoscopic XR devices. Rather, aspects of the invention also exist in single-field-of-view embodiments and in embodiments employing a stereoscopic viewing area 221 but monitoring the gaze of only one eye 201 of the user. In these later embodiments, the gaze direction 229 is based solely on the gaze direction of the monitored eye.

[0089] In a further aspect, as previously mentioned, the XR environment 205 combines radar scanning with user gaze monitoring to identify specific real-world locations 225. Therefore, for example, as Figure 2 As shown, when a user is looking at real-world location 225, radar reflections from that point in the environment will indicate to XR device 203 (via its direction of arrival) a radar reflection direction 231 that is substantially the same as the detected gaze direction 229. The substantial match between these two measurements confirms that the distance that can be determined from this identical radar reflection is the distance to real-world location 225.

[0090] In another aspect of some, but not necessarily all, embodiments of the invention, the XR environment 205 is shared with one or more other XR devices, such as Figure 2Another XR device 241 is shown. XR environments are typically three-dimensional spaces; therefore, both XR device 203 and the other XR device 241 can provide views of elements within the XR device (e.g., the surface 209 of a real-world object or virtual object 211) and interactions with these elements. However, based on the XR device's position within the XR environment 205, each of the two XR devices 203, 241 presents the XR environment experience from its own distinct perspective. Considering this aspect, it is advantageous to specify positions within the XR environment 205 by referring to a common frame of reference (such as a shared map, or more generally, a shared set of one or more locations). Since positions characterized in this way will refer to the same locations within the XR environment, they can be used by any XR device sharing the XR environment. Therefore, for example, if a user of XR device 203 specifies a location (real-world location 225 or view location 227) and places a virtual object 211 there, information about the location and the virtual object can be passed to another XR device 241 (e.g., by publishing the information to a server accessible to the other XR device 241). The other XR device 241 can then create the same virtual object 211 at the same locations 225, 227 within the XR environment 205, allowing the user of the other XR device 241 to experience and, in some embodiments, interact with the virtual object 211. While the location of this virtual object 211 will be the same for all users, the pose of the virtual object is application-dependent. In some instances, it may be desirable for the pose to be constant relative to the XR environment; therefore, for example, a user of XR device 203 may see the front of the virtual object 211, while a user of another XR device 241 may view the virtual object 211 from the side or back. However, in other embodiments, it may be desirable for the pose of the shared virtual object 211 to be constant relative to each user, so that if a user of XR device 203 is looking at the front of the virtual object 211, a user of another XR device 241 will also see the front of the virtual object. This arrangement can be useful, for example, when the virtual object 211 is a display screen or monitor device.

[0091] Various techniques for monitoring a user's gaze are known, including techniques for detecting specific points of interest within the user's field of view, and any such technique can be used in embodiments according to the invention. (By example and reference) Figure 3As the user's eyes look at the corresponding left and right portions of the viewing area 321 of the extended reality device 303, the position of the user's eyes 301 is monitored over time. Through the viewing area 321, the user is able to see a surface 309 on which three objects 311-1, 311-2, and 311-3 are placed. Various gaze directions of the user (e.g., user gazes at 323-1, 323-2, 323-3, and 323-4) are detected and recorded. When a sufficient number of samples are recorded, a so-called "gaze heatmap" can be created, which shows the frequency with which the user's gaze is directed towards any given point within the field of view. Low-intensity areas (such as point 335) are indicated in a certain way and can be distinguished from higher-intensity areas 337 in the gaze heatmap, which indicate the content that the viewer spends the most time looking at. For illustrative purposes only, Figure 3 In the image, three higher-intensity regions 337 correspond to the locations of three objects 311-1, 311-2, and 311-3. The radar object detection mentioned above, combined with gaze heatmap information (or equivalents), can be used to determine candidate locations for augmented reality text and object overlay.

[0092] The embodiments of the invention are characterized by having XR capabilities, such as, but not limited to, AR glasses. Aspects of the embodiments of the invention do not rely on information obtained from an outward-facing camera. Therefore, having an outward-facing camera is not necessary for such a device, although its presence is not excluded if it is required for other purposes.

[0093] Another feature of the embodiments of the invention is gaze tracking capability. For example, as is known in the art, techniques for gaze tracking can be camera-based.

[0094] Another feature of the embodiments of the invention is radar functionality. This can take the form of, for example, dedicated radar circuitry. In one of several possible alternatives, the device of the invention can be equipped with a modem (e.g., a modem configured to support cellular communications) that is configured to provide radar functionality, as known in the art. The modem can also enable communication between the XR device and other devices and / or network services.

[0095] In another feature of the embodiments of the invention, the device has the ability to determine its own position in a reference frame, for example, shared with other devices. In this way, position information obtained by one device is meaningful when transmitted to other devices in a shared environment. As used herein, the term "self-localization" refers to the ability to determine one's own position.

[0096] Furthermore, the embodiments of the invention are characterized by having a mechanism through which digital objects (i.e., virtual objects) can be displayed to the user of the device. AR glasses are one such device with this capability.

[0097] Another feature present in some, but not necessarily all, embodiments is that the user can provide the device with a mechanism to trigger and / or command, for example, to cause a specific process to be executed. For example, and not limitingly, the user may be able to control the device using any one or more of the following: voice commands and eye commands (e.g., by means of a specific blinking pattern).

[0098] To further illustrate these aspects, Figure 4 This is a block diagram of a non-limiting exemplary XR device 401 configured to perform actions according to the present invention. The exemplary XR device 401 includes:

[0099] - Optical unit 403 includes a viewing area 405 through which a user can see a portion of the XR environment. The optical unit is also capable of overlaying computer-generated digital objects onto the viewing area 405.

[0100] - Gaze tracker 407, which monitors the gaze direction of one or both of the user's eyes.

[0101] - Radar circuitry 409 or equivalent radar functionality. As an example of the latter, XR device 401 may include a modem 411 for wireless communication with, for example, a wireless communication network. Such a modem 411 typically operates at frequencies suitable for radar operation. Therefore, modem 411 can be configured to operate as a radar device suitable for use in embodiments of the invention. Multiple radar transceivers (or such) with a distance of several decimeters between them, employing a suitable radar signal bandwidth (e.g., at approximately 1 GHz to provide a radar range with resolution allowing positioning of two objects 15 cm apart). Figure 4 The radar transceiver shown with antenna array 419 can be used to achieve centimeter-level distance estimation accuracy for objects around XR device 401, and at millimeter-wave frequencies, a few centimeters will be sufficient. The actual distance depends on the frequency / wavelength.

[0102] - An inertial measurement unit (IMU) 413 is used to track any movement of the XR device 401. This movement information can be used to stabilize the image presented in the visible area 405 and can also be used as a basis for determining adjustments to the rendered image of digital objects so that the digital objects will appear to have a stable position when the user wearing the XR device 401 moves around.

[0103] - Microphone array 415 that can serve multiple purposes (including but not limited to receiving voice commands and information from users).

[0104] - A controller 417 for controlling the components described above and other components of the XR device 401. The controller 417 can be configured from hard-wired circuitry, a programmable software-controlled processor / component, or a combination of both.

[0105] - Self-location function 421, which can be embodied as a separate circuit or as an additional function performed by controller 417.

[0106] The following non-limiting examples illustrate the capabilities of embodiments according to various aspects of the present invention:

[0107] Assumptions used to determine the link budget for beamforming transmission and angle-of-arrival reception radar:

[0108] • Operating frequency: 60GHz ISM band, wavelength (λ) = 5mm

[0109] • TX array: 4x4 antenna elements, supporting 2D beamforming in 1cm²

[0110] • RX array: 2x2 antenna elements, supporting 5mm x 5mm 2D AoA

[0111] Antenna element gain: G = 3dBi

[0112] Noise figure: NF = 8dB

[0113] • Transmission power: Daily Line P TX =1mW

[0114] Integral time: T int =1ms

[0115] ·SNR min = 20dB

[0116] Radar cross section: RCS=0.1m 2

[0117] Calculation results:

[0118] RX Sensitivity:

[0119] P RX =-174dBm-10*log(T int )+NF+SNR min =-174+30+8+30dBm=-116dBm,

[0120] Maximum effective range: R max=(16*P TX *16*G*G*λ 2 *RCS / (P RX *(4*π) 3 )) 0.25 =27m.

[0121] It can be seen that a distance of 27m can be obtained even without beamforming on the receiving side. For targets within this distance (closer to 27m), the TX beam can be swept while monitoring the angle of arrival of the echo. The orientation of small objects can then be determined with high accuracy. However, for large objects, only the middle portion of the illuminated area is detected. It may then be difficult to determine whether a part of the object is in the user's gaze direction. However, by performing a transmit beam sweep, the range of the object's angle of arrival can be observed while illuminating different parts (especially the edges of the object). Such observation then allows for a more accurate determination of the angular spread of large objects.

[0122] Now refer to Figure 5 Other aspects of at least some embodiments of the invention are described. Figure 5 In one aspect, it is a flowchart of actions performed by a device and / or system to specify points in a 3D XR environment. In other aspects, Figure 5 The boxes depicted in the text can also be considered to represent devices 500 (e.g., hardwired or programmable circuitry or other processing devices) used to perform the described actions.

[0123] exist Figure 5 In an exemplary embodiment, in response to a trigger command issued by a user of the first device, the first device designates a point in 3D space. This point can then be shared with other devices capable of accessing a 3D map of an environment consistent with the environment of the first device.

[0124] This point in 3D space can be used as a placement point for virtual objects in a shared XR environment, and the object should virtually remain stationary at that location even if the user moves or rotates. A non-limiting example is placing a virtual hologram of a person on a physical chair in a room, or placing a virtual screen at a specific location on a wall.

[0125] like Figure 5As shown, initially, the first device obtains its location (e.g., a shared 3D map relative to its XR environment) (step 501). This can be achieved using any form of self-localization algorithm known in the art. The device is also able to access a shared 3D map of the environment (which does not need to be very detailed) (step 503). It should be noted that some embodiments work without access to such a 3D map, although accuracy may be affected by the localization algorithm, radar beam, and gaze tracking accuracy. It should also be noted that in this case, location typically also includes the device's orientation / pose / orientation. In some alternative embodiments, step 501 is performed between steps 507 and 509 described below.

[0126] The first user is looking at a point in 3D space (e.g., at the chair) (step 505). This is the point the first user intends to specify, and optionally, in some embodiments, a virtual object will be positioned if needed in the application.

[0127] The process of designating the first user activation point is referred to herein as "designating the point" (step 507). This activation can be achieved using any of a variety of techniques, such as, but not limited to, issuing a voice command, an eye command (such as a predefined blinking pattern), activating a button associated with the XR device, etc. It is advantageous for the first user to continue looking at the point during the issuance of the command (and in some embodiments, for a short period afterward sufficient for the system to act on the command, typically a few milliseconds). Here, a timer value can be set to ensure that only intentional points are determined.

[0128] When the command is recognized / understood by the XR device / system, it estimates the location of the user-specified point using the following (step 509):

[0129] - Determine the user's gaze direction relative to their position; and

[0130] - Use radar to determine the distance to possible physical objects located in the direction of gaze.

[0131] More specifically, based on the user's position and orientation and possible distances to one or more candidate objects, a 3D map of known objects in the room is correlated, and possible objects (i.e., objects with the correct / matching orientation and the correct / matching distance) are identified. During this step, the position and orientation of the first device may optionally be updated depending on the possibility that the position and orientation from step 501 are not up-to-date.

[0132] In an optional action that is not required to be included in all embodiments, the first device highlights the estimated point generated in step 509 to the first user on the XR device's display (step 511). In response to this optional step, the first user can accept or reject the designated point (step 513, also optional), where rejection means that the device will update the position by some form of incremental fine-tuning (e.g., using a gesture), or will restart the entire process that began at least in step 505.

[0133] In another optional action that is not required to be included in all embodiments, depending on whether the first device has any need to do so, the first device shares the identified location with another device (e.g., so that other users can have a 3D point in the room that is consistent with the first user's point, so that when they look at the "same" virtual object (e.g., another user's hologram), they are looking at the same physical space) (step 515).

[0134] In another optional feature, the data for the specified point shared with one or more other devices may include more data than just the 3D position, such as, but not limited to, the user’s position when the point is specified, and / or the direction toward the user at the time of specification, so that projection onto the appropriate orientation can be performed (e.g., the front of the object is facing the position of the user when the specified point is specified).

[0135] Now refer to Figure 6 Other aspects of at least some alternative embodiments of the invention are described. Figure 6 In one aspect, it is a flowchart of actions performed by multiple XR devices (e.g., AR glasses) and / or systems to specify points in a shared 3D XR environment. In other aspects, in Figure 6 The boxes depicted in the text can also be considered to represent a device 600 (e.g., a hardwired or programmable circuit or other processing device) for performing the described actions.

[0136] exist Figure 6 In an exemplary embodiment, each of the multiple devices sharing a 3D XR environment designates a point in the shared 3D space, which is collaboratively determined by the multiple XR devices sharing the space in response to a trigger command issued by a user of the first device. This point in the 3D space can be used as a placement point for virtual objects in the shared XR environment, and the object should be virtually stationary at that location even if the user moves or rotates. A non-limiting example is placing a virtual hologram of a person on a physical chair in a room, or placing a virtual screen at a specific location on a wall, so that the respective users of the devices each see the virtual object placed at the same location in the shared 3D XR environment.

[0137] like Figure 6As shown, each of the multiple XR devices obtains its location (e.g., a shared 3D map relative to its XR environment) (step 601). This can be achieved using any form of self-localization algorithm known in the art. Each device is also able to access a shared 3D map of the environment (which does not need to be very detailed) (step 603). It should be noted that some embodiments work without access to such a 3D map, although accuracy may be affected by the localization algorithm, radar beam, and gaze tracking accuracy. It should also be noted that in this case, location typically also includes the device's orientation / pose / orientation.

[0138] Multiple users agree on a point in the room and are all looking at that point (e.g., a chair) (step 605), and give a trigger command (e.g., a voice command "specify point") (step 607).

[0139] Since multiple devices are now present, each with its position / pose, gaze direction, and estimated distance to one or more possible objects, these devices send this data to a device or server that calculates a common point (step 609). If the data collection does not result in a sufficiently consistent point, the user may be requested to re-execute the process initiated in (step 607). In cases where there are more than two devices and a sufficiently consistent common point exists for some devices, but there are devices whose data contributions do not conform to that point, only those devices / users may be requested to re-execute the process. Alternatively, in a later action 617, the consistent point derived from other devices is shared only with peripheral devices.

[0140] The estimated location is shared with each device user, and after seeing the estimated point highlighted in the device's viewing area (step 613), each device user can individually accept or reject the point (step 615). Steps 613 and 615 are optional and may depend on whether the estimated point is completely consistent across multiple devices (e.g., how many devices the point is sufficiently consistent with, and the degree of consistency among these devices).

[0141] Users can then share the point with other devices that are not yet able to access it (step 617). This step is also optional.

[0142] Additional aspects of the embodiments of the invention will now be described below.

[0143] In one respect, the "designated point" command can be given in different forms. It can be a voice command of a single word, or it can be pressing a button on a handheld control device, making a gesture, uttering a sound (e.g., clapping), blinking in a certain pattern, etc. Similar command methods can also be used for fine-tuning of a point or its projection direction, possibly along with gaze (e.g., using the handheld control device or the joystick on it; making a gesture with the bare hand; issuing a voice command to adjust the distance; etc.).

[0144] In another embodiment, the device is configured to provide user-named points so that multiple such points can be maintained and mentioned in an explicit manner. The designation can be accomplished in any number of ways, such as, but not limited to, voice commands (e.g., “designate point – chair” or “designate point – Jim’s avatar”). The system can then distinguish between several such points when placing virtual objects or sharing points with other users (e.g., Jim’s avatar should be placed at a defined point, which might be a chair).

[0145] In some, but not necessarily all, embodiments of another class, more than one point can be specified using any of the techniques described above, and these points can be combined to define an area in the XR environment. For example, a projection area, such as a board, can be defined by specifying a corner. The projection onto the board will then fit more closely to the edges of the frame, improving the quality for the user when watching a presentation or movie. A special command (e.g., “Specify corner”) can be given. The XR device can display a polygon representing the area to the user, updating as new corners are specified until completion. For a conventional rectangular board mounted horizontally, two diagonal corners can be specified. A special command for this, such as “Specify diagonal corner”, may exist. Alternatively, a corner and size of the board can be interpreted via gestures or other user input.

[0146] Since radar can be used to create 3D occupancy grid maps (i.e., maps representing the probability that voxels (3D versions of pixels, i.e., small-volume grid points) are occupied), in many embodiments, the device is configured to correct user-specified points in 3D space by, for example, ensuring that any placement of a digital object will not be within an occupied voxel. In a non-limiting example, such correction may involve placing the digital object above an occupied voxel via vertical projection. The reduction in the degrees of freedom of the estimated 3D point will increase accuracy and better conform to the intended behavior of the invention, thus improving the user experience. Similarly, for a projection area, an occupied grid can be used to correct the placement of a virtual screen to remove or minimize occluded areas (e.g., not placing the lowest part of the virtual screen under a table or behind a pillar or other obstacle in a room).

[0147] Now refer to Figure 7Further aspects of some, but not necessarily all, embodiments of the invention are described below. Figure 7 One aspect is a flowchart of actions performed by an XR device according to some, but not necessarily all, embodiments of the invention. In other aspects, Figure 7 The boxes depicted in the text can also be considered to represent a device 700 (e.g., a hardwired or programmable circuit or other processing device) for performing the described actions.

[0148] In an embodiment of this exemplary class, the XR device is configured to identify a user-specified location within an extended reality environment, wherein the extended reality environment is displayed in the viewing area of ​​the extended reality device. The operation of the device includes obtaining a self-location (step 701), wherein the self-location is a location within a shared set of one or more locations of a corresponding known real-world object within the extended reality environment.

[0149] Detect the user's gaze direction toward the viewing area (step 703).

[0150] Before, after, or simultaneously with step 703, radar is used to detect the corresponding distance to one or more sensed real-world objects in the real-world environment (step 705).

[0151] Use the extended reality device’s self-position, the detected user gaze direction, and the corresponding distance of one or more corresponding sensed real-world objects to select one of one or more known real-world objects (step 707).

[0152] The selected location of the chosen known real-world object is determined using a shared set of one or more locations of the corresponding known real-world object in the extended reality environment, and the selected location is used as the user-specified location in the extended reality environment (step 709).

[0153] Now refer to Figure 8 Further aspects of the present invention are described below. Figure 8 An exemplary controller 801 is shown that can be included in an XR device or system to cause any and / or all actions described and illustrated herein associated with that device or system to be performed. In particular, controller 801 includes circuitry configured to perform any one or any combination of the various functions described herein. Such circuitry can be, for example, fully hardwired circuitry (e.g., one or more application-specific integrated circuits, "ASICs"). However, in Figure 8The exemplary embodiments depicted herein are programmable circuits including a processor 803 coupled to one or more memory devices 805 (e.g., random access memory, disk drive, optical disk drive, read-only memory, etc.) and an interface 807 enabling bidirectional communication with other elements of the devices as described above. A complete list of possible other elements is beyond the scope of this specification.

[0154] One or more memory devices 805 store a program 809 (e.g., a processor instruction set) configured to cause processor 803 to control other device elements to perform any of the aspects described herein. One or more memory devices 805 may also store data (not shown) representing various constant and variable parameters that processor 803 may need and / or may generate when performing its functions, such as those specified by program 809.

[0155] Various embodiments of the present invention offer numerous benefits and advantages over conventional techniques. Some of these advantages include providing a simple and efficient technique for specifying points in a shared 3D XR environment based on the detection of the user's gaze direction.

[0156] In some embodiments, a further advantage is the ability to specify the orientation / pose of virtual objects placed in 3D space.

[0157] Another benefit and advantage is the ability to share this information among multiple users.

[0158] Another benefit and advantage is the ability to share this information among multiple users without relying on an externally facing camera.

[0159] The invention has been described with reference to embodiments. However, it will be readily apparent to those skilled in the art that the invention may be embodied in specific forms other than those described above. Therefore, the described embodiments are merely illustrative and should not be considered limiting in any way. The scope of the invention is further defined by the appended claims, and not merely by the foregoing description, and all variations and equivalents falling within the scope of the claims are intended to be included therein.

Claims

1. A method for identifying a user-specified location (225) in an extended reality environment (105, 205), wherein, The extended reality environment (105, 205) is displayed in the viewing area (221) of the extended reality device (103, 203, 401), the method comprising: a) Obtain the self-position of the extended reality device (103, 203, 401) (501, 601, 701), wherein the self-position is a position in a shared set of one or more positions of the corresponding known real-world object in the extended reality environment (105, 205); b) Detect (505, 703) the user's gaze direction (231) toward the viewing area (221); c) Use radar (703) to detect the corresponding distance (113) to one or more sensed real-world objects (107) in the real-world environment. d) Using the self-position of the extended reality device (103, 203, 401) (707), the detected user gaze direction (231), and the corresponding distance (113) of the one or more corresponding sensed real-world objects (107), to select one of the one or more known real-world objects (107); and e) Using the shared set of (709) corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205) to determine the selected location (225) of the selected known real-world object (107), and using (709) the selected location as the user-specified location (225) in the extended reality environment (105, 205).

2. The method according to claim 1, comprising: The pose of the selected known real-world object (107) is determined using the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205), and the pose of the selected known real-world object (107) is used in the extended reality environment (105, 205).

3. The method according to any one of the preceding claims, wherein, The self-position includes the posture of the extended reality devices (103, 203, 401).

4. The method according to any one of claims 1 and 2, wherein, The selected known real-world object (107) has a corresponding sensed real-world object among the one or more sensed real-world objects (107), and wherein the method includes: The pose of the extended reality device (103, 203, 401) is determined by referring to the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment and using the distance (113) of the corresponding one sensed real-world object among the one or more sensed real-world objects (107).

5. The method according to claim 4, comprising: The updated self-position of the extended reality device (103, 203, 401) is determined by referring to the shared set of locations of one or more known real-world objects (107) in the extended reality environment (105, 205) and using the distance (113) of the corresponding sensed real-world object among the one or more sensed real-world objects (107).

6. The method according to any one of the preceding claims, comprising: An indication of the selected location is presented in the viewing area (221) of the extended reality device (103, 203, 401).

7. The method of claim 6, comprising: Receive user input (513, 615) indicating rejection at the selected location; as well as In response to rejection of the selected location, the selected location is adjusted (513, 615) based on further user input.

8. The method of claim 6, comprising: Receive user input (513, 615) indicating rejection at the selected location; as well as In response to a rejection at the selected location, repeat steps b) through e).

9. The method according to any one of the preceding claims, comprising: Position the virtual object (111, 211) at the user-specified location (225) in the extended reality environment (105, 205).

10. The method according to claim 9, wherein, The virtual object (111, 211) has a front portion, and the method includes: The virtual object (111, 211) is placed at the user-specified location (225) in the extended reality environment (105, 205) such that the pose of the virtual object (111, 211) includes the front side of the virtual object (111, 211) facing the extended reality device (103, 203, 401) in the extended reality environment (105, 205).

11. The method of claim 9, comprising: Receive user input (513, 615) indicating rejection at the selected location; as well as In response to rejection at the selected location, one or more of the pose and size of the virtual object (111, 211) are adjusted based on further user input.

12. The method according to claim 10 or 11, comprising: The pose of the virtual object (111, 211) is published (515, 617) to a second extended reality device (241), which uses the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205).

13. The method according to any one of claims 9 to 12, comprising: Information about the virtual object (111, 211) is published (515, 617) to at least one other extended reality device (241).

14. The method according to any one of the preceding claims, comprising: The location, posture, gaze direction (231), and estimated distance (113) to real-world objects (107) in the real-world environment are received from each of one or more other extended reality devices (241). as well as Step d) includes: using the self-position of the extended reality device (103, 203, 401) (707), the detected user gaze direction (231), the corresponding distance (113) of the one or more corresponding sensed real-world objects (107), the received position, the received pose, the received gaze direction (231) and the received estimated distance (113) to identify a common point occupied by a selected known real-world object among the one or more known real-world objects (107).

15. The method of claim 14, comprising: The common point occupied by the selected known real-world object (107) is published (515, 617) to the one or more other extended reality devices (241).

16. The method of claim 15, comprising: The public point occupied by the selected known real-world object (107) is published (515, 617) to one or more additional extended reality devices that do not include the one or more other extended reality devices (241).

17. The method according to any one of the preceding claims, comprising: In response to the detection of a user command, execution of steps a) to e) of (507, 607) is initiated, wherein the user command is one or more of the following: Voice commands; Button or switch activated; Reference gestures; Reference sound; Reference movement of one or more eyelids; and A stable user gaze that persists in the reference direction for at least the reference time period.

18. The method according to any one of the preceding claims, comprising: Receive tags; as well as Associate the tag with the user-specified location (225).

19. The method according to claim 18, wherein, Receiving the label includes: perceiving one or more spoken words, and using the one or more spoken words as the label.

20. The method according to any one of the preceding claims, comprising: For each of the multiple locations in the extended reality environment (105, 205), repeat steps a) through e). as well as The corners of the projection area in the extended reality environment (105, 205) are defined using the plurality of locations in the extended reality environment (105, 205).

21. The method according to any one of the preceding claims, comprising: Access the 3D occupied grid map of the extended reality environment (105, 205); as well as The position in the 3D occupied grid map is adjusted using the user-specified position in the extended reality environment (105, 205) to prevent the position from being within an occupied voxel.

22. The method according to any one of the preceding claims, wherein, Detecting (505, 703) the user's gaze direction (231) toward the viewing area (221) includes: Monitor the user's gaze direction over time (231), and detect when the monitored user gaze has stabilized for at least a reference time.

23. The method according to any one of the preceding claims, wherein, The shared set of the corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205) is a shared map of the corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205).

24. A computer program (809) comprising instructions that, when executed by at least one processor (803), cause the at least one processor (803) to perform the method according to any one of claims 1 to 23.

25. A carrier comprising the computer program (809) according to claim 24, wherein, The carrier is one of electronic signals, optical signals, radio signals, and non-transitory computer-readable storage media (805).

26. An apparatus for identifying a user-specified location (225) in an extended reality environment (105, 205), wherein, The extended reality environment (105, 205) is displayed in the viewing area (221) of the extended reality device (103, 203, 401), wherein the device is configured to cause the extended reality device (103, 203, 401) to perform: a) Obtain the self-position of the extended reality device (103, 203, 401) (501, 601, 701), wherein the self-position is a position in a shared set of one or more positions of the corresponding known real-world object in the extended reality environment (105, 205); b) Detect (505, 703) the user's gaze direction (231) toward the viewing area (221); c) Using radar (703) to detect the corresponding distance (113) to one or more sensed real objects (107) in the real-world environment. d) Using the self-position of the extended reality device (103, 203, 401) (707), the detected user gaze direction (231), and the corresponding distance (113) of the one or more corresponding sensed real-world objects (107), to select one of the one or more known real-world objects (107); and e) Using the shared set of (709) corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205) to determine the selected location (225) of the selected known real-world object (107), and using (709) the selected location as the user-specified location (225) in the extended reality environment (105, 205).

27. The apparatus according to claim 26, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The pose of the selected known real-world object (107) is determined using the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205), and the pose of the selected known real-world object (107) is used in the extended reality environment (105, 205).

28. The apparatus according to any one of claims 26 to 27, wherein, The self-position includes the posture of the extended reality devices (103, 203, 401).

29. The apparatus according to any one of claims 26 and 27, wherein, The selected known real-world object (107) has a corresponding sensed real-world object among the one or more sensed real-world objects (107), and wherein the device is configured to cause the extended reality device (103, 203, 401) to perform: The pose of the extended reality device (103, 203, 401) is determined by referring to the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205) and using the distance (113) of the corresponding sensed real-world object among the one or more sensed real-world objects (107).

30. The apparatus according to claim 29, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The updated self-position of the extended reality device (103, 203, 401) is determined by referring to the shared set of locations of one or more known real-world objects (107) in the extended reality environment (105, 205) and using the distance (113) of the corresponding sensed real-world object among the one or more sensed real-world objects (107).

31. The apparatus according to any one of claims 26 to 30, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: An indication of the selected location is presented in the viewing area (221) of the extended reality device (103, 203, 401).

32. The apparatus according to claim 31, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Receive (513, 615) user input indicating rejection at the selected location; and In response to rejection of the selected location, the selected location is adjusted (513, 615) based on further user input.

33. The apparatus according to claim 31, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Receive (513, 615) user input indicating rejection at the selected location; and In response to a rejection at the selected location, repeat steps b) through e).

34. The apparatus according to any one of claims 26 to 33, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Position the virtual object (111, 211) at the user-specified location (225) in the extended reality environment (105, 205).

35. The apparatus according to claim 34, wherein, The virtual objects (111, 211) have a front portion, and the means are configured to cause the extended reality device (103, 203, 401) to perform: The virtual object (111, 211) is placed at the user-specified location (225) in the extended reality environment (105, 205) such that the pose of the virtual object (111, 211) includes the front of the virtual object (111, 211) facing the extended reality device (103, 203, 401) in the extended reality environment (105, 205).

36. The apparatus according to claim 34, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Receive (513, 615) user input indicating rejection at the selected location; and In response to rejection at the selected location, one or more of the pose and size of the virtual object (111, 211) are adjusted based on further user input.

37. The apparatus according to claim 35 or 36, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The pose of the virtual object (111, 211) is published (515, 617) to a second extended reality device (241), which uses the shared set of corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205).

38. The apparatus according to any one of claims 34 to 37, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Information about the virtual object (111, 211) is published (515, 617) to at least one other extended reality device (241).

39. The apparatus according to any one of claims 26 to 38, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The location, posture, gaze direction (231), and estimated distance (113) to real-world objects (107) in the real-world environment are received from each of one or more other extended reality devices (241). as well as Step d) includes: using the self-position of the extended reality device (103, 203, 401) (707), the detected user gaze direction (231), the corresponding distance (113) of the one or more corresponding sensed real-world objects (107), the received position, the received pose, the received gaze direction (231) and the received estimated distance (113) to identify a common point occupied by a selected known real-world object among the one or more known real-world objects (107).

40. The apparatus according to claim 39, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The common point occupied by the selected known real-world object (107) is published (515, 617) to the one or more other extended reality devices (241).

41. The apparatus according to claim 40, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: The public point occupied by the selected known real-world object (107) is published (515, 617) to one or more additional extended reality devices that do not include the one or more other extended reality devices (241).

42. The apparatus according to any one of claims 26 to 41, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: In response to the detection of a user command, execution of steps a) to e) of (507, 607) is initiated, wherein the user command is one or more of the following: Voice commands; Button or switch activated; Reference gestures; Reference sound; Reference movement of one or more eyelids; and A stable user gaze that persists in the reference direction for at least the reference time period.

43. The apparatus according to any one of claims 26 to 42, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Receive tags; as well as Associate the tag with the user-specified location (225).

44. The apparatus according to claim 43, wherein, Receiving the label includes: perceiving one or more spoken words, and using the one or more spoken words as the label.

45. The apparatus according to any one of claims 26 to 44, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: For each of the multiple locations in the extended reality environment (105, 205), repeat steps a) to e); and The corners of the projection area in the extended reality environment (105, 205) are defined using the plurality of locations in the extended reality environment (105, 205).

46. ​​The apparatus according to any one of claims 26 to 45, wherein, The device is configured to cause the extended reality device (103, 203, 401) to perform: Access the 3D occupancy grid map of the extended reality environment (105, 205); and The position in the 3D occupied grid map is adjusted using the user-specified position in the extended reality environment (105, 205) to prevent the position from being within an occupied voxel.

47. The apparatus according to any one of claims 26 to 46, wherein, Detecting (505, 703) the user's gaze direction (231) toward the viewing area (221) includes: Monitor the user's gaze direction over time (231), and detect when the monitored user gaze has stabilized for at least a reference time.

48. The apparatus according to any one of claims 26 to 47, wherein, The shared set of the corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205) is a shared map of the corresponding known real-world objects (107) at one or more locations in the extended reality environment (105, 205).

Images