Private control interface for extended reality

Abstract

Systems, methods, and non-transitory media for generating a private control interface for an extended reality (XR) experience are provided. An example method can include determining a pose of an XR device within a mapped scene of a physical environment associated with the XR device; detecting a private region in the physical environment and a location of the private region relative to the XR device pose, the private region comprising an area estimated to be within a field of view (FOV) of a user of the XR device and outside a FOV of a person in the physical environment, a recording device in the physical environment, and / or an object in the physical environment; mapping a virtual private control interface to the private region based on the pose of the XR device and the location of the private region; and rendering the virtual private control interface within the private region.

Description

Technical Field

[0001] This disclosure generally relates to generating and registering private control interfaces in extended reality and other applications. Background Technology

[0002] Extended reality (AR) technologies can be used to present virtual content to users and / or combine real-world environments with virtual environments to provide extended reality experiences. The term AR can include virtual reality, augmented reality, mixed reality, and so on. Each of these forms of AR allows users to experience or interact with immersive virtual environments or content. For example, an AR experience can allow users to interact with a real or physical environment enhanced or augmented by virtual content.

[0003] Extended reality technologies can be implemented to enhance user experiences across a wide range of contexts, such as entertainment, healthcare, retail, education, and social media. However, in many settings, it may not be desirable to reveal certain extended reality content or interactions with certain extended reality interfaces. In one example, a user might not want authentication interfaces, such as password hints, rendered in a way that could allow other users to see or infer specific user interactions with authentication interfaces or associated authentication information. Summary of the Invention

[0004] Systems, methods, and computer-readable media for providing a private control interface for an extended reality experience are disclosed. According to at least one example, a method for generating a private control interface for an extended reality experience is provided. The method may include: determining the pose of an extended reality device within a mapped scene of a physical environment associated with the extended reality device; detecting a private region in the physical environment and the position of the private region relative to the pose of the extended reality device, the private region comprising areas estimated to be within the field of view (FOV) of a user of the extended reality device and outside the corresponding FOV of at least one of a person in the physical environment, a recording device in the physical environment, and an object in the physical environment that allows viewing access from outside the physical environment; mapping a virtual private control interface to the private region based on the pose of the extended reality device and the detected private region, the virtual private control interface including one or more input interface elements; and rendering the virtual private control interface within the private region of the mapped scene.

[0005] According to at least one example, a non-transitory computer-readable medium is provided for providing a private control interface for an extended reality experience. The non-transitory computer-readable medium may include computer-readable instructions that, when executed by one or more processors, cause the one or more processors to: determine the pose of an extended reality device within a mapped scene of a physical environment associated with the extended reality device; detect a private region in the physical environment and the position of the private region relative to the pose of the extended reality device, the private region comprising areas estimated to be within the field of view (FOV) of a user of the extended reality device and outside the corresponding FOV of at least one of a person in the physical environment, a recording device in the physical environment, and an object in the physical environment that enables viewing access from outside the physical environment; map a virtual private control interface to the private region based on the pose of the extended reality device and the detected private region, the virtual private control interface including one or more input interface elements; and render the virtual private control interface within the private region in the mapped scene.

[0006] According to at least one example, an apparatus is provided for providing a private control interface for extended reality experiences. The apparatus may include at least one memory and one or more processors implemented in circuitry and configured to: determine the pose of the apparatus within a mapped scene of a physical environment associated with the apparatus; detect a private region in the physical environment and the position of the private region relative to the apparatus pose, the private region including areas estimated to be within the field of view (FOV) of a user of the apparatus and outside the corresponding FOV of at least one of a person in the physical environment, a recording device in the physical environment, and / or an object in the physical environment that can be viewed and accessed from outside the physical environment; map a virtual private control interface to the private region based on the apparatus pose and the detected private region, the virtual private control interface including one or more input interface elements; and render the virtual private control interface within the private region in the mapped scene.

[0007] According to at least one example, another apparatus is provided for providing a private control interface for extended reality experiences. The apparatus may include: a unit for determining the pose of the device within a mapped scene of a physical environment associated with the device; a unit for detecting a private region in the physical environment and the position of the private region relative to the pose of the device, the private region comprising areas estimated to be within the field of view (FOV) of a user of the device and outside the corresponding FOV of at least one of a person in the physical environment, a recording device in the physical environment, and / or an object in the physical environment that enables viewing access from outside the physical environment; a unit for mapping a virtual private control interface to the private region based on the pose of the device and the position of the private region, the virtual private control interface including one or more input interface elements; and a unit for rendering the virtual private control interface within the private region in the mapped scene.

[0008] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above may include determining that the region is outside the corresponding FOV of at least one of the person, the recording device, and / or the object by: detecting one or more objects in the physical environment; and determining that the corresponding FOV of at least one of the person, the recording device, and the object is obscured by the one or more objects.

[0009] In some examples, detecting a private region may include: calculating face location data associated with a first face detected in an image capturing the mapped scene, wherein the first face corresponds to the person; calculating the person's field of view (FOV) based on the face location data; and determining, based on the person's FOV and the location of the private region, that the private region is occluded outside the person's line of sight by one or more objects located between the private region and the person.

[0010] In some examples, detecting a private region may further include: calculating face location data associated with a second face detected in an image capturing the mapped scene, wherein the second face corresponds to the user; calculating the user's field of view (FOV) based on the face location data associated with the second face; and determining that the private region is within the user's FOV based on the location of the private region.

[0011] In some examples, mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region may include: determining at least one of a first size and / or a first orientation of the private region; determining at least one of a second size and / or a second orientation of the virtual private control interface based on the first size and / or the first orientation of the private region, wherein the second size of the virtual private control interface matches or is adapted to the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region; generating the virtual private control interface according to the second size and / or the second orientation; and aligning the virtual private control interface with the private region.

[0012] In some examples, objects in the physical environment that allow viewing access from outside the physical environment may include windows, glass doors, and / or open doors.

[0013] In some examples, the private area may include the surface of a body part associated with the user, and detecting the private area may include determining that the surface of the body part faces the user and is away from at least one of a person, a recording device in the physical environment, and / or an object in the physical environment. In some cases, the body part may include a hand, and the surface of the body part may include the palm of the hand, and detecting the private area may include: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the user; and determining that the biometric information matches the previously registered biometric information.

[0014] In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: identifying the palm of a hand used for rendering a second virtual private control interface; and prior to rendering the second virtual private control interface on the palm of the hand: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determining whether the biometric information matches the previously registered biometric information.

[0015] In some aspects, the above-described methods, non-transitory computer-readable media, and apparatus may include: determining that the biometric information does not match the previously registered biometric information; and in response to determining that the biometric information does not match the previously registered biometric information, determining that the second virtual private control interface is not rendered on the palm of the hand.

[0016] In some aspects, detecting a private area may include: detecting one or more objects in the physical environment; detecting one or more obstructions in the physical environment based on the one or more objects in the physical environment; and detecting the private area by determining whether the one or more obstructions block the visibility of the person, the recording device, and / or the object to the private area.

[0017] In some examples, detecting one or more occlusions in a physical environment can include tracing the path of light within the physical environment.

[0018] In some cases, the recording device may include a camera, and the virtual private control interface may include an augmented reality interface. In some examples, each of the above devices may be a mobile device. In other examples, each of the above devices may include an augmented reality device.

[0019] In some aspects, the device may include or part of the following: a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors.

[0020] This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of this patent specification, any or all of the accompanying drawings, and each claim.

[0021] The foregoing, along with other features and embodiments, will become clearer when referred to the following description, claims, and drawings. Attached Figure Description

[0022] To describe in detail the various advantages and features that can be obtained from this disclosure, the above principles will be described in more detail by referring to specific embodiments illustrated in the accompanying drawings. It is understood that these drawings depict only exemplary embodiments of this disclosure and should not be considered as limiting its scope. The principles herein are described and explained with additional specificity and detail using the drawings, in which:

[0023] Figure 1 This is a simplified block diagram illustrating an example extended reality system according to some examples of this disclosure;

[0024] Figure 2 This is a diagram illustrating an example of an extended reality system worn by a user according to some examples of this disclosure;

[0025] Figure 3 This is a diagram illustrating example landmark points of a hand, according to some examples of this disclosure, that can be used to track the position of the hand and the interaction between the hand and a virtual interface;

[0026] Figure 4 This is a diagram illustrating an example system flow for generating a private control interface for extended reality experiences, according to some examples of this disclosure;

[0027] Figure 5 This is a flowchart illustrating an example process for generating a private control interface for an extended reality experience, according to some examples of this disclosure;

[0028] Figures 6A to 6D This is a diagram illustrating an example of a private control interface rendered according to some examples of this disclosure;

[0029] Figure 7 This is a diagram illustrating an example rendering flow for rendering a private control interface based on user-requested rendering preferences and rendering based on preferences received from the user, according to some examples of this disclosure.

[0030] Figure 8 This is a diagram illustrating examples of security indicators in a rendered presentation according to some examples of this disclosure;

[0031] Figure 9 This is a diagram illustrating an example scheme for verifying biometric information before rendering a private control interface on a user's body, according to some examples of this disclosure;

[0032] Figure 10 This is a flowchart illustrating example methods for generating private control interfaces for extending real-world experiences, according to some examples of this disclosure; and

[0033] Figure 11 An example computing device architecture is shown according to some examples of this disclosure. Detailed Implementation

[0034] Certain aspects and embodiments of this disclosure are provided below. It will be apparent to those skilled in the art that some of these aspects and embodiments can be applied independently and some can be combined. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The accompanying drawings and description are not intended to be limiting.

[0035] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0036] Extended reality devices (such as head-mounted displays (HMDs) and smart glasses) can track a user's hands and / or fingertips to allow the user to use their hands and / or fingertips to control interfaces rendered within an extended reality (XR) environment, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or other XR environments. Furthermore, in an XR environment, the user's hand or fingertip interactions with the control interface may be visible to other users in the environment. However, it is generally undesirable to reveal such interactions with the control interface and / or allow such interactions to potentially be visible to other users. For example, if a user participating in a multi-user AR experience inputs private information using a control interface rendered during the multi-user AR experience, other users in the multi-user AR experience will be able to see that control interface and the user's interactions with it. Therefore, other users in the multi-user AR experience may be able to determine any information provided by the user through their interactions with the control interface. Consequently, the user may unintentionally expose such information to other users, which in some cases may include private, confidential, or personal information.

[0037] In some cases, even if other users cannot see the control interface or interface elements that the user is interacting with during an XR experience, the user's hand or fingertip interactions may still be interpretable to others, allowing them to infer the input (and corresponding information) associated with the interaction. For example, if the user is participating in an AR experience around other users, those users may be able to see the user's hand and / or fingertip interactions with the control interface. Furthermore, the user's hand and / or fingertip interactions may indicate that the user is typing in the air on a virtual keyboard or interacting with some other virtual interface. Based on the user's hand and / or fingertip movements or patterns of these movements, other users may be able to infer the type of virtual interface the user is interacting with and interpret the input provided by the user. This input may thus reveal information about the user, which in many cases may involve private or confidential information.

[0038] Furthermore, if a user's interaction with the virtual interface is within the field of view (FOV) of a recording device (such as a camera), the user's interaction may be unintentionally recorded. Another user with access to the recording could then view the recording and interpret the user's interactions and related information from it. The recording may even allow other users to interpret the user's interactions and related information more easily and / or more accurately. Moreover, in some cases, computers may be used to process the recording and identify user interactions and related information from the recorded video, potentially jeopardizing the privacy of user information.

[0039] In some cases, artificial intelligence (AI) interpreters or systems may be used to process recorded video and identify information provided by users through virtual interfaces. AI interpreters or systems can potentially identify information provided by users through virtual interfaces very accurately. Therefore, when users participate in XR experiences, they may want to protect their privacy when interacting with virtual interfaces rendered in the XR environment, even if other users are not participating in the XR experience or can see the virtual interfaces, in order to avoid exposing their input and associated information to other users.

[0040] In an illustrative example, a user's interaction with a rendered virtual interface (such as a virtual keyboard) during an XR experience to enter a password could potentially be interpreted by other users, allowing them to identify or infer the password entered by the user. As previously mentioned, even if other users are not involved in the same XR experience or cannot see the virtual interface, they may still be able to infer the password entered by the user. To illustrate, in a typing XR setup using a virtual keyboard, a user's typing actions may indicate interaction with the virtual keyboard. Typing actions may allow other users to infer key locations and typed data, such as the entered password. Such exposure of typed data (such as password information) poses serious security and privacy concerns for users.

[0041] In another example, a user's interaction with a virtual voting button in a virtual classroom poll or quiz might be interpreted by other users, potentially allowing them to identify or infer the user's submitted vote and compromise the integrity of the vote and / or the poll or quiz. For instance, other users participating in the same virtual classroom poll or quiz might be able to see the virtual interface the user uses to interact with the virtual button. Therefore, if other users see the user's interaction with the virtual button, they might be able to identify the user's answer to the poll or quiz. Even if other users are not participating in the virtual classroom poll or quiz or cannot see the virtual voting button for other reasons, the user might still be able to infer the location and significance of the input button chosen by the user by observing multiple questions. By inferring the location and importance of the input button, other users might deduce the vote submitted by the user through the virtual classroom poll or quiz.

[0042] Similarly, in multi-user AR gaming experiences such as multi-user AR tabletop games, a user's interaction with virtual game controls can be interpreted by other users. For example, a user in a multi-user AR gaming experience might see virtual game content blended with the physical world around them (e.g., rendered on physical objects or areas), and therefore might see the virtual game controls provided to the user, as well as the user's interaction with these controls. Thus, other users might be able to identify or infer the user's input through the virtual game controls. Furthermore, in some cases, even if the user cannot see the same virtual game controls, other users might be able to infer the location and meaning of the user's selected input button by observing the user's interactions throughout the game. This can allow other users in the game to deduce the user's input and related information within the game. However, in many situations, such as competitive game scenarios, users may not want other users in a multi-user AR game to know their input through virtual game controls, as this would put them at a disadvantage in competition with other users.

[0043] This disclosure describes systems, methods, and computer-readable media for generating private control interfaces for XR experiences. In some examples, the methods disclosed herein can hide a virtual control interface (also referred to as a “virtual private control interface”) presented to a user as part of an XR environment from other users, devices, and / or objects in the user’s physical environment, and can hide or obfuscate the user’s interaction with the virtual control interface to protect the privacy of user interactions and data from nearby users and / or recording devices. The virtual control interface may also be referred to herein as a virtual interface. In some examples, the virtual control interface can be rendered within a private area of ​​the user’s physical environment (e.g., a real-world environment). As used herein, a control interface (e.g., a private control interface, a virtual control interface, a virtual private control interface, etc.) can include a rendered interface with input fields and / or input interface elements, but can also include rendered fields without input fields and / or input interface elements, such as interfaces with static or display-only content.

[0044] A private area can be, for example, but not limited to, an area or location in the user's physical environment (e.g., the real world) that is estimated to be outside the field of view of other users or recording devices (e.g., cameras, microphones) and / or obscured from other users or recording devices. Private areas can be used to render virtual control interfaces for the user during an XR experience to prevent accidental users and devices from seeing or interpreting the user's interactions with the virtual control interface and / or inferring data entered by the user in such virtual control interfaces. In some examples, rendering a virtual control interface on a private area can include rendering or displaying the virtual control interface on the XR device in a manner that appears to a user viewing the virtual control interface through / from the XR device as if the virtual control interface were presented, projected, or located on the private area. For example, rendering a virtual control interface on a private area can include rendering a virtual overlay of the virtual control interface on the private area. In other examples, rendering a virtual control interface on a private area can include projecting the virtual control interface onto an area or location in the physical environment that corresponds to the private area. Furthermore, because the virtual control interface appears to be rendered on (or projected onto) a private area to users with XR devices, users can interact with the virtual control interface as if it were actually located on a private area within the physical environment. Since the private area is invisible to other users and / or recording devices in the physical environment, these other users and / or recording devices will not be able to see the user's interaction with the virtual control interface or infer the input data associated with such interaction.

[0045] For example, in the virtual keyboard example above, the virtual keyboard could be rendered in a private location within the user's physical environment (e.g., an object, surface, or plane in the real world) to prevent other users in the physical environment from inferring key locations or typed data. The user can see both the private location and the virtual keyboard through an XR device (e.g., a head-mounted display, AR glasses, etc.), but other users or devices will not be able to see the user's interaction with the virtual keyboard while it is rendered in the private location. Therefore, other users will not be able to see the user's interaction with the virtual keyboard to potentially infer the information entered by the user through the virtual keyboard, even if these users are also using XR devices and participating in the same XR experience as the user.

[0046] In the virtual classroom voting or quiz example above, the virtual button can be rendered in a private location within the physical environment to prevent other users in the physical environment from inferring the button's location and meaning, and thus inferring the user's input. Similarly, the user can see both the private location and the virtual button through an XR device, but other users or devices will not be able to see the user's interaction with the virtual button while it is rendered in the private location. In the multi-user AR game example above, the virtual button in the AR game can be rendered in a private location within the physical environment to prevent other users in the physical environment from inferring the button's location and meaning, and thus inferring the user's input. Again, the user can see both the private location and the virtual button through an XR device, but other users or devices will not be able to see the user's interaction with the virtual button while it is rendered in the private location.

[0047] The terms "private location" and "private region" are used interchangeably herein and may include surfaces, objects, planes, regions, configurations associated with the physical environment, occlusions, etc., within the user's field of view (FOV) but outside the FOV of other users or recording devices in the physical environment, and / or may be invisible to other users or recording devices through objects in the physical environment (e.g., windows or doors). Therefore, when a private control interface is rendered in a private location within the user's physical environment, the user can interact with the private control interface without exposing interaction and associated information to users, devices, and / or objects in the physical environment.

[0048] In some examples, a user's XR device can identify a frontal surface or plane in the physical environment relative to the user's position / location within the physical environment, and determine whether that surface or plane is outside the line of sight of other users or recording devices, or whether the plane behind the surface is occluded by other users or recording devices. If the surface or plane is outside the line of sight of other users or recording devices, or is occluded by other users or recording devices, the surface or plane can be selected as a private location. The XR device can then render any virtual control interfaces at that private location to prevent other users or recording devices in the physical environment from seeing the user's interaction with such virtual control interfaces and / or inferring the importance and information associated with such interactions. The user can see the virtual control interfaces rendered at the private location through the XR device, which allows the user to see the virtual content rendered by the XR as well as the user's physical environment. However, other users in the physical environment cannot see the private location and therefore cannot see or interpret the user's interaction with the virtual control interfaces, even if these users are participating in the same XR experience through their respective XR devices.

[0049] For example, a user's XR device can detect a book, clipboard, the user's hand, mobile phone, or any other object in front of the user (or within the user's field of view) in the physical environment, and can determine that the surface of the book, clipboard, hand, mobile phone, or other object facing the user (or within the user's field of view) is outside the line of sight of other users and / or recording devices in the physical environment or is obscured so that other users and / or recording devices in the physical environment cannot see it (e.g., obscured by one or more objects). The XR device can then render any virtual control interface such that, when viewed from the XR device, it appears to the user to be on the surface of the book, clipboard, hand, mobile phone, or other object. Because the virtual control interface appears to the user with the XR device to be rendered on the surface of the book, clipboard, hand, mobile phone, or other object, the user will interact with the virtual control interface as if it were actually on the surface of the book, clipboard, hand, mobile phone, or other object in the physical environment. Because the surface of the book, clipboard, hand, mobile phone, or other object is invisible to other users and / or recording devices in the physical environment, other users and / or recording devices will not be able to see the user's interaction with the virtual control interface or infer the input data associated with such interaction. Therefore, by rendering virtual interfaces on books, clipboards, hands, mobile phones, or other objects, XR devices can protect user interactions and data.

[0050] In some cases, XR devices can detect faces (e.g., the faces of users associated with the XR device and / or other users) and use face recognition data (e.g., data indicating the presence and / or location of one or more users) and / or occlusion data (e.g., data indicating that an area or location is occluded by one or more objects or otherwise obscured from view by other users or devices) to detect any private locations (e.g., private surfaces, areas, locations, planes, etc.) in the physical environment that are occluded from view by other users and devices (or outside their line of sight), except for the user associated with the XR device. Face recognition data can be reported by other devices themselves, provided by users associated with the XR device, detected by the XR device using facial recognition technology (e.g., neural networks, facial recognition algorithms, etc.), and / or detected by a server communicating with the XR device using information provided by the XR device and using facial recognition technology. In some cases, ray tracing or other mechanisms can be used to compute occlusion data. The XR device can then render any virtual control interface within a private location in the physical environment.

[0051] In some examples, XR devices can resize (e.g., scale), reorient, and / or reconfigure the virtual control interface so that it fits within a private location and is aligned with that location to maintain a virtual control interface that is hidden or occluded for other users and / or recording devices. For example, if the private location is the surface of a book facing the user associated with the XR device and outside the FOV of other users or recording devices in the environment, the XR device can resize, reorient, and / or reconfigure the virtual control interface so that it can render within the book's surface and align within the book's surface boundaries. In some cases, the XR device can also select an optimal alignment or shape and / or configuration for the virtual control interface that fits within the private location.

[0052] In some cases, XR devices may determine whether to render a virtual control interface in a private location based on, but not limited to, user preferences, system preferences (e.g., XR device preferences), application preferences (e.g., XR application preferences), application and / or content terms, laws or regulations, best practices, security specifications, potential security vulnerabilities or risks associated with the virtual control interface and / or associated data, the type of information associated with the virtual control interface, the type of the virtual control interface, and privacy levels. For example, a user can specify which applications, controls, and / or data types should be private. The XR device can then render the virtual control interface associated with such applications, controls, and / or data types in a private location.

[0053] This technology will be described in the following disclosures. The discussion begins with a description of an example system and technology for providing proprietary interface controls for XR experiences, such as... Figures 1 to 9As shown in the image. Then proceed as follows: Figure 10 The description shown illustrates an example method for providing private interface controls for XR experiences. The discussion concludes with a description of an example computing device architecture that includes example hardware components suitable for performing XR functionality and providing private interface controls for XR experiences, such as... Figure 11 As shown. This disclosure now turns to Figure 1 .

[0054] Figure 1 This is a diagram illustrating an example extended reality system 100 according to some aspects of this disclosure. The extended reality system 100 can run (or execute) XR applications and implement XR operations. In some examples, the extended reality system 100 can perform: tracking and localization; mapping of the physical world and / or environment (e.g., scene) surrounding the extended reality system 100; and localization and rendering of virtual content on a screen, display, and / or visible plane / area as part of an XR experience. For example, the extended reality system 100 can generate a map (e.g., a three-dimensional (3D) map) of the scene in the physical world surrounding the extended reality system 100, track the pose (e.g., localization and position) of the extended reality system 100 relative to the scene (e.g., relative to a 3D map of the scene), localize and / or anchor virtual content at a specific location on the scene map, and render the virtual content on the display / screen such that the virtual content appears to be at a location in the scene corresponding to the specific location on the scene map where the virtual content is localized and / or anchored.

[0055] In this illustrative example, the extended reality system 100 includes one or more image sensors 102, accelerometers 104, gyroscopes 106, storage devices 108, computing components 110, an XR engine 120, a proprietary control interface management engine 122, an image processing engine 124, and a rendering engine 126. It should be noted that... Figure 1 The components 102-126 shown are non-limiting examples provided for illustrative and explanatory purposes, and other examples may include more than... Figure 1 The components shown may include more, fewer, or different components. For example, in some cases, the extended reality system 100 may include one or more additional sensors (e.g., one or more inertial measurement units (IMUs), radar, light detection and ranging (LIDAR) sensors, audio sensors, etc.), one or more display devices, one or more other processing engines, one or more other hardware components, and / or Figure 1 One or more other software and / or hardware components not shown in the diagram. The following is about... Figure 11 Further descriptions of example architectures and example hardware components that can be implemented by the extended reality system 100 are provided.

[0056] Furthermore, for simplicity and explanation purposes, one or more image sensors 102 will be referred to herein as image sensor 102 (e.g., in the singular). However, those skilled in the art will recognize that the extended reality system 100 may include a single image sensor or multiple image sensors. Furthermore, any component of the extended reality system 100 referred to in the singular or plural form (e.g., 102-126) should not be construed as limiting the number of such components implemented in the extended reality system 100 to one or more. For example, mentioning accelerometer 104 in the singular should not be construed as limiting the number of accelerometers implemented in the extended reality system 100 to one. Those skilled in the art will recognize that for... Figure 1 Any of the components 102-126 shown, the extended reality system 100 may include only one such component or more such components.

[0057] The extended reality system 100 may be a single computing device or part of or implemented by multiple computing devices. In some examples, the extended reality system 100 may be part of electronic devices (or multiple devices) such as camera systems (e.g., digital cameras, IP cameras, camcorders, surveillance cameras, etc.), telephone systems (e.g., smartphones, cellular phones, conferencing systems, etc.), desktop computers, laptops or notebook computers, tablets, set-top boxes, smart TVs, display devices, gaming terminals, video streaming devices, IoT (Internet of Things) devices, smart wearable devices (e.g., head-mounted displays (HMDs), smart glasses, etc.), or any other suitable electronic devices.

[0058] In some implementations, one or more image sensors 102, accelerometers 104, gyroscopes 106, storage devices 108, computing components 110, XR engines 120, proprietary control interface management engines 122, image processing engines 124, and rendering engines 126 may be part of the same computing device. For example, in some cases, one or more image sensors 102, accelerometers 104, gyroscopes 106, storage devices 108, computing components 110, XR engines 120, proprietary control interface management engines 122, image processing engines 124, and rendering engines 126 may be integrated into smartphones, laptops, tablets, smart wearable devices, gaming systems, and / or any other computing device. However, in some implementations, one or more image sensors 102, accelerometers 104, gyroscopes 106, storage devices 108, computing components 110, XR engines 120, proprietary control interface management engines 122, image processing engines 124, and rendering engines 126 may be part of two or more independent computing devices. For example, in some cases, some of the components 102-126 may be part of or implemented by a computing device, while the remaining components may be part of or implemented by one or more other computing devices.

[0059] Image sensor 102 may include any image and / or video sensor or capture device, such as a digital camera sensor, camcorder sensor, smartphone camera sensor, image / video capture device on an electronic device such as a television or computer, camera, etc. In some cases, image sensor 102 may be part of a camera or computing device, such as an XR device (e.g., an HMD, smart glasses, etc.), digital camera, smartphone, smart TV, gaming system, etc. In some examples, image sensor 102 may be part of a multi-camera assembly, such as a dual-camera assembly. Image sensor 102 may capture image and / or video content (e.g., raw image and / or video data), which may then be processed by computing component 110, XR engine 120, proprietary control interface management engine 122, image processing engine 124, and / or rendering engine 126, as described herein.

[0060] In some examples, image sensor 102 can capture image data and generate frames based on the image data and / or provide the image data or frames to XR engine 120, proprietary control interface management engine 122, image processing engine 124, and / or rendering engine 126 for processing. Frames can include still images or video frames from video sequences. Frames can include pixel arrays representing a scene. For example, a frame can be a red-green-blue (RGB) frame with red, green, and blue components per pixel; a luminance, chrominance red, chrominance blue (YCbCr) frame with one luminance component and two chrominance (color) components (chrominance red and chrominance blue) per pixel; or any other suitable type of color or monochrome image.

[0061] Accelerometer 104 can detect the acceleration of the augmented reality system 100 and generate acceleration measurements based on the detected acceleration. Gyroscope 106 can detect and measure the orientation and angular velocity of the augmented reality system 100. For example, gyroscope 106 can be used to measure the pitch, roll, and yaw of the augmented reality system 100. In some examples, image sensor 102 and / or XR engine 120 can use the measurements obtained by accelerometer 104 and gyroscope 106 to calculate the attitude of the augmented reality system 100. As mentioned above, in other examples, the augmented reality system 100 may also include other sensors, such as magnetometers, machine vision sensors, smart scene sensors, voice recognition sensors, shock sensors, vibration sensors, position sensors, tilt sensors, etc.

[0062] Storage device 108 can be any storage device used for storing data. Furthermore, storage device 108 can store data from any component of the extended reality system 100. For example, storage device 108 can store data from image sensor 102 (e.g., image or video data), data from accelerometer 104 (e.g., measurements), data from gyroscope 106 (e.g., measurements), data from computing component 110 (e.g., processing parameters, preferences, virtual content, rendered content, scene maps, tracking and positioning data, object detection data, privacy data, XR application data, facial recognition data, occlusion data, etc.), data from XR engine 120, data from proprietary control interface management engine 122, data from image processing engine 124, and / or data from rendering engine 126 (e.g., output frames). In some examples, storage device 108 may include a buffer for storing frames processed by computing component 110.

[0063] One or more computing components 110 may include a central processing unit (CPU) 112, a graphics processing unit (GPU) 114, a digital signal processor (DSP) 116, and / or an image signal processor (ISP) 118. Computing components 110 can perform various operations, such as image enhancement, computer vision, graphics rendering, extended reality (e.g., tracking, localization, pose estimation, mapping, content anchoring, content rendering, etc.), image / video processing, sensor processing, recognition (e.g., text recognition, face recognition, object recognition, feature recognition, tracking or pattern recognition, scene recognition, occlusion detection, etc.), machine learning, filtering, and any of the various operations described herein. In this example, computing component 110 implements an XR engine 120, a proprietary control interface management engine 122, an image processing engine 124, and a rendering engine 126. In other examples, computing component 110 may also implement one or more other processing engines.

[0064] The operation of the XR engine 120, the proprietary control interface management engine 122, the image processing engine 124, and the rendering engine 126 (and any image processing engine) can be implemented by any computing component 110. In an illustrative example, the operation of the rendering engine 126 can be implemented by the GPU 114, and the operation of the XR engine 120, the proprietary control interface management engine 122, and the image processing engine 124 can be implemented by the CPU 112, the DSP 116, and / or the ISP 118. In some cases, the computing component 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof to perform any of the various operations described herein.

[0065] In some examples, the XR engine 120 can perform XR operations based on data from one or more sensors, such as one or more IMUs, radar, etc., on the image sensor 102, accelerometer 104, gyroscope 106, and / or extended reality system 100. In some examples, the XR engine 120 can perform tracking, localization, attitude estimation, mapping, content anchoring operations, and / or any other XR operations / functions.

[0066] The private control interface management engine 122 can perform various operations to determine (and manage) how, where, and / or when the control interface, including the private control interface, is rendered during an XR experience. An XR experience may include presenting XR content (e.g., virtual reality content, augmented reality content, mixed reality content, etc.) to a user using the extended reality system 100. In some examples, XR content and experiences may be provided by the extended reality system 100 through XR applications that offer specific XR experiences, such as XR game experiences, XR classroom experiences, XR shopping experiences, XR entertainment experiences, XR activities (e.g., interactive, troubleshooting activities, etc.). During an XR experience, the user can use the extended reality system 100 to view and / or interact with virtual content. In some cases, the user can view and / or interact with virtual content while also viewing and / or interacting with the physical environment around them, allowing for an immersive experience between the physical environment and the virtual content mixed with or integrated with it.

[0067] Private control interfaces may include any virtual interface that displays virtual data and / or receives input data that is determined to be private and / or should be protected against unauthorized viewing and / or device access, such as virtual input or output interfaces for private and / or protected data. This type of data may be defined or determined to be private and / or protected based on (e.g., but not limited to) user preferences, system preferences, application preferences, application and / or content terms, laws or regulations, best practices, security specifications, potential security vulnerabilities, etc. Non-limiting examples of this type of private and / or protected data may include personal user data, passwords, personal identification numbers (PINs), medical information, financial information, verification information, player information in multi-user games, security information, password-protected data, data protected by access control lists (ACLs) and / or licenses, user account data, privileged or confidential data, etc.

[0068] In some examples, the private control interface management engine 122 can use data from image sensor 102, accelerometer 104, gyroscope 106, and / or any other sensors to detect objects in the scene (e.g., windows, doors, walls, tables, books, devices, chairs, etc.), recognize faces in the scene, detect surveillance devices in the scene (e.g., cameras, microphones, etc.), and / or any other conditions and / or characteristics of the scene. In some examples, the private control interface management engine 122 can also use data from other devices or applications, such as self-reported data from other user devices and / or data provided by users of the extended reality system 100. The private control interface management engine 122 can use such data to identify occlusions in the scene, which can be used to render / display the private control interface to users associated with the extended reality system 100 in a manner that protects the privacy of the private control interface from being viewed or accessed by other users, surveillance devices, etc., and / or potentially compromising the privacy of the private control interface and associated data in other ways. Obstructions may include, for example, but not limited to, areas, surfaces, planes, objects, and / or regions that are visible in the scene or within the field of view (FOV) of the user associated with the extended reality system 100, but not visible or not within the FOV of other users and / or monitoring devices in the scene and / or not visible through certain objects (e.g., windows or doors) in the scene.

[0069] When the private control interface management engine 122 identifies one or more occluders, it can coordinate with the XR engine 120 and / or the rendering engine 126 to render the private control interface (e.g., on surfaces, planes, regions, areas, positions, relative proximity, etc.) on the one or more occluders to prevent viewing access to such private control interfaces from other users, devices, etc. In some examples, the private control interface management engine 122 can calculate and / or determine the configuration and / or rendering of the private control interfaces on one or more occluders. For example, the private control interface management engine 122 can determine the size, orientation, position, interface design or scheme, and / or any other configuration of the private control interface. For illustration, the private control interface management engine 122 can determine the configuration of the private control interface that allows it to fit within the area of ​​one or more occluders and / or maintain a specific position, orientation, etc., relative to one or more occluders and / or other objects or users in the scene.

[0070] In some cases, the private control interface management engine 122 can detect conditions and / or events that trigger occlusion detection, private control interface rendering, private control interface modification, or configuration. For example, the private control interface management engine 122 can detect: when certain types of data and / or interfaces should be rendered as private control interfaces or within private control interfaces (e.g., based on preferences, rules, content types, scene characteristics, objects in the scene, users in the scene, etc.), when private control interfaces should be removed from rendering, when private control interfaces should be moved from one location to another, when private control interfaces should remain locked in one location or within an occlusion, and when private control interfaces should be reconfigured (e.g., scaling, redirection, modification, or redesign, etc.) to maintain privacy or security protection, etc.

[0071] Image processing engine 124 can perform one or more image processing operations. In some examples, image processing engine 124 can perform image processing operations based on data from image sensor 102. In some cases, image processing engine 124 can perform image processing operations such as filtering, depigmentation, scaling, color correction, color conversion, segmentation, noise reduction filtering, spatial filtering, artifact correction, etc. Rendering engine 126 can obtain image data generated and / or processed by computing component 110, image sensor 102, XR engine 120, and / or image processing engine 124 and render video and / or image frames for presentation on a display device.

[0072] Although the Extended Reality System 100 is shown to include certain components, those skilled in the art will understand that the Extended Reality System 100 may include more than [other components]. Figure 1 The components shown may include more or fewer components. For example, in some cases, the extended reality system 100 may also include one or more memory devices (e.g., RAM, ROM, cache, etc.), one or more network interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or Figure 1 Other hardware or processing devices not shown in the diagram. See below for details. Figure 11 Illustrative examples of computing devices and hardware components that can be implemented using Extended Reality System 100.

[0073] Figure 2 This is a diagram illustrating an example of an extended reality system 220 worn by a user 200. The extended reality system 220 may include... Figure 1 It contains some or all of the same components as the Extended Reality System 100 shown and described above, and can perform some or all of the same functions as the Extended Reality System 100. While the Extended Reality System 220 is in Figure 2The image shows AR glasses, but the extended reality system 220 can include any suitable type of XR device, such as an HMD or other XR device. The extended reality system 220 is described as an optical see-through AR device that allows user 200 to view the real world while wearing the extended reality system 220. For example, user 200 can view objects 202 in the real-world environment on a plane 204 at a distance from user 200. The extended reality system 220 has an image sensor 218 and a display 210 (e.g., glasses, screen, lens, or other display) that allows user 200 to view the real-world environment and also allows AR content to be displayed thereon. The image sensor 218 can be... Figure 1 The image sensor 102 shown is similar to or the same as the one shown. Although in Figure 2 The image shows an image sensor 218 and a display 210, but in some embodiments, the extended reality system 220 may include multiple cameras and / or multiple displays (e.g., a display for the right eye and a display for the left eye). AR content (e.g., images, videos, graphics, virtual or AR objects, or other AR content) may be projected or otherwise displayed on the display 210. In one example, the AR content may include an enhanced version of object 202. In another example, the AR content may include additional AR content related to object 202 or related to one or more other objects in the real-world environment.

[0074] like Figure 2 As shown, the Extended Reality system 220 may include a computing component 216 and a memory 212, or may communicate with the computing component 216 and the memory 212 via wired or wireless communication. The computing component 216 and the memory 212 may store and execute instructions for performing the techniques described herein. In embodiments where the Extended Reality system 220 communicates with the memory 212 and the computing component 216 (wired or wireless), the device housing the memory 212 and the computing component 216 may be a computing device, such as a desktop computer, laptop computer, mobile phone, tablet computer, gaming terminal, or other suitable device. The Extended Reality system 220 also includes an input device 214 or communicates with it (wired or wirelessly). The input device 214 may include any suitable input device, such as a touchscreen, pen or other pointing device, keyboard, mouse, buttons or keys, microphone for receiving voice commands, gesture input device for receiving gesture commands, any combination thereof, and / or other input devices. In some cases, an image sensor 218 may capture images that can be processed to interpret gesture commands.

[0075] Image sensor 218 can capture color images (e.g., images with red-green-blue (RGB) color components, images with luminance (Y) and chromaticity (C) color components such as YCbCr images, or other color images) and / or grayscale images. As described above, in some cases, extended reality system 220 may include multiple cameras, such as dual front cameras and / or one or more front cameras and one or more rear cameras, which may also include various sensors. In some cases, image sensor 218 (and / or other cameras of extended reality system 220) may capture still images and / or video including multiple video frames (or images). In some cases, image data received by image sensor 218 (and / or other cameras) may be in a raw, uncompressed format and may be compressed and / or otherwise processed (e.g., by an image signal processor (ISP) or other processors of extended reality system 220) before being further processed and / or stored in memory 212. In some cases, image compression may be performed by computing component 216 using lossless or lossy compression techniques (e.g., any suitable video or image compression technique).

[0076] In some cases, image sensor 218 (and / or other cameras of the extended reality system 220) may be configured to also capture depth information. For example, in some embodiments, image sensor 218 (and / or other cameras) may include an RGB depth (RGB-D) camera. In some cases, the extended reality system 220 may include one or more depth sensors (not shown) that are separate from image sensor 218 (and / or other cameras) and may capture depth information. For example, such a depth sensor may acquire depth information independently of image sensor 218. In some examples, the depth sensor may be physically mounted in the same general location as image sensor 218, but may operate at a different frequency or frame rate than image sensor 218. In some examples, the depth sensor may take the form of a light source that can project a structured or textured light pattern onto one or more objects in the scene, the structured or textured light pattern including one or more narrowband lights. Depth information can then be obtained by utilizing the geometrical distortion of the projected pattern caused by the shape of the object surface. In one example, depth information can be obtained from a stereo sensor, such as a combination of an infrared structured light projector and an infrared camera registered to a camera (e.g., an RGB camera).

[0077] In some implementations, the extended reality system 220 includes one or more sensors. These sensors may include one or more accelerometers (e.g., 104), one or more gyroscopes (e.g., 106), and / or other sensors. The sensors may provide velocity, orientation, and / or other position-related information to the computing component 216. In some cases, the sensors may include at least one inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of one or more accelerometers, one or more gyroscopes, and / or one or more magnetometers to measure the specific force, angular rate, and / or orientation of the extended reality system 220. In some examples, the sensors may output measured information associated with the capture of images by the image sensor 218 (and / or other cameras of the extended reality system 220) and / or depth information obtained using one or more depth sensors of the extended reality system 220.

[0078] Computational component 216 can use the output of one or more sensors (e.g., one or more IMUs) to determine the pose of extended reality system 220 (also referred to as head pose) and / or the pose of image sensor 218 (or other cameras of extended reality system 100). In some cases, the pose of extended reality system 220 and the pose of image sensor 218 (or other cameras) may be the same. The pose of image sensor 218 refers to the position and orientation of image sensor 218 relative to a reference frame (e.g., relative to object 202). In some implementations, camera pose can be determined for 6 degrees of freedom (6DOF), which refers to three translational components (e.g., which can be given by X (horizontal), Y (vertical), and Z (depth) coordinates relative to a reference frame such as an image plane) and three angular components (e.g., roll, pitch, and yaw relative to the same reference frame).

[0079] In some aspects, the pose of image sensor 218 and / or extended reality system 220 can be determined and / or tracked by computing component 216 using a visual tracking solution based on images captured by image sensor 218 (and / or other cameras of extended reality system 220). In some examples, computing component 216 can perform tracking using computer vision-based tracking, model-based tracking, and / or simultaneous localization and mapping (SLAM) techniques. For example, computing component 216 can perform SLAM or can communicate (wired or wirelessly) with a SLAM engine (not shown). SLAM refers to a class of techniques in which an environment map (e.g., an environment map modeled by extended reality system 220) is created while simultaneously tracking the pose of the camera (e.g., image sensor 218) and / or extended reality system 220 relative to that map. This map can be called a SLAM map and can be three-dimensional (3D). SLAM technology can be performed using color or grayscale image data captured by image sensor 218 (and / or other cameras of extended reality system 220), and can be used to generate an estimate of the 6DOF pose measurement of image sensor 218 and / or extended reality system 220. Such SLAM technology configured to perform 6DOF tracking can be referred to as 6DOF SLAM. In some cases, the output of one or more sensors can be used to estimate, correct, and / or otherwise adjust the estimated pose.

[0080] In some cases, 6DOF SLAM (e.g., 6DOF tracking) can associate features observed from certain input images from image sensor 218 (and / or other cameras) with a SLAM map. 6DOF SLAM can use feature point association from the input images to determine the pose (position and orientation) of image sensor 218 and / or extended reality system 220 for the input images. 6DOF mapping can also be performed to update the SLAM map. In some cases, the SLAM map maintained using 6DOF SLAM can contain 3D feature points from two or more image triangulations. For example, keyframes can be selected from the input image or video stream to represent the observed scene. For each keyframe, the corresponding 6DOF camera pose associated with the image can be determined. The pose of image sensor 218 and / or extended reality system 220 can be determined by projecting features from the 3D SLAM map onto the image or video frame and updating the camera pose according to a verified 2D-3D correspondence.

[0081] In an illustrative example, computational component 216 can extract feature points from each input image or each keyframe. The feature points used here (also called registration points) are distinctive or identifiable parts of an image, such as a part of a hand, the edge of a table, etc. Features extracted from a captured image can represent different feature points along three-dimensional space (e.g., coordinates on the X, Y, and Z axes), and each feature point can have an associated feature location. Feature points in a keyframe may match (identical to or corresponding to) feature points in previously captured input images or keyframes. Feature detection can be used to detect feature points. Feature detection can include image processing operations that examine one or more pixels of an image to determine if a feature exists at a particular pixel. Feature detection can be used to process the entire captured image or parts of an image. For each image or keyframe, once a feature is detected, a local image patch around the feature can be extracted. Features can be extracted using any suitable technique, such as Scale Invariant Feature Transform (SIFT) (which locates features and generates descriptions of them), Speed-Up Robust Features (SURF), Gradient Location-Orientation Histogram (GLOH), Normalized Cross-Correlation (NCC), or other suitable techniques.

[0082] In some examples, AR (or virtual) objects can be registered or anchored to feature points detected in the scene (e.g., positioned relative to feature points detected in the scene). For example, user 200 might be looking across the street at a restaurant from where user 200 is standing. In response to identifying the restaurant and AR content associated with it, computing component 216 can generate an AR object that provides information related to the restaurant. Computing component 216 can also detect feature points from a portion of an image including a sign on the restaurant and can register the AR object to the feature points of the sign so that the AR object is displayed relative to the sign (e.g., above the sign so that user 200 can easily identify it as being related to the restaurant).

[0083] Extended Reality System 220 can generate and display various AR objects for user 200 to view. For example, Extended Reality System 220 can generate and display virtual interfaces, such as a virtual keyboard, as AR objects for user 200 to input text and / or other characters as needed. The virtual interface can be registered to one or more physical objects in the real world. However, in many cases, real-world objects with unique features that can be used as references for registration purposes may be lacking. For example, if the user is staring at a blank whiteboard, the whiteboard may not have any unique features that can be registered to a virtual keyboard. The outdoor environment may provide less unique points that can be used to register a virtual interface, such as based on: the lack of points in the real world, unique objects that are farther away in the real world than when the user is indoors, the presence of many moving points in the real world, distant points, etc.

[0084] However, in some cases, the extended reality system 220 can utilize the user 200's hands to register the virtual interface. For example, one or more of the user 200's hands and fingers can be used as real-world registration points for the virtual interface (e.g., anchoring the virtual interface in space). By registering the virtual interface to hands and fingers, the challenges of outdoor operation are reduced.

[0085] In some cases, the extended reality system 220 can also track the user 200's hands and / or fingers to allow the user 200 to control virtual interfaces in the AR environment. For example, the extended reality system 220 can track the gestures and / or movements of the user 200's hands and / or fingertips to recognize or translate user interactions with the virtual interface. User interactions may include, for example, but not limited to, moving the virtual interface, resizing the virtual interface, selecting input interface elements in the virtual interface, providing input through the virtual interface, and so on.

[0086] In some examples, image sensor 218 can capture images of a scene associated with user 200, which the augmented reality system 220 can use to detect objects and people / faces in the scene. For example, image sensor 218 can capture images of people / faces and / or any objects in the scene, such as other devices (e.g., recording devices, monitors, etc.), windows, doors, desks, tables, chairs, walls, etc. The augmented reality system 220 can use the images to identify the captured faces and / or objects and estimate the relative positions of such faces and / or objects. For illustration, the augmented reality system 220 can perform face recognition to detect any face in the scene and can use the images captured by image sensor 218 to estimate the position of the face within the scene. As another example, the augmented reality system 220 can analyze images from image sensor 218 to detect any capturing devices (e.g., cameras, microphones, etc.) or signs indicating the presence of capturing devices and estimate the position of the capturing devices (or signs).

[0087] The extended reality system 220 can also use images to detect any occlusions within the user 200's field of view (FOV), which may be located or positioned such that any information rendered on the surface of such an occlusion or within a region of such an occlusion is invisible to other detected users or capture devices or is outside their FOV. For example, the extended reality system 220 can detect that the palm of the user 200's hand is in front of and facing the user 200, and therefore within the user 200's FOV. The extended reality system 220 can also determine that the palm of the user 200's hand is outside the FOV of other detected users and / or capture devices in the scene, and therefore the surface of the user 200's palm is occluded relative to those users and / or capture devices. When the extended reality system 220 presents to the user 200 any AR content that the extended reality system 220 determines should be private and / or protected from being visible to other users and / or capture devices (e.g., the private control interface described herein), the extended reality system 220 may render such AR content on the palm of the user 200's hand to protect the privacy of such AR content and prevent other users and / or capture devices from being able to see the AR content and / or the user 200 from interacting with the AR content.

[0088] Figure 3 This is a diagram showing example landmark points of the hand 300 that can be used to track the position of the hand 300 and its interaction with a virtual interface (such as the proprietary control interface described herein). Figure 3 The landmark points shown correspond to different parts of the hand 300, including landmark point 335 on the palm of the hand 300, landmark point 330 on the thumb of the hand 300, landmark point 332 on the index finger of the hand 300, landmark point 334 on the middle finger of the hand 300, landmark point 336 on the ring finger of the hand 300, and landmark point 338 on the little finger of the hand 300. The palm of the hand 300 can move in three translational directions (e.g., measured in the X, Y, and Z directions relative to a plane (e.g., an image plane)) and three rotational directions (e.g., measured in yaw, pitch, and roll relative to a plane), thus providing six degrees of freedom (6DOF) for registration and / or tracking. The 6DOF motion of the palm in Figure 3 It is shown as a square, as indicated in Figure 340.

[0089] Different joints of the fingers in hand 300 allow for varying degrees of movement, as shown in Figure 340. Figure 3 As shown in the rhombus (e.g., rhombus 333), the base of each finger (corresponding to the metacarpophalangeal joint (MCP) between the proximal phalanx and metacarpal bones) has two degrees of freedom (2DOF), corresponding to flexion and extension, and abduction and adduction. Figure 3As shown in the circles (e.g., circle 331), each superior joint of each finger (corresponding to the interphalangeal joints between the distal, middle, and proximal phalanges) has one degree of freedom (2DOF) corresponding to flexion and extension. As a result, the hand 300 provides 26 degrees of freedom (26DOF), from which the hand 300 and its interactions with the virtual interface rendered by the extended reality system 100 can be tracked.

[0090] The Extended Reality System 100 can use one or more landmark points on the hand 300 to track the hand 300 (e.g., track the pose and / or movement of the hand 300) and to track interactions with a virtual interface rendered by the Extended Reality System 100. As described above, as a result of detecting one or more landmark points on the hand 300, the pose of the landmarks (and therefore the hand and fingers) in their relative physical positions relative to the Extended Reality System 100 can be established. For example, landmark points on the palm of the hand 300 (e.g., landmark point 335) can be detected in an image, and the positions of the landmark points can be determined relative to the image sensor 102 of the Extended Reality System 100. Points (e.g., center points, such as centroids or other center points) of the virtual interface rendered by the Extended Reality System 100 and / or interface elements on the virtual interface selected by or interacting with the hand 300 can be translated into positions on the display 210 of the Extended Reality System 100 (or renderings on the display 210) relative to the landmark points on the palm of the hand 300. In some examples, points of a portion of the virtual interface that interacts with the hand 300 can be registered relative to the location of one or more landmark points on the hand 300.

[0091] In some examples, the extended reality system 100 may also register the virtual interface and / or hand 300 to points in the real world (such as those detected in one or more images) and / or to other parts of the user. For example, in some implementations, in addition to determining the physical pose of the hand 300 relative to the extended reality system 100 and / or the virtual interface, the extended reality system 100 may also determine the location of other landmarks, such as unique points on walls (called keypoints), one or more corners of an object, features on the floor, points on a person's face, points on nearby devices, and so on. In some cases, the extended reality system 100 may place the virtual interface within a specific location relative to keypoints detected in the environment, which may correspond to, for example, objects and / or people detected in the environment.

[0092] In some examples, the pose of the extended reality system 100 (and / or the user 200's head) can be determined using, for example, image data from image sensor 102 and / or measurements from one or more other sensors such as accelerometer 104, gyroscope 106, and / or one or more other sensors (e.g., one or more magnetometers, one or more inertial measurement units (IMUs), etc.). Head pose can be used to determine the position of the virtual interface, hand 300, and / or objects and / or person in the environment.

[0093] In some examples, the virtual interface can remain at the rendered location. While the virtual interface is rendered at the defined location, the user can then virtually interact with it. For example, the virtual interface can remain at the rendered location and the user can virtually input text through it. In some cases, the virtual interface can remain stationary while the user interacts with it. In some cases, the virtual interface can remain at the defined location until a re-registration event is detected by the extended reality system 100. If the virtual interface is a privately controlled interface, the extended reality system 100 can move, remove, or reconfigure the virtual interface if it detects an event indicating that the virtual interface may now be within the FOV of another user or risk object. Risk objects can include any object that can record the virtual interface (and / or associated interactions) or allow the user to see the virtual interface from or through risk objects such as recording devices, windows, glass, doors, etc. For example, the extended reality system 100 can move the virtual interface to a different location determined by the extended reality system 100 to be outside the FOV of another user or risk object. If the extended reality system 100 determines that there are no other locations (or cannot identify other locations) within the user 200's FOV but outside the FOV of other users or risk objects, the extended reality system 100 may simply remove the virtual interface, notify the user that the virtual interface is no longer private, and / or prompt the user to choose whether to leave the currently rendered virtual interface, move the virtual interface to a different location, remove the virtual interface, or perform other operations.

[0094] Figure 4 This is a diagram illustrating an example system flow 400 for generating a proprietary control interface for an XR experience. A proprietary control interface can be generated to allow users to interact with it while preventing other users and recording devices from seeing the control interface or the user's interaction with it, and interpreting such interactions to determine their meaning and the data associated with them.

[0095] In this example, device tracker 420 may receive measurements 402 from accelerometer 104 and gyroscope 106, and image data 404 from image sensor 102. In some examples, measurements 402 may include motion measurements from accelerometer 104 and orientation measurements from gyroscope 106. For example, measurements 402 may include one or more translation vectors (e.g., up / down, left / right, forward / backward) from accelerometer 104 and one or more rotation vectors (e.g., pitch, yaw, roll) from gyroscope 106. Furthermore, image data 404 may include one or more images or frames captured by image sensor 102. One or more images or frames may capture a scene and / or one or more portions of a scene (e.g., one or more areas, objects, people, etc.) associated with extended reality system 100.

[0096] Device tracker 420 may be a tracking component implemented by extended reality system 100. In some examples, device tracker 420 may be part of or implemented by XR engine 120 on extended reality system 100. In other examples, device tracker 420 may be separate from XR engine 120 and implemented by one or more computing components 110 on extended reality system 100.

[0097] Device tracker 420 can use measurements 402 and image data 404 to track the pose (e.g., 6DOF pose) of extended reality system 100. For example, device tracker 420 can fuse visual data from image data 404 with inertial data from measurements 402 to determine the position and motion of extended reality system 100 relative to the physical world (e.g., a scene) and a map of the physical world. In some examples, while tracking the pose of extended reality system 100, device tracker 420 can generate a three-dimensional (3D) map of the scene (e.g., the real world) and / or generate updates to the 3D map of the scene. 3D map updates may include, for example, but not limited to, new or updated features and / or landmarks associated with the scene and / or the 3D map of the scene, location updates identifying or updating the position of extended reality system 100 within the scene and the 3D map of the scene, etc. The 3D map can provide a digital representation of the scene in the real / physical world. In some examples, the 3D map can anchor location-based objects and / or content to real-world coordinates and / or objects. Extended Reality System 100 can use mapped scenes (e.g., scenes in the physical world represented by and / or associated with 3D maps) to merge the physical and virtual worlds and / or merge virtual content or objects with the physical environment.

[0098] Device tracker 420 can provide tracking data 406 generated from measurement values ​​402 and image data 404 to hand tracker 422, XR application 424, and proprietary control interface management engine 122. Tracking data 406 may include the pose of extended reality system 100 and map data calculated by device tracker 420. Map data may include a 3D map of the scene and / or map updates of the 3D map of the scene, as previously described.

[0099] The hand tracker 422 can be a component implemented by the extended reality system 100 for tracking the user associated with the extended reality system 100 (e.g., ...). Figure 2 User 200) hands (for example, Figure 3 The hand tracker 422 (300) and / or the fingertips of the user's hand, as previously described. For simplicity and explanation purposes, the hand tracker 422 will be described herein as a component for tracking the hand. However, it should be noted that in other examples, the hand tracker 422 may track other objects and / or body parts. For example, as previously described, the hand tracker 422 may track the fingers or fingertips of the hand to supplement or replace tracking the hand itself.

[0100] In some examples, the device tracker 420 may be part of or implemented by the XR engine 120 on the extended reality system 100. In other examples, the device tracker 420 may be separate from the XR engine 120 and implemented by one or more computing components 110 on the extended reality system 100.

[0101] The hand tracker 422 can also receive image data 404 from the image sensor 102. The hand tracker 422 can use the image data 404 and tracking data 406 to track hand pose 408 (e.g., hand pose and / or fingers / fingertips). In some examples, the hand tracker 422 can determine hand pose 408, as previously discussed... Figure 3 As described. The hand tracker 422 can then provide hand gestures 408 to the XR application 424. In some examples, the XR application 424 may be an application on the extended reality system 100 designed and / or configured to provide a specific XR experience, such as an AR gaming experience, an AR classroom experience, and / or any other XR experience. The XR application 424 may be part of or implemented by the XR engine 120, or it may be separate from the XR engine 120.

[0102] The private control interface management engine 122 can use image data 404 and tracking data 406 to determine when, how, and / or where to render the private control interface as part of the XR experience. The private control interface can be a virtual control interface designed and / or configured to: hide the control interface, interactions between control interfaces, and / or data inputs / outputs associated with the control interface from any user and / or recording device in the scene that is not authorized to view such information and / or all users and / or recording devices in the scene other than the intended user associated with the extended reality system 100.

[0103] In some examples, the private control interface management engine 122 can determine whether a virtual control interface should be rendered as a private control interface in order to hide the virtual control interface and associated interactions / data from users and / or recording devices in the scene. The private control interface management engine 122 can determine whether to render a virtual control interface as a private control interface based on one or more factors, such as user preferences, system preferences, application preferences, the interface type of the virtual control interface (e.g., password hints, PIN pad, medical forms, exam or quiz questions, voting, banking interfaces, private forms, competitive game control interfaces, etc.), the type of data input / output associated with the virtual interface (e.g., medical data, financial data, verification credentials, confidential data, personal or private data, protected data, security information, sensitive data, etc.), the security information or access level associated with the virtual interface and / or associated data, the application type associated with the virtual interface (e.g., banking, healthcare, security, gaming, etc.), the current environment associated with the extended reality system 100 (e.g., public areas, surveillance areas, high-crime areas, densely populated areas, areas with nearby users or devices, etc.), and / or any other factors.

[0104] In some cases, the private control interface management engine 122 can identify private regions in the scene where private control interfaces can be rendered. Private regions can include areas, areas, locations, objects, surfaces, and / or planes in the scene that are within the field of view (FOV) of the user associated with the extended reality system 100, but outside the FOV of other users and / or any known or potential recording devices in the scene. In other words, a private region can be somewhere in the scene that a user might see but other users and / or recording devices in the scene cannot. In some cases, if occlusion in the scene prevents (e.g., blocks) other users and / or recording devices from seeing the private region, then the private region can be determined to be outside the FOV of other users and / or recording devices.

[0105] For example, a stack of books on a table might obstruct other users and / or recording devices in the scene from seeing an area of ​​the table on a particular side of the stack of books (e.g., the side opposite to other users and / or recording devices). If there is no obstruction of the table area for a user associated with the extended reality system 100 (e.g., within the user's field of view), the stack of books can be considered an obstruction and the table area can be considered a private area that can be used to render a private control interface for the user without exposing the private control interface and / or associated interactions / data to other users and / or recording devices in the scene.

[0106] In some examples, to identify private areas in a scene, the private control interface management engine 122 can use image data 404 to identify any objects and faces in the scene and their associated positions relative to the extended reality system 100. Objects may include risky objects. Risky objects may include any device with recording capabilities, such as a camera, or anything that would allow other users to see the private area, such as windows, open doors, or glass doors. In some cases, the private control interface management engine 122 may use information about objects and / or faces in the scene and their associated positions within the scene to determine whether any area, size, location, object, surface, and / or plane in the scene can be selected as a private area outside the FOV of other users and / or risky objects but within the FOV of the user associated with the extended reality system 100. For example, the private control interface management engine 122 may determine whether any area or plane exists in the scene within a relative position, angle, distance, etc., that would prevent other users and / or risky objects from seeing that area or plane.

[0107] In some cases, the private control interface management engine 122 can identify any objects and faces (and their associated locations) in a scene based on input from users and / or other devices. For example, a user can input information about any user and object in the scene, including, for example, a description of such users and objects, a description of the location of such users and objects, indications of possible occlusion and / or private areas, and so on. As another example, such information can be reported automatically from other devices in the scene. If such devices are trusted, the private control interface management engine 122 can rely on such information when identifying objects, faces, and associated locations.

[0108] In some cases, the private control interface management engine 122 can perform occlusion calculations to determine if any occlusions prevent other users and / or risky objects in the scene from seeing specific areas, sizes, locations, objects, surfaces, and / or planes within the scene. For example, the private control interface management engine 122 can determine if any areas in a mapped scene (e.g., a scene in the physical world represented by and / or associated with a 3D map) occlude for all users except the user associated with the extended reality system 100. In some examples, the private control interface management engine 122 can use ray tracing to find occlusions and / or areas in the scene that occlude for other users and / or risky objects. In other examples, the private control interface management engine 122 can analyze image data 404 to determine if any occlusions in the scene prevent other users and / or risky objects from seeing specific areas within the scene. Occlusions can include any object or obstacle to the field of view (FOV) of other users and / or risky objects in the scene. Furthermore, occlusions can be used to identify private areas that can be used to render the private control interface for the user.

[0109] In some examples, if a surface in a scene faces the user associated with the extended reality system 100 within a specific angle, the private control interface management engine 122 can infer or assume that the surface in the scene is a private region. For example, the private control interface management engine 122 can identify planar surfaces in the scene that face the user within a programmable angle and infer or assume that such planar surfaces are private. A programmable angle can be, for example, an angle within a threshold difference or distance (e.g., angle) relative to the angle or orientation of the user's face. For example, if the user's face is at a 90-degree angle, the programmable angle can be a 90-degree angle or an angle within 90 degrees x degrees. In some cases, the private control interface management engine 122 can use plane segmentation (e.g., via Random Sample Consensus (RANSAC), Hough transform, neural networks, etc.) on the mapped scene to find planar surfaces facing the user within a programmable angle. In some examples, the private control interface management engine 122 can analyze detected objects, faces, and / or occlusions, as described above, to confirm that the planar surface is indeed private.

[0110] In some cases, when the private control interface management engine 122 identifies a private region, it can also determine the configuration of the private region (e.g., size, shape, angle, distance, etc.) and / or the configuration of the private control interface (e.g., size, shape, angle, design, scheme, etc.) to suit and / or align the private control interface on or within the private region. For example, the private control interface management engine 122 can determine the size and shape of the private region and select the size, shape, and / or design of the private control interface to ensure that the private control interface is suitable for and aligned with the private region.

[0111] Next, the private control interface management engine 122 can provide private control interface data 410 to the XR application 424. The private control interface data 410 may include indications of private regions that the XR application 424 can use to render the private control interface. In some examples, the private control interface data 410 may include indications of the location and configuration (e.g., shape, size, angle, etc.) of the private regions. In some examples, the private control interface data 410 may also include configuration information for the private regions. Configuration information may include, for example, size, shape, design, interface type or scheme, layout, alignment, and / or any other characteristics that the XR application 424 can use to render, adapt, align, and / or lock the private control interface on the private regions.

[0112] As previously described, XR application 424 can also receive tracking data 406 from device tracker 420 and hand pose 408 from hand tracker 422. XR application 424 can use tracking data 406, hand pose 408, and private control interface data 410 to resize, position, align, and render a private control interface on a private region. Rendering a private control interface on a private region may include: providing a virtual / digital overlay of the private control interface on the private region; and / or rendering / displaying the private control interface so that it appears to be located, presented, or projected onto the private region for a user viewing the private control interface from / through extended reality system 100. In some cases, XR application 424 may use tracking data 406, hand pose 408, and private control interface data 410 to map the private control interface onto a private region on a 3D map of the scene. In addition, in some cases, XR application 424 may use tracking data 406, hand gesture 408 and private control interface data 410 to map and / or transform the user’s interaction with the private control interface (e.g., via the hand) and / or otherwise allow the private control interface to interact with the user.

[0113] XR application 424 may provide output 412 to display 210, which may present a private control interface to a user. Output 412 may include rendering of the private control interface within a private area. For example, output 412 may include a virtual / digital overlay of the private control interface on the private area (e.g., rendered on, appearing on, anchored on, locked on, etc.). In some cases, output 412 may include rendering / displaying the private control interface so that it appears to a user viewing the private control interface from / through extended reality system 100 as if the private control interface were located, presented on, or projected onto the private area. In some cases, output 412 may include additional information associated with the private control interface and / or XR content provided by XR application 424.

[0114] Figure 5 This is a flowchart illustrating an example process 500 for generating a private control interface for an XR experience. The operations outlined herein are non-limiting examples provided for illustrative purposes and can be implemented in any combination thereof, including excluding, adding, or modifying combinations of certain operations or steps. Furthermore, for simplicity and explanation purposes, refer to [reference provided]. Figure 1 The operation is described by the extended reality system 100 configured as the operation of the practice process 500.

[0115] In box 502, the extended reality system 100 may first detect privacy conditions. Privacy conditions can be any event and / or condition configured to trigger the rendering of a private control interface. For example, privacy conditions can be any event and / or condition that triggers the extended reality system 100 to render a virtual interface in a private mode (e.g., on a private area) to prevent other users or devices from visible access to the virtual interface, user interactions with the virtual interface, and / or data associated with the virtual interface.

[0116] In some examples, privacy conditions can be defined based on one or more factors. For example, privacy conditions can be defined by or based on the following: user preferences, system preferences (e.g., preferences from extended reality system 100), application preferences (e.g., preferences from XR application 424), application and / or content items, laws or regulations, best practices, security specifications, potential security implications, the type of XR application, rules regarding conditions in the environment (e.g., whether the environment is publicly accessible, whether there are other users or devices in the environment, whether it is a densely populated area, whether there are any privacy expectations, etc.), and the data types associated with the XR application and / or virtual interface (e.g., personal user data, authentication data (passwords, PINs, codes, etc.), medical information, financial information, player information in multi-user games, security information, password-protected data, data protected by access control lists (ACLs) and / or permissions, user account data, privileged or confidential data, etc.).

[0117] In box 504, the extended reality system 100 can determine whether a private area has been detected. The extended reality system 100 can detect private areas, as previously mentioned... Figure 4As explained above. Furthermore, a private region can be a location, area, region, plane, object, and / or surface that is only visible to the user associated with the Extended Reality System 100 (e.g., not visible to other users or recording devices), which the Extended Reality System 100 can use to render a private control interface. In some cases, the Extended Reality System 100 can make exceptions for certain predetermined users, devices, and / or objects, allowing it to treat a region as a private region even if it is visible to those users, devices, and / or objects. For example, in a multi-game scenario, if a user associated with the Extended Reality System 100 is playing as part of a team that includes other users in the scene, then those other users can be allowed to see the private control interface. Therefore, when identifying a private region, the Extended Reality System 100 can ignore the fact that other users might see the private region and still select that region as a private region to render the private control interface.

[0118] If the Extended Reality System 100 detects a private region at box 504, it can render a private control interface on the private region at box 508. In some cases, if the Extended Reality System 100 detects multiple private regions, it can select a specific private region and / or prioritize them. The Extended Reality System 100 can select / prioritize private regions based on one or more factors. For example, the extended reality system 100 may select / prioritize private regions based on the following: user preferences, features associated with the private region (e.g., shape, size, angle, position, relative distance, position relative to the user and / or other users / devices' FOV, motion (or lack of motion) associated with the private region, etc.), confidence level of the private region outside the FOV of other users and / or risky objects, type of private region (e.g., user body parts, surfaces on objects, planes facing the user at certain angles, etc.), number and / or type of occluders blocking other users and / or risky objects (e.g., larger objects or objects with greater obstruction capabilities may be given higher priority / weight), data type used to identify the private region (e.g., self-reported data from other devices, user-reported data, object detection data, plane segmentation data, image data, occluder data, biometric data, etc.), and confidence level of occluders.

[0119] In some cases, before rendering the private control interface on a private region, at box 506, the extended reality system 100 may optionally determine whether any private control interface preference settings exist. Private control interface preferences may include, but are not limited to, preferences for configuring the private control interface (e.g., shape, size, design, scheme, layout, render height, depth, etc.), preferences for whether a virtual interface should be rendered as a private control interface, timing preferences, rendering preferences, preferences for whether to lock the private control interface on a private region or allow the private control interface to be turned off / removed after a certain period or event, preferences for how to manage the private control interface if movement is detected relative to the private control interface or its surrounding environment (e.g., private region, objects in the scene, users in the scene, occlusions, users, etc.), privacy levels, private region selection and / or priority preferences (e.g., if multiple private regions are detected) and / or any other preferences.

[0120] If Extended Reality System 100 recognizes any private control interface preference, then in box 510, Extended Reality System 100 can select such private control interface preference, and in box 508, it uses such preference to render the private control interface on the private region.

[0121] Returning to box 504, if the Extended Reality System 100 does not detect a private area, then in box 512, the Extended Reality System 100 may determine whether to render the control interface in a publicly visible area. A publicly visible area can include any area or region that may be visible to other users and / or risky objects (e.g., recording devices, windows, glass doors, etc.). In some cases, the Extended Reality System 100 may prompt the user to choose whether to render the control interface in a publicly visible area. In other cases, the Extended Reality System 100 may determine whether to render the control interface in a publicly visible area based on predefined rules and / or preferences regarding how to proceed when no private area is found. Predefined rules and / or preferences may include, for example, user preferences, rules based on the type of data and / or application associated with the control interface, security levels, one or more conditions associated with the scene (e.g., expectations of privacy, the number of users and / or devices in the scene, the type of area associated with the scene, etc.), one or more risk factors, the urgency of rendering the control interface and / or the flexibility to delay rendering the control interface, and / or any other rules or preferences.

[0122] In box 514, if the Extended Reality system 100 determines that the control interface is to be rendered in a publicly visible area, the Extended Reality system 100 may continue to render the control interface in a specific area within the scene, even if such an area is potentially visible to other users and / or risky objects. Alternatively, if the Extended Reality system 100 determines that the control interface is not to be rendered in a publicly visible area, then in box 516, the Extended Reality system 100 may skip rendering the control interface. The Extended Reality system 100 may skip rendering the control interface entirely, or it may delay rendering for a specific period of time or until a specific event occurs. For example, the Extended Reality system 100 may delay rendering the control interface until it receives a request for rendering the control interface from the user or until a certain amount of time has elapsed.

[0123] In some cases, after a certain amount of time has elapsed, the extended reality system 100 may return to box 504 to determine if it can detect a private region. In some examples, after a certain period of time, the extended reality system 100 may also, or alternatively, ask the user whether to render the control interface in the publicly visible area and / or continue trying to find a private region to render the control interface.

[0124] Figure 6A An example private control interface rendering 600 is shown. Here, the extended reality system 100 has detected a private region 602 that is visible to the user 200 associated with the extended reality system 100, but not visible to other users and / or risk objects (e.g., recording devices, windows, glass doors, etc.). In this example, the private region 602 is the surface of a notebook held by the user 200 using the user's hand 300. The surface of the notebook faces the user 200 and is away from the open door 606A and another user 606B in scene 608. Therefore, the surface of the notebook (e.g., the private region 602) is not visible from the open door 606A or within the FOV of the other user 606B.

[0125] Scene 608 can be the physical environment (e.g., in the real / physical world), setting, and / or surrounding environment of user 200 and extended reality system 100. For example, scene 608 can be an office, room, lobby, garage, street, field, sidewalk, elevator, and / or any other physical space in the real / physical world in which user 200 and extended reality system 100 are located (and / or around / surround user 200 and extended reality system 100).

[0126] In some examples, the extended reality system 100 can verify that the private area 602 is not visible to other users and / or risky objects behind the user 200. For example, the extended reality system 100 can use a rear-facing image sensor and / or a user body model to determine whether the private area 602 is visible to other users and / or risky objects behind the user 200.

[0127] Since the extended reality system 100 detects a private region 602 in scene 608, the extended reality system 100 can render a private control interface 604 on the private region 602. In this example, the private control interface 604 is a virtual keyboard. The user 200 can use the virtual keyboard to provide input to an XR application (e.g., 424) to obtain an XR experience associated with the virtual keyboard. The private control interface 604 can remain locked on the private region 602 or can be removed once the user 200 has finished using the private control interface 604, the XR application no longer needs the private control interface 604, the private control interface 604 has timed out, the user 200 moves the laptop (e.g., the private region 602), or another user moves and the private region 602 becomes visible to that user.

[0128] Figure 6B Another example of private control interface rendering 610 is shown. In this example, the extended reality system 100 has used the surface of the palm of user 200's hand 300 as a private region 612 to render the private control interface 614. The palm of user 200's hand 300 faces user 200 and is away from the open door 606A and another user 606B in scene 608. Therefore, the palm of user 200's hand 300 is visible to user 200 but not to the open door 606A or the other user 606B.

[0129] In some examples, before using the user's palm (or any other body part) as a private region, the extended reality system 100 can use biometric data to verify that the palm belongs to the user. For example, the extended reality system 100 can determine whether the palm biometrics are recognized (e.g., matching previously registered biometrics associated with the user). If the palm biometrics are recognized / verified, the extended reality system 100 can render the private control interface 614 on the palm. Otherwise, the extended reality system 100 can choose not to render the private control interface 614 on the palm. This provides additional security to ensure that the private control interface is not rendered on other users' hands or body parts.

[0130] The private control interface 614 in this example includes option buttons. Therefore, Figure 6B The private control interface 614 in the middle is different from Figure 6AThe private control interface 604 is used. The private control interface 614 can differ for one or more reasons. For example, the private control interface 614 can differ because the type of requested / expected data / input is different. To illustrate, if the user needs to enter text, the rendered private control interface could be a keyboard. On the other hand, if the user does not need to enter text but needs to select options, the rendered private control interface could be... Figure 6B The interface for the option buttons shown.

[0131] As another example, the private control interface 614 may vary depending on rendering requirements, such as the size of the private region (e.g., a smaller private region may trigger a smaller interface or be better suited to different interface designs / schemes within a smaller region), the shape of the private region, user preferences, etc.

[0132] Figure 6C Another example of a private control interface rendering 620 is shown. In this example, the private area 622 is a flat surface on a table 626. The private area 622 is within the field of view (FOV) of user 200. However, an obstruction 624 on the table 626 blocks the visibility of the private area 622 from the open door 606A. The obstruction 624 also blocks the visibility of the private area 622 from other users 606B. Therefore, the private area 622 is visible to user 200 but not to door 606A or other users 606B.

[0133] In this example, the obstruction 624 is a laptop standing upright on table 626, obscuring the flat surface of table 626 corresponding to the private area 622. Furthermore, the extended reality system 100 has rendered a private control interface 604 on the private area 622 because the private area 622 is obscured relative to the open door 606A and other users 606B by the laptop on table 626 (e.g., obstruction 624). User 200 can interact with the private control interface 604 rendered on the private area 622 without exposing such interaction and / or associated data to other users 606B or to any user or recording device that may be outside the open door 606A.

[0134] Figure 6DAnother example of a private control interface rendering 630 is shown. In this example, a private area 632 is on the flat surface of table 626, but is obscured by the hand 300 and arm 638 of user 200. The private area 632 faces user 200 and the obstructions (e.g., hand 300 and arm 638) do not block the user's visibility of the private area 632. However, the visibility of the private area 632 from an open door 606A or by another user 606B is blocked by user 200's hand 300 and arm 638. Therefore, the private area 632 is within user 200's field of view (FOV) but outside the FOV of open door 606A or user 606B.

[0135] The private control interface in this example includes a joystick 636 and a button 634. Joystick 636 and button 634 are rendered on a private region 632. Because the private region 632 is a flat, horizontal surface, joystick 636 and button 634 are easier to use when rendered on the private region 632, which is the opposite of when the private region is a vertical surface, as... Figures 6A-6B As shown. Therefore, the extended reality system 100 may choose or prioritize the joystick 636 and button 634 as interface designs / solutions for use in this example rendering 630. However, in other examples where the private area is a vertical surface, such as Figures 6A-6B As shown, the extended reality system 100 may alternatively choose or prioritize different interface designs / schemes that may be easier to use on a vertical surface, such as a virtual keyboard or option buttons.

[0136] Figure 7 An example rendering flow 700 is shown for requesting rendering preferences from user 200 and rendering a private control interface based on the preferences received from user 200. In the first view 702 of flow 700, when extended reality system 100 needs to render a control interface that extended reality system 100 determines should be private, extended reality system 100 may render a prompt 704 to user 200, requesting user 200 to provide a private control area on which the control interface is rendered.

[0137] In some cases, a 704 warning may include or represent a placeholder box for a private control interface. In some examples, the placeholder box (e.g., warning 704) may be world-locked or header-locked. In response to a 704 warning, user 200 may provide a private region and / or move the 704 warning (e.g., the placeholder box) to a private region where user 200 wants to render the private control interface. For example, user 200 may box the 704 warning on a private region depending on how user 200 wants to render the private control interface.

[0138] In the second view 710 of process 700, user 200 has provided private region 706 and moved prompt 704 onto private region 706. Extended Reality System 100 can then determine that user 200 has selected to render the private control interface on private region 706, as shown in second view 710.

[0139] In the third view 712 of process 700, the extended reality system 100 may render a private control interface 714 on a private region 706, consistent with how the user 200 places or frames a cue 704 on the private region 706. In some examples, the private control interface 714 rendered by the extended reality system 100 may be globally locked to the private region 706. In some cases, the private control interface 714 may move with the private region 706 at any time it is moved. In other cases, the extended reality system 100 may remove the private control interface 714 if the private region 706 is moved or removed. In still other cases, to maintain a consistent placement of the private control interface 714, the extended reality system 100 may continue to render the private control interface 714 at its current location while the private region 706 is moved / removed, regardless of privacy concerns (e.g., even if doing so would make the private control interface 714 visible to an open door 606A or another user 606B).

[0140] Figure 8 Example security indicators are shown in rendering 800. Here, a private control interface 806 is rendered on a private region 804 in scene 802. Private region 804 is a flat surface on a table occluded by object 808. In this example, object 808 is a notebook. However, the object providing occlusion can be any other object capable of blocking the visibility of private region 804, such as a clipboard, telephone, hand, book, monitor, chair, lamp, etc.

[0141] Because object 808 is blocking the visibility of private region 804, extended reality system 100 can determine that private control interface 806 is private (e.g., blocked relative to other users or risky objects, outside the FOV of other users or risky objects, etc.). Therefore, extended reality system 100 can render a security indicator 810 indicating to the user that the private control interface is currently private.

[0142] If the object 808 obscuring the private control interface 806 is removed and the private control interface 806 becomes exposed or visible to other users or risky objects, the Extended Reality system 100 can determine that the private control interface 806 being rendered is no longer private. The Extended Reality system 100 can then render a security indicator 812 to indicate to the user that the private control interface 806 is no longer private. The user can request the removal or retention of the private control interface 806. If the user requests the removal of the private control interface 806, the Extended Reality system 100 can stop rendering the private control interface 806 accordingly. On the other hand, if the user requests to leave the private control interface 806 in the rendering position, the Extended Reality system 100 can continue rendering the private control interface 806, even though the private control interface 806 is no longer private.

[0143] Figure 9 An example scheme 900 for verifying biometric information before rendering a private control interface on a user's body is shown. The extended reality system 100 can verify biometric information as a security measure before rendering the private control interface on a user's body to ensure that the private control interface is not rendered on another user's body.

[0144] In the first view 902, the user raises their hand 300, with the palm 904 facing the extended reality system 100. The extended reality system 100 can capture one or more images of the palm 904 and analyze the images to identify features 906 on the palm 904. The extended reality system 100 can use the features 906 to identify the palm 904. The extended reality system 100 can also use the features 906 as a palm biometric feature to verify whether the palm 904 belongs to the user or another user.

[0145] In some examples, the extended reality system 100 may implement a neural network to identify feature 906 on palm 904 and recognize palm 904 as belonging to a user. The extended reality system 100 can identify palm 904 as belonging to a user by matching feature 906, which can be used as a palm biometric, with previously registered biometric data associated with the user's palm. If the extended reality system 100 determines that the biometrics of palm 904 (e.g., feature 906) match previously registered biometrics associated with the user's palm, the extended reality system 100 can recognize palm 904 as the user's palm (as shown in operation 908). On the other hand, if the extended reality system 100 determines that the biometrics of palm 904 do not match previously registered biometrics associated with the user's palm, the extended reality system 100 can determine that palm 904 is not recognized (e.g., not verified as the user's palm) (as shown in operation 914).

[0146] In view 910, after recognizing the hand 904 as belonging to a user, the extended reality system 100 can render a private control interface 912 on the hand 904. Alternatively, in view 916, if the extended reality system 100 does not recognize the hand 904, it may not render the private control interface on the hand 904, as doing so could expose the private control interface and associated data to another user.

[0147] Having disclosed the example system, components, and concepts, this disclosure now turns to example method 1000 for generating private control interfaces for XR experiences, such as... Figure 10 As shown. The steps outlined herein are non-limiting examples provided for illustrative purposes and can be implemented in any combination thereof, including excluding, adding, or modifying certain combinations of steps.

[0148] At box 1002, method 1000 may include determining the pose of the extended reality device (e.g., 100, 220) within a mapped scene of the physical environment associated with the extended reality device. In some examples, the pose of the extended reality device may be calculated based on one or more images of the scene (in the physical / real world / environment) obtained by one or more image sensors (e.g., 102, 118) and measurements from one or more other sensors. In some cases, one or more other sensors may include an accelerometer (e.g., 104) and a gyroscope (e.g., 106). Furthermore, the measurements may include rotation vectors (e.g., yaw, pitch, roll) and motion vectors (e.g., up / down, left / right, forward / backward).

[0149] At box 1004, method 1000 may include detecting private regions (e.g., 602, 612, 622, 632) in the physical environment (and / or a scene in the physical world / an environment mapped by a 3D map) and the position of the private regions relative to the pose of the extended reality device. In some examples, a private region may include a region estimated to be within the FOV of the user (e.g., 200) of the extended reality device and outside the respective FOV of a person in the physical environment (and / or the mapped scene), a recording device (e.g., a camera) in the physical environment (and / or the mapped scene), and / or an object in the physical environment (and / or the mapped scene) that allows viewing access from outside the physical environment. In some cases, objects may include, for example, but not limited to, windows, glass doors, open doors, and / or any object that allows viewing access from other users.

[0150] In some examples, detecting private regions may include: calculating face location data associated with a first face corresponding to a person; calculating the person's field of view (FOV) based on the face location data; and determining, based on the person's FOV and the location of the private region, that the private region is occluded relative to the person's field of view by one or more objects located between the private region and the person. In some cases, the first face may be detected based on an image of the captured mapped scene and the first face. Furthermore, in some cases, face location data may be calculated using a neural network configured for face recognition and / or location estimation.

[0151] In some aspects, detecting private regions may also include: calculating face location data associated with a second face corresponding to a user; calculating the user's field of view (FOV) based on the face location data associated with the second face; and determining whether the private region is within the user's FOV based on the location of the private region. In some examples, the second face can be detected based on an image capturing a mapped scene (and / or physical environment). For example, the image may also capture a second face, which can be analyzed to detect the second face within the image.

[0152] In some examples, the private region may include the surface of a body part associated with the user, and detecting the private region may include determining that the surface of the body part faces the user and is away from people, recording devices, and / or objects in the physical environment. In some cases, the body part may include a hand, and the surface of the body part may include the palm of the hand. In some examples, detecting the private region may include detecting biometric information associated with the palm of the hand based on one or more images of the palm; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the user; and determining that the biometric information matches the previously registered biometric information.

[0153] In some cases, detecting private regions may include: detecting one or more objects in a physical environment (and / or within a mapped scene); detecting one or more occluders in a physical environment (and / or within a mapped scene) based on one or more objects in the physical environment (and / or within a mapped scene); and detecting private regions by determining whether one or more occluders obstruct the visibility of people, recording devices, and / or objects to the private region. In some examples, detecting one or more occluders in a physical environment (and / or within a mapped scene) may include tracing ray paths within the physical environment (e.g., ray tracing).

[0154] In box 1006, method 1000 may include mapping a virtual private control interface (e.g., 604, 614, 634, 636) to a private region based on the pose of the extended reality device and the detected private region. In some examples, the virtual private control interface may be mapped to a location within a private region in the mapped scene and / or physical environment. In some cases, the virtual private control interface may include one or more input interface elements. In some examples, the virtual private control interface may include a virtual keyboard, a virtual keypad, a virtual joystick, a virtual controller, a virtual button, and / or a virtual input field. A virtual private control interface may refer to a virtual interface that is rendered as if located in and / or appears to be located in a private region / location in the physical scene / environment (e.g., within the user's FOV but outside the FOV of other people and / or risky objects), is hidden relative to the FOV of other people and / or risky objects, and / or is occluded / obfuscated relative to the FOV of other people and / or risky objects.

[0155] At box 1008, method 1000 may include rendering a virtual private control interface within a private region in the mapped scene. Rendering the private control interface within the private region may include rendering a virtual / digital overlay of the private control interface on or within the private region and / or rendering the private control interface so that it appears to a user viewing the private control interface from / through an extended reality system (e.g., 100, 220) as if it were located, presented, or projected onto the private region.

[0156] In some cases, rendering a virtual private control interface (VPC) within a private region may include resizing and / or aligning the VPC to fit within and be aligned with the private region. In other cases, the VPC may be an augmented reality (AR) interface, and this AR interface may be rendered via an AR application.

[0157] In some aspects, method 1000 may include determining that the area is outside the corresponding FOV of a person, recording device, and / or object. In some examples, method 1000 may determine that the area is outside the corresponding FOV of a person, recording device, and / or object by detecting one or more objects in the physical environment and determining that the corresponding FOV of the person, recording device, and / or object is occluded by one or more objects.

[0158] In some aspects, mapping a virtual private control interface to a private region and rendering the virtual private control interface within the private region may include: determining a first size and / or a first orientation of the private region; determining a second size and / or a second orientation of the virtual private control interface based on the first size and / or the first orientation of the private region; generating the virtual private control interface according to the second size and / or the second orientation; and aligning the virtual control interface with the private region. In some examples, the second size of the virtual private control interface may match or fit within the first size of the private region, and the second orientation of the virtual control interface may be at least partially aligned with the first orientation of the private region.

[0159] In some aspects, method 1000 may include identifying the palm of a hand used to render a second virtual private control interface, and prior to rendering the second virtual private control interface on the palm of the hand, detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand, comparing the biometric information with previously registered biometric information associated with a previously registered palm of the user, and determining whether the biometric information matches the previously registered biometric information.

[0160] In some examples, method 1000 may further include: determining that the biometric information does not match previously registered biometric information, and in response to determining that the biometric information does not match previously registered biometric information, determining to render a second virtual private control interface on the palm of the hand that is not in hand.

[0161] In some examples, method 1000 may be performed by one or more computing devices or apparatuses. In an illustrative example, method 1000 may be performed by... Figure 1 The extended reality system 100 shown and / or having Figure 11 One or more computing devices, as illustrated in the computing device architecture 1100, are used to perform the actions. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of method 1000. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, head-mounted display, mobile device, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are decoupled from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to transmit data.

[0162] Components of a computing device can be implemented in circuitry. For example, components may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. A computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to transmit and / or receive data, any combination thereof, and / or other components. The network interface may be configured to transmit and / or receive Internet Protocol (IP)-based data or other types of data.

[0163] Method 1000 is shown as a logic flowchart, whose operations represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0164] Additionally, method 1000 can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors, implemented in hardware, or a combination thereof. As described above, the code can be stored, for example, in the form of a computer program comprising multiple instructions executable by one or more processors on a computer-readable or machine-readable storage medium. The computer-readable or machine-readable storage medium can be non-transitory.

[0165] Figure 11 An example computing device architecture 1100 is shown, illustrating an example computing device capable of implementing the various technologies described herein. For example, computing device architecture 1100 can implement… Figure 1 The extended reality system 100 and / or shown Figure 2At least some portions of the extended reality system 220 are shown. Components of the computing device architecture 1100 are shown to be in electrical communication with each other using a connection 1105 such as a bus. The example computing device architecture 1100 includes a processing unit (CPU or processor) 1110 and a computing device connection 1105 that couples various computing device components to the processor 1110. The computing device components include computing device memory 1115, such as read-only memory (ROM) 1120 and random access memory (RAM) 1125.

[0166] The computing device architecture 1100 may include a cache of high-speed memory that is directly connected to, adjacent to, or integrated into the processor 1110. The computing device architecture 1100 can copy data from memory 1115 and / or storage device 1130 to the cache 1112 for fast access by the processor 1110. In this way, the cache can provide performance improvements by avoiding latency for the processor 1110 while waiting for data. These and other modules can control or be configured to control the processor 1110 to perform various actions. Other computing device memory 1115 may also be available. Memory 1115 may include various different types of memory with different performance characteristics. The processor 1110 may include any general-purpose processor and hardware or software services of dedicated processors stored in storage device 1130 and configured to control the processor 1110, as well as software instructions incorporated into the processor design. The processor 1110 may be a standalone system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0167] To enable users to interact with the computing device architecture 1100, input device 1145 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. Output device 1135 can also be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some cases, multi-mode computing devices allow users to provide multiple types of input to communicate with the computing device architecture 1100. Communication interface 1140 typically governs and manages user input and computing device output. There are no limitations on operation on any particular hardware arrangement, therefore the basic features described herein can be readily replaced with improved hardware or firmware arrangements as they are developed.

[0168] Storage device 1130 is a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic tape cassette, flash memory card, solid-state storage device, digital universal disk, magnetic tape cartridge, random access memory (RAM) 1125, read-only memory (ROM) 1120, and combinations thereof. Storage device 1130 may include software, code, firmware, etc., for controlling processor 1110. Other hardware or software modules are contemplated. Storage device 1130 may be connected to computing device connection 1105. On the one hand, a hardware module performing a specific function may include software components stored in a computer-readable medium combined with necessary hardware components, such as processor 1110, connection 1105, output device 1135, etc., to perform that function.

[0169] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include, but does not include, non-transitory media on which data can be stored, but excludes carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted in any suitable manner, including memory sharing, message passing, token passing, network transmission, etc.

[0170] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0171] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that these embodiments can be practiced without these specific details. For clarity of explanation, in some cases, the technology may be presented as comprising individual functional blocks, including devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0172] Various embodiments may be described above as processes or methods, which are described as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0173] The processes and methods described in the examples above can be implemented using computer-executable instructions stored or otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a specific function or group of functions. Part of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, and so on.

[0174] Devices implementing the processes and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) that perform the necessary tasks may be stored on a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. As a further example, such functionality may also be implemented on circuit boards between different chips or between different processes executing in a single device.

[0175] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means of providing the functionality described in this disclosure.

[0176] In the foregoing description, aspects of this application have been described in conjunction with specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, while illustrative embodiments of this application have been described in detail herein, it should be understood that the concepts of the invention can be embodied and employed in other ways, and the appended claims are intended to be construed as including such variations unless limited by prior art. Various features and aspects of the above applications can be used individually or in combination. Furthermore, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, methods are described in a particular order. It should be appreciated that, in alternative embodiments, these methods may be performed in a different order than that described.

[0177] Those skilled in the art will recognize that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols without departing from the scope of this specification.

[0178] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits), or any combination thereof.

[0179] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., via a wired or wireless connection to another component, and / or other suitable communication interface).

[0180] The use of "at least one" and / or "one or more" in the language of a claim or other set of expressions indicates that one or more members of that set (in any combination) satisfy the claim. For example, the language in a claim that expresses "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim term "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The use of "at least one" and / or "one or more" in the language set does not limit the set to items listed in that set. For example, the language in a claim that expresses "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0181] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0182] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or multi-purpose integrated circuit devices, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product and may include packaging materials. The computer-readable medium can include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. These technologies may also, or alternatively, be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures and can be accessed, read and / or executed by a computer, such as a propagating signal or wave.

[0183] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0184] Illustrative examples of this disclosure include:

[0185] Example 1: A method operable on an extended reality device, the method comprising: determining the pose of the extended reality device within a mapped scene of a physical environment associated with the extended reality device; detecting a private region in the physical environment and the position of the private region relative to the pose of the extended reality device, the private region including areas outside the respective FOV of at least one of a person, a recording device in the physical environment, and an object in the physical environment that can be viewed from outside the physical environment, estimated to be within the field of view (FOV) of a user of the extended reality device; mapping a virtual private control interface to the private region based on the pose of the extended reality device and the detected private region, the virtual private control interface including one or more input interface elements; and rendering the virtual private control interface within the private region in the mapped scene.

[0186] Example 2: According to the method of Example 1, it further includes determining that the area is outside the corresponding FOV of at least one of a person, recording device, and object in the physical environment by: detecting one or more objects in the physical environment; and determining that the corresponding FOV of at least one of the person, the recording device, and the object is occluded by the one or more objects.

[0187] Example 3: The method according to any one of Examples 1 or 2, wherein detecting the private region includes: calculating face location data associated with a first face detected in an image capturing the mapped scene, wherein the first face corresponds to the person; calculating the person's field of view (FOV) based on the face location data; and determining, based on the person's FOV and the location of the private region, that the private region is occluded outside the person's field of view by one or more objects located between the private region and the person.

[0188] Example 4: The method according to any one of Examples 1 to 3, wherein detecting the private region further includes: calculating face location data associated with a second face detected in an image capturing the mapped scene, wherein the second face corresponds to the user; calculating the user's FOV based on the face location data associated with the second face; and determining that the private region is within the user's FOV based on the location of the private region.

[0189] Example 5: A method according to any one of Examples 1 to 4, wherein mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region comprises: determining at least one of a first size of the private region and a first orientation of the private region; determining at least one of a second size of the virtual private control interface and a second orientation of the virtual private control interface based on the first size of the private region and the first orientation of the private region, wherein the second size of the virtual private control interface matches or fits the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region; generating the virtual private control interface according to the second size and the second orientation; and aligning the virtual private control interface with the private region.

[0190] Example 6: According to any one of Examples 1 to 5, the object in the physical environment that enables viewing access from outside the physical environment includes at least one of windows, glass doors, and open doors.

[0191] Example 7: The method according to any one of Examples 1 to 6, wherein the private area includes the surface of a body part associated with the user, and wherein detecting the private area includes determining that the surface of the body part faces the user and is away from at least one of a person in the physical environment, a recording device in the physical environment, and an object in the physical environment.

[0192] Example 8: According to the method of Example 7, wherein the body part includes a hand, and the surface of the body part includes the palm of the hand, wherein detecting the private region includes: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determining that the biometric information matches the previously registered biometric information.

[0193] Example 9: The method according to any one of Examples 1 to 8 further includes: identifying the palm of a hand used to render the second virtual private control interface; and prior to rendering the second virtual private control interface on the palm of the hand: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determining whether the biometric information matches the previously registered biometric information.

[0194] Example 10: The method according to Example 9 further includes: determining that the biometric information does not match the previously registered biometric information; and in response to determining that the biometric information does not match the previously registered biometric information, determining that the second virtual private control interface is not rendered on the palm of the hand.

[0195] Example 11: The method according to any one of Examples 1 to 10, wherein detecting a private area includes: detecting one or more objects in the physical environment; detecting one or more obstructions in the physical environment based on the one or more objects in the physical environment; and detecting the private area by determining that the one or more obstructions block the visibility of the person, the recording device, or the object to the private area.

[0196] Example 12: The method according to Example 11, wherein detecting one or more obstructions in the physical environment includes tracing the path of light within the physical environment.

[0197] Example 13: The method according to any one of Examples 1 to 12, wherein the recording device includes a camera, and wherein the virtual private control interface includes an augmented reality interface.

[0198] Example 14: An apparatus comprising: at least one memory; and one or more processors implemented in a circuit and configured to: determine the pose of the apparatus within a mapped scene of a physical environment associated with the apparatus; detect a private region in the physical environment and the position of the private region relative to the pose of the apparatus, the private region including an area estimated to be within the field of view (FOV) of a user of the apparatus and outside the corresponding FOV of at least one of a person in the physical environment, a recording device in the physical environment, and an object in the physical environment accessible from outside the physical environment; map a virtual private control interface to the private region based on the pose of the apparatus and the detected private region, the virtual private control interface including one or more input interface elements; and render the virtual private control interface within the private region of the mapped scene.

[0199] Example 15: According to the apparatus of Example 14, the one or more processors are configured to determine that the area is outside the respective FOV of at least one of a person, recording device, and object in the physical environment by: detecting one or more objects in the physical environment; and determining that the respective FOV of at least one of the person, the recording device, and the object is occluded by the one or more objects.

[0200] Example 16: An apparatus according to any one of Examples 14 or 15, wherein detecting the private region comprises: calculating face location data associated with a first face detected in an image of the captured mapped scene, wherein the first face corresponds to the person; calculating the person's field of view (FOV) based on the face location data; and determining, based on the person's FOV and the location of the private region, that the private region is occluded outside the person's field of view by one or more objects located between the private region and the person.

[0201] Example 17: The apparatus according to any one of Examples 14 to 16, wherein detecting a private region further comprises: calculating face location data associated with a second face detected in an image of a captured mapping scene, wherein the second face corresponds to the user; calculating the user's FOV based on the face location data associated with the second face; and determining that the private region is within the user's FOV based on the location of the private region.

[0202] Example 18: An apparatus according to any one of Examples 14 to 17, wherein mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region comprises: determining at least one of a first size of the private region and a first orientation of the private region; determining at least one of a second size of the virtual private control interface and a second orientation of the virtual private control interface based on the first size of the private region and the first orientation of the private region, wherein the second size of the virtual private control interface matches or adapts to the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region; generating the virtual private control interface according to at least one of the second size and the second orientation; and aligning the virtual private control interface with the private region.

[0203] Example 19: The apparatus according to any one of Examples 14 to 18, wherein the object in the physical environment that enables viewing access from outside the physical environment includes at least one of a window, a glass door, and an open door.

[0204] Example 20: An apparatus according to any one of Examples 14 to 19, wherein the private area includes a surface of a body part associated with the user, and wherein detecting the private area includes determining that the surface of the body part faces the user and is opposite to at least one of a person in the physical environment, a recording device in the physical environment, or an object in the physical environment.

[0205] Example 21: The apparatus according to Example 20, wherein the body part includes a hand and the surface of the body part includes the palm of the hand, wherein detecting the private area includes: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determining that the biometric information matches the previously registered biometric information.

[0206] Example 22: In the apparatus of any one of Examples 14 to 21, the one or more processors are configured to: identify the palm of a hand for rendering a second virtual private control interface; and before rendering the second virtual private control interface on the palm of the hand: detect biometric information associated with the palm of the hand based on one or more images of the palm of the hand; compare the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determine whether the biometric information matches the previously registered biometric information.

[0207] Example 23: According to the apparatus of Example 22, the one or more processors are configured to: determine that the biometric information does not match the previously registered biometric information; and in response to determining that the biometric information does not match the previously registered biometric information, determine not to render the second virtual private control interface on the palm of the hand.

[0208] Example 24: An apparatus according to any one of Examples 14 to 23, wherein detecting a private area comprises: detecting one or more objects in the physical environment; detecting one or more obstructions in the physical environment based on the one or more objects in the physical environment; and detecting the private area by determining that the one or more obstructions block the visibility of the person, the recording device, or the object to the private area.

[0209] Example 25: The apparatus according to Example 24, wherein detecting the one or more obstructions in the physical environment includes tracing the path of light within the physical environment.

[0210] Example 26: An apparatus according to any one of Examples 14 to 25, wherein the recording device includes a camera, and wherein the virtual private control interface includes an augmented reality interface.

[0211] Example 27: A device according to any one of Examples 14 to 26, wherein the device is a mobile device.

[0212] Example 28: An apparatus according to any one of Examples 14 to 27, wherein the apparatus includes an extended reality device.

[0213] Example 29: At least one non-transitory computer-readable storage medium having stored instructions that, when executed by one or more processors, cause the one or more processors to: determine the pose of an extended reality device within a mapped scene of a physical environment associated with the extended reality device; detect a private region in the physical environment and the position of the private region relative to the pose of the extended reality device, the private region including an area outside the corresponding FOV of at least one of a person, a recording device in the physical environment, and an object in the physical environment that can be viewed and accessed from outside the physical environment, estimated to be within the field of view (FOV) of a user of the extended reality device; map a virtual private control interface to the private region based on the pose of the extended reality device and the detected private region, the virtual private control interface including one or more input interface elements; and render the virtual private control interface within the private region of the mapped scene.

[0214] Example 30: At least one non-transitory computer-readable storage medium according to Example 29, comprising determining that the area is outside the corresponding FOV of at least one of the person, the recording device, and the object in a physical environment by: detecting one or more objects in the physical environment; and determining that the corresponding FOV of at least one of the person, the recording device, and the object is occluded by the one or more objects.

[0215] Example 31: At least one non-transitory computer-readable storage medium according to any one of Examples 29 or 30, wherein detecting a private region comprises: calculating face location data associated with a first face detected in an image capturing the mapped scene, wherein the first face corresponds to the person; calculating the person's field of view (FOV) based on the face location data; and determining, based on the person's FOV and the location of the private region, that the private region is occluded outside the person's field of view by one or more objects located between the private region and the person.

[0216] Example 32: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 31, wherein detecting a private region further comprises: calculating face location data associated with a second face detected in an image capturing the mapped scene, wherein the second face corresponds to the user; calculating the user's FOV based on the face location data associated with the second face; and determining that the private region is within the user's FOV based on the location of the private region.

[0217] Example 33: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 32, wherein mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region comprises: determining at least one of a first size of the private region and a first orientation of the private region; determining at least one of a second size of the virtual private control interface and a second orientation of the virtual private control interface based on the first size of the private region and the first orientation of the private region, wherein the second size of the virtual private control interface matches or fits the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region; generating the virtual private control interface according to the second size and the second orientation; and aligning the virtual private control interface with the private region.

[0218] Example 34: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 33, wherein the object in the physical environment that enables viewing access from outside the physical environment includes at least one of windows, glass doors, and open doors.

[0219] Example 35: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 34, wherein the private area includes a surface of a body part associated with the user, and wherein detecting the private area includes determining that the surface of the body part faces the user and is facing away from at least one of a person in the physical environment, a recording device in the physical environment, or an object in the physical environment.

[0220] Example 36: At least one non-transitory computer-readable storage medium according to Example 35, wherein the body part includes a hand, and the surface of the body part includes the palm of the hand, wherein detecting the private region includes: detecting biometric information associated with the palm of the hand based on one or more images of the palm of the hand; comparing the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determining that the biometric information matches the previously registered biometric information.

[0221] Example 37: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 36, further comprising, when executed by one or more processors, instructions to cause the one or more processors to perform the following operations: identify the palm of a hand for rendering a second virtual private control interface; and prior to rendering the second virtual private control interface on the palm of the hand: detect biometric information associated with the palm of the hand based on one or more images of the palm of the hand; compare the biometric information with previously registered biometric information associated with a previously registered palm of the hand associated with the user; and determine whether the biometric information matches the previously registered biometric information.

[0222] Example 38: The at least one non-transitory computer-readable storage medium according to Example 37 further includes instructions, when executed by one or more processors, to cause the one or more processors to perform the following operations: determine that the biometric information does not match the previously registered biometric information; and in response to determining that the biometric information does not match the previously registered biometric information, determine that the second virtual private control interface is not rendered on the palm of the hand.

[0223] Example 39: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 38, wherein detecting a private area comprises: detecting one or more objects in the physical environment; detecting one or more obstructions in the physical environment based on the one or more objects in the physical environment; and detecting the private area by determining that the one or more obstructions block the visibility of the person, the recording device, or the object to the private area.

[0224] Example 40: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 39, wherein detecting one or more obstructions in the physical environment includes tracing light paths within the physical environment.

[0225] Example 41: At least one non-transitory computer-readable storage medium according to any one of Examples 29 to 40, wherein the recording device includes a camera, and wherein the virtual private control interface includes an augmented reality interface.

[0226] Example 42: An apparatus comprising a unit for performing a method according to any one of Examples 1 to 13.

Claims

1. A method capable of operating on an extended reality device, the method comprising: Determine the pose of the extended reality device within a mapped scene of the physical environment associated with the extended reality device; Detecting private regions in the mapped scene and the position of the private regions relative to the pose of the extended reality device, the private regions including regions estimated to be within the field of view (FOV) of the user of the extended reality device and outside the corresponding FOV of at least one of the following: people in the physical environment, recording devices in the physical environment, and objects in the physical environment that can be viewed and accessed from outside the physical environment; Based on the posture of the extended reality device and the detected private region, the virtual private control interface is mapped to the private region; as well as The virtual private control interface is rendered within the private region of the mapped scene.

2. The method of claim 1, further comprising determining that the area is outside the respective FOV of at least one of the person in the physical environment, the recording device in the physical environment, and the object in the physical environment that enables viewing access from outside the physical environment: Detect one or more objects in the physical environment; and It is determined that the FOV of at least one of the following in the physical environment—the person, the recording device, and the object in the physical environment that enables viewing access from outside the physical environment—is obscured by the one or more objects.

3. The method as described in claim 1, wherein, Detecting the private area includes: Calculate face location data associated with a first face detected in an image capturing the mapped scene, wherein the first face corresponds to the person; Based on the facial location data, calculate the person's field of view (FOV); and Based on the person's field of view (FOV) and the location of the private area, it is determined that the private area is obstructed relative to the person's line of sight by one or more objects located between the private area and the person.

4. The method of claim 3, wherein, Detecting the private area also includes: Calculate face location data associated with a second face detected in an image capturing the mapped scene, wherein the second face corresponds to the user; Based on the face location data associated with the second face, the user's field of view (FOV) is calculated; and The location of the private area determines whether the private area is within the user's field of view (FOV).

5. The method of claim 1, wherein, Mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region includes: Determine at least one of a first size of the private region and a first orientation of the private region; A second size of the virtual private control interface and a second orientation of the virtual private control interface are determined based on at least one of the first size and the first orientation of the private region, wherein the second size of the virtual private control interface matches or is adapted to the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region. The virtual private control interface is generated according to at least one of the second size and the second orientation; and Align the virtual private control interface with the private region.

6. The method of claim 1, wherein, The objects in the physical environment that enable viewing from outside the physical environment include at least one of windows, glass doors, and open doors.

7. The method of claim 1, wherein, The private area includes the surface of a body part associated with the user, and wherein detecting the private area includes determining that the surface of the body part faces the user and is away from at least one of the following: a person in the physical environment, a recording device in the physical environment, and an object in the physical environment that enables viewing access.

8. The method of claim 7, wherein, The body part includes a hand, and the surface of the body part includes the palm of the hand, wherein detecting the private area includes: Detect biometric information associated with the palm of the hand based on one or more images of the palm of the hand; The biometric information is compared with previously registered biometric information, which is associated with a previously registered palm print associated with the user; and The biometric information is determined to match the previously registered biometric information.

9. The method of claim 1, further comprising: Identify the palm of the hand used to render the second virtual private control interface; as well as Before rendering the second virtual private control interface on the palm of the hand: Detect biometric information associated with the palm of the hand based on one or more images of the palm of the hand; The biometric information is compared with previously registered biometric information, which is associated with a previously registered palm print associated with the user; and Determine whether the biometric information matches the previously registered biometric information.

10. The method of claim 9, further comprising: Determine that the biometric information does not match the previously registered biometric information; as well as In response to determining that the biometric information does not match the previously registered biometric information, it is determined not to render the second virtual private control interface on the palm of the hand.

11. The method of claim 1, wherein, Detecting the private area includes: Detect one or more objects in the mapped scene; Detect one or more occlusions in the mapped scene based on the one or more objects in the mapped scene; and The private area is detected by determining whether the one or more obstructions block the visibility of the person, the recording device, or the object in the physical environment that enables viewing access from outside the physical environment to the private area.

12. The method of claim 11, wherein, Detecting the one or more obstructions includes tracing the path of light within the physical environment.

13. The method of claim 1, wherein, The recording device includes a camera, and the virtual private control interface includes an augmented reality interface.

14. An apparatus comprising: At least one memory; as well as One or more processors, which are implemented in a circuit and configured to: Determine the attitude of the device within a mapped scene of the physical environment associated with the device; Detecting private regions in the mapped scene and the position of the private regions relative to the orientation of the device, the private regions including areas estimated to be within the field of view (FOV) of the user of the device and outside the corresponding FOV of at least one of the following: people in the physical environment, recording devices in the physical environment, and objects in the physical environment that enable viewing and access from outside the physical environment; Based on the attitude of the device and the detected private region, a virtual private control interface is mapped to the private region, and the virtual private control interface includes one or more input interface elements; as well as The virtual private control interface is rendered within the private region of the mapped scene.

15. The apparatus of claim 14, wherein the one or more processors are configured to determine that the area is outside the respective field of view (FOV) of at least one of the person in the physical environment, the recording device in the physical environment, and the object in the physical environment that enables viewing access from outside the physical environment: Detect one or more objects in the physical environment; and It is determined that the FOV of at least one of the person in the physical environment, the recording device in the physical environment, and the object in the physical environment that enables viewing access from outside the physical environment is obscured by the one or more objects.

16. The apparatus of claim 14, wherein, Detecting the private area includes: Calculate and capture face location data associated with a first face detected in an image of the mapped scene, wherein the first face corresponds to the person; Calculate the person's field of view (FOV) based on the facial location data; and Based on the person's field of view (FOV) and the location of the private area, it is determined that the private area is obstructed relative to the person's line of sight by one or more objects located between the private area and the person.

17. The apparatus of claim 16, wherein, Detecting the private area also includes: Calculate face location data associated with a second face detected in an image capturing the mapped scene, wherein the second face corresponds to the user; The user's field of view (FOV) is calculated based on the face location data associated with the second face; and The location of the private area determines whether the private area is within the user's field of view (FOV).

18. The apparatus of claim 14, wherein, Mapping the virtual private control interface to the private region and rendering the virtual private control interface within the private region includes: Determine at least one of a first size of the private region and a first orientation of the private region; A second size of the virtual private control interface and a second orientation of the virtual private control interface are determined based on at least one of the first size and the first orientation of the private region, wherein the second size of the virtual private control interface matches or is adapted to the first size of the private region, and wherein the second orientation of the virtual private control interface is at least partially aligned with the first orientation of the private region. The virtual private control interface is generated according to at least one of the second size and the second orientation; and Align the virtual private control interface with the private region.

19. The apparatus of claim 14, wherein, The objects in the physical environment that enable viewing from outside the physical environment include at least one of windows, glass doors, and open doors.

20. The apparatus of claim 14, wherein, The private area includes the surface of a body part associated with the user, and wherein detecting the private area includes determining that the surface of the body part is facing the user and is facing away from at least one of the following: a person in the physical environment, a recording device in the physical environment, and an object in the physical environment that enables the viewing access.

21. The apparatus of claim 20, wherein, The body part includes a hand, and the surface of the body part includes the palm of the hand, wherein detecting the private area includes: Detect biometric information associated with the palm of the hand based on one or more images of the palm of the hand; The biometric information is compared with previously registered biometric information, which is associated with a previously registered palm print associated with the user; and The biometric information is determined to match the previously registered biometric information.

22. The apparatus of claim 14, wherein the one or more processors are configured to: Identify the palm of the hand used to render the second virtual private control interface; and Before rendering the second virtual private control interface on the palm of the hand: Based on one or more images of the palm of the hand, detect biometric information associated with the palm of the hand; The biometric information is compared with previously registered biometric information, which is associated with a previously registered palm print associated with the user; and Determine whether the biometric information matches the previously registered biometric information.

23. The apparatus of claim 22, wherein the one or more processors are configured to: Determining that the biometric information does not match the previously registered biometric information; and In response to determining that the biometric information does not match the previously registered biometric information, it is determined not to render the second virtual private control interface on the palm of the hand.

24. The apparatus of claim 14, wherein, Detecting the private area includes: Detect one or more objects in the physical environment; Based on the one or more objects in the physical environment, detect one or more occlusions in the physical environment; and The private area is detected by determining whether the one or more obstructions block the visibility of the person, the recording device, or the object in the physical environment that enables viewing access from outside the physical environment to the private area.

25. The apparatus of claim 24, wherein, Detecting one or more obstructions in the physical environment includes tracing the path of light within the physical environment.

26. The apparatus of claim 14, wherein, The recording device includes a camera, and the virtual private control interface includes an augmented reality interface.

27. The apparatus of claim 14, wherein, The device is a mobile device.

28. The apparatus of claim 14, wherein, The device includes an augmented reality device.

29. At least one non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to: Determine the pose of the extended reality device within a mapped scene of the physical environment associated with the extended reality device; Detecting private regions in the mapped scene and the position of the private regions relative to the pose of the extended reality device, the private regions including regions estimated to be within the field of view (FOV) of the user of the extended reality device and outside the corresponding FOV of at least one of the following: people in the physical environment, recording devices in the physical environment, and objects in the physical environment that can be viewed and accessed from outside the physical environment; Based on the pose of the extended reality device and the detected private region, a virtual private control interface is mapped to the private region, the virtual private control interface including one or more input interface elements; and The virtual private control interface is rendered within the private region of the mapped scene.

30. The at least one non-transitory computer-readable storage medium as described in claim 29, wherein, The area is determined to be outside the respective field of view (FOV) of at least one of the following in the physical environment: the person in the physical environment, the recording device in the physical environment, and the object in the physical environment that allows viewing access from outside the physical environment: Detect one or more objects in the physical environment; and It is determined that the FOV of at least one of the following in the physical environment—the person, the recording device, and the object in the physical environment that enables viewing access from outside the physical environment—is obscured by the one or more objects.

Images