Systems and methods for user role management in applications with virtual content

Abstract

User role management systems and techniques are described. A system identifies a profile associated with a first user. The profile includes data defining each avatar representing the first user and conditions for displaying the respective avatar. The system determines that at least a first condition is satisfied based on a feature associated with the first user. The system selects a displayed avatar of the avatars based on a determination that at least the first condition is satisfied. The system outputs the display avatar for presentation to the second user, for example by displaying the display avatar on the display and / or by sending the display avatar to a user device associated with the second user. The display avatar may be presented according to features associated with the first user.

Description

Technical Field

[0001] The present application is related to user persona management. More specifically, the present application relates to systems and methods for user persona management in applications with virtual content (such as extended reality environments or metaverse environments) to select different avatars to display based on different characteristics and situations. Background Art

[0002] A network-based interactive system allows users to interact with each other over a network, in some cases even if those users are geographically far away from each other. A network-based interactive system may include video conferencing technology. In a video conference, each user is connected via a user device that captures the user's video and / or audio and sends the video and / or audio to other users in the video conference so that each user in the video conference can see and hear each other. A network-based interactive system may include a network-based multiplayer game, such as a massively multiplayer online (MMO) game. A network-based interactive system may include extended reality (XR) technology, such as virtual reality (VR) or augmented reality (AR). At least a portion of the XR environment displayed to a user of the XR device may be virtual, including in some examples representations of other users with whom the user can interact in the XR environment.

[0003] In some examples, the network-based interactive system may use a camera to obtain image data of a user and / or a portion of a real-world environment that the user is in. In some examples, the network-based interactive system sends the image data to other users. However, sending image data that may include representations of the user and / or other people in the environment may raise privacy issues because some of these people may not want their images to be captured and / or shared publicly and / or shared with certain users. Summary of the invention

[0004] In some examples, systems and techniques for user role management are described. User role management systems and techniques are described. The user role management system identifies a profile associated with a first user, for example, based on identifying the first user in an image. The profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar from the plurality of avatars. The user role management system determines that at least a first condition of the one or more conditions is satisfied based on one or more features associated with the first user. The user role management system selects a display avatar from the plurality of avatars based on determining that at least the first condition is satisfied. For example, by modifying an image to use a display avatar in place of the first user, the user role management system outputs a display avatar for presentation to a second user. The display avatar will be presented based on one or more features associated with the first user.

[0005] In one example, a device for media processing is provided. The device includes a memory and one or more processors coupled to the memory (e.g., implemented in a circuit). The one or more processors are configured and capable of: identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar in the plurality of avatars; selecting a display avatar from the plurality of avatars based on a first condition in the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and outputting the display avatar for presentation to a second user, wherein the display avatar is presented according to the one or more features associated with the first user.

[0006] In another example, a method of image processing is provided. The method includes: identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar in the plurality of avatars; selecting a display avatar in the plurality of avatars based on a first condition in the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and outputting the display avatar for presentation to a second user, wherein the display avatar is presented according to the one or more features associated with the first user.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: identify a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar from the plurality of avatars; select a display avatar from the plurality of avatars based on a first condition from among the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and output the display avatar for presentation to a second user, wherein the display avatar is to be presented based on the one or more features associated with the first user.

[0008] In another example, an apparatus for image processing is provided. The apparatus includes: a component for identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar from the plurality of avatars; a component for selecting a display avatar from the plurality of avatars based on a first condition from the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and a component for outputting the display avatar for presentation to a second user, wherein the display avatar is presented based on the one or more features associated with the first user.

[0009] In some aspects, the one or more features associated with the first user include an identity of a second user to whom the display avatar is to be presented, wherein the first condition corresponds to whether the identity of the second user is identified in a predetermined data structure corresponding to the first user. In some aspects, the one or more features associated with the first user include a relationship category between the second user to whom the display avatar is to be presented and the first user, wherein the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user.

[0010] In some aspects, the one or more characteristics associated with the first user include a location of the first user in the environment, wherein the first condition corresponds to whether the location of the first user in the environment falls within at least one of a predetermined area, a predetermined area type, or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user. In some aspects, the one or more characteristics associated with the first user include an activity performed by the first user, wherein the first condition corresponds to whether the activity is identified in the predetermined data structure corresponding to the first user.

[0011] In some aspects, identifying the profile associated with the first user includes identifying the profile associated with the first user based on one or more communications from a user device associated with the first user.

[0012] In some aspects, one or more of the above-mentioned methods, devices, and computer-readable media also include: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying corresponding avatars in the second plurality of avatars; selecting a second display avatar from the second plurality of avatars based on a second condition in the second set of one or more conditions, wherein the second condition is associated with a second set of one or more characteristics associated with the second user; and outputting a second display avatar for presentation to the second user, wherein the second display avatar will be presented based on the second set of one or more characteristics associated with the third user.

[0013] In some aspects, one or more of the above-mentioned methods, devices, and computer-readable media also include: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying corresponding avatars in the second plurality of avatars; selecting a second display avatar from the second plurality of avatars based on none of the conditions in the second set of one or more conditions being satisfied; and outputting the second display avatar for presentation to the second user.

[0014] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: receiving an image of an environment; detecting at least a portion of a first user in the image, wherein identifying a profile associated with the first user is based on at least a portion of the first user detected in the image; and generating a modified image at least in part by modifying the image according to one or more characteristics of the first user to replace at least a portion of the first user in the image with a displayed avatar, wherein outputting the displayed avatar for presentation to a second user includes outputting the modified image for presentation to the second user. In some aspects, at least a portion of the environment in the image includes one or more virtual elements. In some aspects, at least a portion of the image of the environment is captured by an image sensor. In some aspects, generating the modified image includes using the displayed avatar and at least a portion of the first user in the image as inputs to a trained machine learning model that modifies the image according to one or more characteristics of the first user to replace at least a portion of the first user with a displayed avatar.

[0015] In some aspects, the portion of the first user includes one or more facial features of the first user. In some aspects, identifying the profile associated with the first user includes using facial recognition to identify the identity of the one or more facial features of the first user in the image.

[0016] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: modifying the display avatar based on one or more display preferences associated with the second user before outputting the display avatar for presentation to the second user.

[0017] In some aspects, one or more of the above-mentioned methods, devices and computer-readable media also include: selecting a second display avatar from multiple avatars based on a change in one or more conditions from a first condition to a second condition; and transitioning from outputting the display avatar for presentation to the second user to outputting a second display avatar for presentation to the second user, wherein the second display avatar is presented based on one or more characteristics associated with the first user.

[0018] In some aspects, one or more of the above methods, devices, and computer-readable media also include: generating a display avatar before outputting the display avatar for presentation to a second user, wherein generating the display avatar includes providing one or more inputs associated with the first user to a trained machine learning model, and the trained machine learning model generates the display avatar based on the one or more inputs associated with the first user.

[0019] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: recognizing a facial expression of the first user; and modifying the displayed avatar to apply the facial expression to the displayed avatar before outputting the displayed avatar for presentation to the second user.

[0020] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: identifying a head gesture of the first user; and modifying the displayed avatar to apply the head gesture to the displayed avatar before outputting the displayed avatar for presentation to the second user.

[0021] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include: identifying a lighting condition for the first user; and modifying the display avatar to apply the lighting condition to the display avatar before outputting the display avatar for presentation to the second user.

[0022] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: causing the display avatar to be displayed using a display. In some aspects, one or more apparatuses include a display. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: causing the display avatar to be sent to at least one user device associated with the second user using at least one communication interface. In some examples, one or more apparatuses include a communication interface.

[0023] In some examples, the one or more devices include at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device. In some aspects, one or more of the above methods, devices, and computer readable media also include: outputting the display avatar for presentation to the second user at least in part by sending the display avatar to a user device associated with the second user. In some aspects, the one or more devices include one or more network servers.

[0024] In some aspects, the device is a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a part of a combination thereof and / or includes the above devices. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0025] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings and each claim.

[0026] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0028] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0029] Figure 2A is a block diagram illustrating an example architecture of an imaging process performed by an imaging system having one or more servers and two user devices according to some examples;

[0030] 2B is a block diagram illustrating an example architecture of an imaging process performed by an imaging system having two user devices according to some examples;

[0031] Figure 3A is a perspective view illustrating a head mounted display (HMD) used as part of an imaging system according to some examples;

[0032] Figure 3B is an illustration of a device worn by a user according to some examples Figure 3A A perspective view of a head mounted display (HMD);

[0033] Figure 4A is a perspective view illustrating a front surface of a mobile phone that includes a front-facing camera and can be used as part of an imaging system according to some examples;

[0034] Figure 4B is a perspective view illustrating a rear surface of a mobile phone that includes a rear camera and can be used as part of an imaging system according to some examples;

[0035] Figure 5A is a conceptual diagram illustrating generating a modified image from an image by replacing a representation of the user in the image with a first avatar of the user according to some examples;

[0036] Figure 5B is an illustration of how to replace a representation of a user in an image with a second avatar of the user according to some examples. Figure 5A Generate a concept map of the modified image from the image;

[0037] Figure 5C is an illustration of how to replace a representation of a user in an image by using a third avatar of the user according to some examples. Figure 5A Generate a concept map of the modified image from the image;

[0038] Fig. 6A is a conceptual diagram illustrating a second user viewing an interactive environment using a head mounted device according to some examples, the interactive environment including a first user represented by a first avatar displayed to the second user through the head mounted device based on one or more conditions;

[0039] Figure 6B is a conceptual diagram illustrating a second user viewing an interactive environment using a head mounted device according to some examples, the interactive environment including a first user represented by a second avatar displayed to the second user through the head mounted device based on one or more conditions;

[0040] Figure 7 is a conceptual diagram illustrating combining identity input and expression input according to some examples to generate a combined face having an identity of the identity input and an expression of the expression input;

[0041] Figure 8 is a block diagram illustrating an example of a neural network that may be used for image processing operations according to some examples;

[0042] Fig. 9 is a flow chart illustrating an imaging process according to some examples; and

[0043] Fig.10 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0044] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, as will be apparent to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. The drawings and description are not intended to be limiting.

[0045] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the example aspects will provide an enabling description of implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the application as described in the appended claims.

[0046] A camera is a device that receives light and captures image frames (such as still images or video frames) using an image sensor. The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings will produce images with different appearances. Before or during the capture of one or more image frames, some camera settings, such as ISO, exposure time, aperture size, f / stop, shutter speed, focal length, and gain, are determined and applied. For example, settings or parameters can be applied to an image sensor to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changing contrast, brightness, saturation, sharpness, color scale, curves, or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by an image sensor.

[0047] An extended reality (XR) system or device may provide a user with virtual content and / or a real-world view that may combine a physical environment (scene) and a virtual environment (including virtual content). The XR system facilitates user interaction with such a combined XR environment. The real-world view may include real-world objects (also referred to as physical objects) such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. The XR system or device may facilitate interaction with different types of XR environments (e.g., a user may interact with an XR environment using an XR system or device). The XR system may include a virtual reality (VR) system that facilitates interaction with a VR environment, an augmented reality (AR) system that facilitates interaction with an AR environment, a mixed reality (MR) system that facilitates interaction with an MR environment, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, the XR system may track portions of a user (e.g., a user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0048] Video conferencing is a network-based technology that allows multiple users, who may each be located in different locations, to connect to a video conference through a network using their respective user devices, each of which typically includes a display and a camera. In a video conference, each camera of each user device captures image data representing the user using the user device, and sends the image data to other user devices connected to the video conference to be displayed on the displays of other users using those other user devices. At the same time, the user device displays image data representing other users in the video conference, which are captured by the corresponding cameras of the other user devices connected to the video conference by those other users. When users are in different locations, a user group can use video conferencing to have a virtual face-to-face conversation. Despite travel restrictions, such as those related to the epidemic, video conferencing can be a valuable way for users to meet virtually. Video conferencing can be performed using interconnected user devices, in some cases performed by one or more servers. In some examples, the user device may include a laptop, phone, tablet, mobile phone, video game console, car computer, desktop computer, wearable device, TV, media center, XR system, or other computing device discussed herein.

[0049] A network-based interactive system allows users to interact with each other over a network, in some cases even if those users are geographically far away from each other. A network-based interactive system may include video conferencing technologies such as those described above. A network-based interactive system may include extended reality (XR) technologies such as those described above. At least a portion of the XR environment displayed to a user of the XR device may be virtual, including in some examples representations of other users with whom the user can interact in the XR environment. A network-based interactive system may include a network-based multiplayer game, such as a massively multiplayer online (MMO) game. A network-based interactive system may include a network-based interactive environment, such as a "metaphysical reality" environment.

[0050] In some examples, the network-based interaction system may use sensors to capture sensor data, and obtain (multiple) representations of the user and / or the portion of the real-world environment in which the user is located in the sensor data. For example, the network-based interaction system may use a camera (e.g., an image sensor of a camera) to capture image data, and obtain (multiple) descriptions of the user and / or the portion of the real-world environment in which the user is located in the image data. In some examples, the network-based interaction system sends the sensor data (e.g., image data) to other users. However, sending sensor data (e.g., image data) that may include representations of the user and / or other people in the environment may cause privacy issues, because some of these people may not want their (multiple) representations (e.g., representations of their faces and / or (multiple) other parts of their respective bodies) to be captured and / or shared (e.g., publicly shared, shared with specific users, etc.) by the network-based interaction system. For example, the user may use the network-based interaction system outdoors, in a coffee shop, a store, at home, in the office, or at school. In each of these cases, the (multiple) sensors of the network-based interaction system may eventually capture sensor data with representations of the user and / or other people other than the user. For example, those other people may eventually walk into the field of view of the sensor(s), or the field of view of the sensor(s) may move (e.g., when the user moves their head while wearing an HMD as part of a network-based interactive system) to include other people. For some people, such as children, privacy laws in certain countries or regions may prohibit or otherwise regulate image capture and / or sharing using such network-based interactive systems. In some network-based interactive systems, such as those that include a head-mounted display (HMD) device, it may be difficult for a user to control the field of view of the camera(s) and / or other sensor(s) to prevent the capture of sensor data with representations of other people. In some cases, the user himself / herself may not wish to share his / her own image (or other representation) using the network-based interactive system, for example, if the user is not ready for the day, has a bad hair day, is not feeling well or unwell, is feeling tired or sleepy, is wearing clothing that they do not want other users to see them wearing (e.g., pajamas or gym clothes), is eating while using the network-based interactive system, or some combination thereof.

[0051] In some examples, the network-based interactive system may modify the image data to protect the privacy of the user and / or other persons depicted or otherwise represented in the image data. In some examples, the network-based interactive system may blur, pixelate, or cover faces of people in images to protect the privacy of the people. However, while this may protect privacy, this may undermine the immersion of users of the network-based interactive system and may prevent users from seeing the person's facial expressions and / or other expressiveness. In addition, blurring and pixelation can produce imperfect privacy and therefore pose security risks because image sharpening techniques can sometimes be used to reconstruct faces from blurred or pixelated images.

[0052] In some examples, systems and techniques for image processing are described. An imaging system identifies a profile associated with a first user. For example, the imaging system may identify a profile associated with the first user based on receiving an image of the environment and detecting at least a portion of the first user in the image (e.g., one or more facial features). The profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar in the plurality of avatars. The imaging system determines that at least a first condition of the one or more conditions is satisfied based on one or more features associated with the first user. The imaging system selects a display avatar from the plurality of avatars based on determining that at least the first condition is satisfied. The imaging system outputs the display avatar for presentation to a second user. The display avatar will be presented based on one or more features associated with the first user. For example, the imaging system may output the display avatar for presentation to the second user at least in part by modifying the image of the environment to replace at least a portion of the first user with the display avatar, and outputting the modified image for presentation to the second user.

[0053] In some examples, the first condition may be related to the identity of the second user and / or the type of relationship between the first user and the second user. For example, the first user may want to see friends and / or family in one way (e.g., their real appearance or a realistic avatar) and strangers in a different way. In some examples, the first condition may be related to the location of the first user and / or the second user in the environment (e.g., in the real world and / or in an at least partially virtual environment). For example, the first user may want to see one way in a "home" environment (e.g., their real appearance or a realistic avatar) and see a different way in other environments. In some examples, the first condition may be related to the activities performed by the first user and / or the second user in the environment (e.g., in the real world and / or in an at least partially virtual environment). For example, the first user may want to see one way (e.g., an avatar associated with an artist or sports team) when attending a concert or sports game, and see a different way (e.g., a more neutral avatar) in other environments.

[0054] In some examples, the profile of the first user may include a list of approval conditions (whitelist) under which a particular avatar will be used, for example, indicating certain approved identities of the second user, an approved relationship between the first user and the second user, (multiple) approved locations of the first user and / or the second user, approved activities of the first user and / or the second user, or a combination thereof. In some examples, the profile of the first user may include a list of blocking conditions (blacklist) under which a particular avatar will not be used, for example, indicating certain blocked identities of the second user, a blocked relationship between the first user and the second user, (multiple) blocked locations of the first user and / or the second user, blocked activities of the first user and / or the second user, or a combination thereof. In some examples, the profile of the first user may include a list of approval conditions (whitelist) and a list of blocking conditions (blacklist).

[0055] In some examples, the second user's profile may include further modifications to how the avatar is presented to the second user. For example, if the second user is color blind, the modifications may modify the avatar to avoid certain colors or color schemes, replace certain colors with alternative colors, etc. If the second user likes or dislikes a certain color, the modifications may modify the avatar to use or remove that color, respectively.

[0056] The imaging systems and techniques described herein provide many technical improvements to existing imaging systems. For example, the imaging systems and techniques described herein provide increased privacy and security for network-based interactive systems and / or other imaging systems, at least in part, through alternative representations of people represented in sensor data (e.g., avatars). Compared to other privacy enhancement techniques such as facial blurring, facial pixelation, and covering faces with black boxes, the imaging systems and techniques described herein provide enhanced immersion, and / or do not detract from immersion. The imaging systems and techniques described herein provide increased flexibility, customization, and personalization because users can indicate how they are presented to other users under various conditions. The imaging systems and techniques described herein provide increased flexibility and expressiveness because users can customize avatars for different conditions and / or convey expressions (e.g., facial expressions, gestures) through their avatars.

[0057] Various aspects of the application will be described with reference to the accompanying drawings. Figure 11 is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). Image capture and processing system 100 can capture independent images (or photos) and / or can capture a video including multiple images (or video frames) of a specific sequence. Lens 115 of system 100 faces scene 110 and receives light from scene 110. Lens 115 bends light toward image sensor 130. Light received by lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by image sensor 130. In some examples, scene 110 is a scene in an environment. In some examples, scene 110 is a scene of at least a portion of a user. For example, scene 110 can be a scene of one or both eyes of a user and / or at least a portion of a user's face.

[0058] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include multiple mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms in addition to those illustrated, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0059] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a storage register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may adjust the focus by moving the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo mechanism. In some cases, additional lenses may be included in the system 100, such as one or more microlenses on each photodiode of the image sensor 130, each microlens bending light received from the lens 115 toward the corresponding photodiode before the light reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0060] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a storage register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0061] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a storage register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of a lens element assembly (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servo mechanisms to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a varifocal zoom lens. In some examples, the lens assembly may include a focusing lens (which may be a lens 115 in some cases) that first receives light from the scene 110, and then the light passes through an afocal zoom system between the focusing lens (e.g., lens 115) and the image sensor 130 before the light reaches the image sensor 130. In some cases, the afocal zoom system can include two positive (e.g., converging, convex) lenses of equal or similar focal lengths (e.g., within a threshold difference) with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more lenses in the afocal zoom system, such as a negative lens and one or two positive lenses.

[0062] The image sensor 130 includes one or more photodiode arrays or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, so that light that matches the color of the color filter covering the photodiode can be measured. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of the image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald green") filters instead of or in addition to red, blue, and / or green filters. Some image sensors may not have color filters at all, but may use different photodiodes (in some cases vertically stacked) instead throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves and therefore respond to light of different wavelengths. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0063] In some cases, the image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching a specific photodiode or a portion of a specific photodiode at a specific time and / or from a specific angle, which can be used for phase detection autofocus (PDAF). The image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode, and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms 120 may alternatively or additionally be included in the image sensor 130. The image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0064] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1010 discussed with respect to the computing system 1000. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some embodiments, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM , Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0065] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging of image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1020, read-only memory (ROM) 145 and / or 1025, cache, storage unit, another storage device, or some combination thereof.

[0066] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a trackpad, a touch-sensitive surface, a printer, any other output device 1035, any other input device 1045, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B via a physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that enable a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that enable a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. Peripheral devices may include any of the aforementioned types of I / O devices 160 , and once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector, they themselves may be considered I / O devices 160 .

[0067] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B can be wirelessly coupled together, for example, via one or more wires, cables, or other electrical connectors and / or via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0068] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, representing an image capture device 105A and an image processing device 105B. The image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. The image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), a RAM 140, a ROM 145, and an I / O 160. In some cases, some components illustrated in the image capture device 105A, such as the ISP 154 and / or the host processor 152, may be included in the image capture device 105A.

[0069] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 1002.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some embodiments, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing device.

[0070] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 The components shown are more components. The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, the components of the image capture and processing system 100 may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0071] Figure 2A1 is a block diagram illustrating an example architecture of an imaging process performed by an imaging system 200A having one or more servers 205 and two user devices. Specifically, the imaging system 200A includes one or more servers 205, a user device 210 associated with a user 215, and a user device 220 associated with a user 225. Each of the (multiple) servers 205, the user devices 210, and / or the user devices 220 may include at least one computing system 1000. Each of the (multiple) servers 205, the user devices 210, and / or the user devices 220 may include, for example, one or more laptop computers, phones, tablet computers, mobile phones, video game consoles, in-vehicle computers, desktop computers, wearable devices, televisions, media centers, XR systems, head-mounted display (HMD) devices, other types of computing devices discussed herein, or combinations thereof. In some examples, the user device 210 includes (multiple) components illustrated and / or described herein as included in the user device 220. In some examples, the user device 210 may perform (multiple) operations illustrated and / or described herein performed by the user device 220. In some examples, user device 220 includes (multiple) components illustrated and / or described herein as included in user device 210. In some examples, user device 220 can perform (multiple) operations performed by user device 210 illustrated and / or described herein. In some examples, user device 210 and / or user device 220 include (multiple) components illustrated and / or described herein as included in (multiple) servers 205. In some examples, user device 210 and / or user device 220 can perform (multiple) operations performed by (multiple) servers 205 illustrated and / or described herein. In some examples, (multiple) servers 205 include (multiple) components illustrated and / or described herein as included in user device 210 and / or user device 220. In some examples, (multiple) servers 205 can perform (multiple) operations performed by user device 210 and / or user device 220 illustrated and / or described herein.

[0072] The imaging system 200A and the corresponding imaging process may be used for a network-based interactive system application, such as an application for video conferencing, extended reality (XR), video games, virtual reality environments, or a combination thereof. For example, the user device 210 includes an interactive client application 280A, the user device 220 includes an interactive client application 280B, and the server 205 includes an interactive server application 285. In some examples, the interactive client application 280A and the interactive client application 280B may be client instances of a software application for a network-based interactive system application, such as a software application for video conferencing, extended reality (XR), video games, virtual reality environments, or a combination thereof. In some examples, the interactive server application 285 may be a server instance of a software application for a network-based interactive system application, such as a software application for video conferencing, extended reality (XR), video games, virtual reality environments, or a combination thereof. In some examples, the interaction client application 280A, the interaction client application 280B, and / or the interaction server application 285 can generate a virtual environment, virtual elements to be incorporated into a real-world environment (e.g., as represented in image data captured using the sensor(s) 230), or a combination thereof. In some examples, the representation of the user 215 and / or the representation of the user 225 is located in an at least partially virtual environment and / or is able to move throughout the environment, where the virtual elements of the environment are generated using the interaction client application 280A, the interaction client application 280B, and / or the interaction server application 285. For example, the at least partially virtual environment can be a video game environment, a VR environment, an AR environment, an MR environment, an XR environment, a meta-reality environment, a video conferencing environment, a teleconferencing environment, or a combination thereof. Figure 2A , a graphic representation of interaction client application 280A and interaction client application 280B illustrates a user (e.g., user 215 or user 225) wearing an HMD device (e.g., user device 210 or user device 220) and seeing representations (e.g., virtual representations) of two people (e.g., two people depicted in sensor data 235 and / or display avatar data 265) in an XR environment (e.g., the XR environment may be at least partially virtual). The XR environment may be generated at least in part using interaction client application 280A, interaction client application 280B, and / or interaction server application 285.

[0073] The user device 210 of the user 215 includes one or more sensors 230. In some examples, the sensor(s) 230 include one or more image sensors or one or more cameras. The image sensor(s) capture image data, which may include one or more images, one or more videos, portions thereof, or a combination thereof. In some examples, at least one of the sensor(s) 230 may be oriented toward the user 215 (e.g., may be facing the user 215), and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the user 215. In some examples, at least one of the sensor(s) 230 may be away from the user 215 (e.g., may be facing away from the user 215) and / or toward the environment in which the user 215 is located, and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the environment. In some examples, sensor data captured by at least one sensor(s) 230 away from user 215 and / or toward user 215 may have a field of view (FoV) that includes, is included by, overlaps with, and / or otherwise corresponds to the FoV of the eyes of user 215. Figure 2A , a graphic representing sensor(s) 230 illustrates sensor(s) 230 as including a camera facing an environment in which two persons are present.

[0074] In some examples, sensor(s) 230 capture sensor data that measures and / or tracks information about various aspects of the body of user 215 and / or the behavior of user 215 (e.g., features of user 215). In some examples, sensor 230 includes one or more image sensors of one or more cameras facing at least a portion of the user (e.g., at least a portion of the face and / or head of user 215). The one or more cameras may include one or more image sensors that capture image data of one or more images including at least a portion of user 215. For example, sensor 230 may include one or more image sensors focused on one or both eyes (and / or eyelids) of user 215, wherein the image sensors of the cameras capture image data of one or both eyes of user 215. The one or more image sensors may also be referred to as eye capture sensors. In some embodiments, one or more image sensors may capture image data including a series of images over time, which in some examples may be arranged together in a time sequence, for example, arranged into a video. These image series may depict or otherwise indicate, for example, movement of the user's 215 eye(s), pupil dilation, blinking (using eyelids), squinting (using eyelids), glances, gazes, eye wetness levels, optokinetic reflexes or responses, vestibulo-ocular reflexes or responses, accommodation reflexes or responses, other properties related to the eyes and / or eyelids described herein, or combinations thereof.

[0075] The sensor(s) 230 may include one or more sensors that track information about the user 215 and / or the environment, including posture (e.g., position and / or orientation), the body of the user 215, and / or the behavior of the user 215. For example, the sensor(s) 230 may include one or more cameras, image sensors, microphones, heart rate monitors, oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, the one or more sensors 230 include at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or combinations thereof. In some examples, one or more sensors 230 include at least one input device 1045 of computing system 1000. In some implementations, one or more sensors 230 can supplement or refine sensor readings from other sensor(s) 230. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor can be used to identify the posture (e.g., position and / or orientation) of user device 210 and / or user 215 in an environment, and / or the gaze of user 215 through user device 210.

[0076] Sensor(s) 230 of user device 210 capture sensor data 235. In some examples, sensor data 235 includes information indicating the location of user 215 in the real world and / or in an at least partially virtual environment. In some examples, sensor data 235 includes information indicating a physical posture of user 215, e.g., indicating a gesture that user 215 is performing and / or a facial expression detected on the face of user 215. In some examples, sensor(s) 230 include image sensor(s), and sensor data 235 includes images captured by the image sensor(s) of the environment (and / or user 215). Figure 2A , a graphic representing sensor data 235 illustrates an image as depicting an environment in which two persons are present. In some examples, one of the two persons is user 215. In some examples, one of the two persons is user 225. User device 210 may send sensor data 235 to server(s) 205.

[0077] The server(s) 205 receive the sensor data 235. In some examples, a face detector 240 of the server(s) 205 detects a face of a person depicted in an image of the sensor data 235. The person is referred to herein as a first user, and in some cases may be user 215. The face detector 240 may detect, extract, recognize, and / or track features of the first user's part(s) (e.g., face, facial features, head, body), object(s), and / or environment in order to detect the first user's face. In some examples, the face detector 240 detects the first user's face by first detecting the first user's body and then detecting the face based on an expected location of the face within the body structure.

[0078] In some examples, the face detector 240 detects the face of the first user by inputting the sensor data 235 into one or more of the one or more trained machine learning (ML) models 295 discussed herein, and receiving an output indicating the position and / or orientation of the face. The (multiple) trained machine learning (ML) models 295 may be trained (e.g., by the user device 210, the user device 220, the server (s) 205, and / or another device) for use by the face detector 240 using training data including images, the images including faces whose positions and / or orientations are predetermined. In some examples, the face detector 240 detects the position of the face within the sensor data 235 (e.g., pixel coordinates), the position of the face within the environment (e.g., 3D coordinates within a 3D volume of the environment), the orientation of the face within the sensor data 235 (e.g., pitch, yaw, and / or roll) (e.g., along an axis visible in the sensor data 235), and / or the orientation of the face within the environment (e.g., pitch, yaw, and / or roll). For example, the pose (e.g., position and / or orientation) of a face in an environment can be based on a comparison of a distance (e.g., inter-eye distance) between two features on a face in the sensor data 235 with a reference distance (e.g., inter-eye distance) for an average person. In some examples, the face detector 240 detects the face of the first user using feature detection, feature extraction, feature recognition, feature tracking, object detection, object recognition, object tracking, face detection, face recognition, face tracking, first user detection, first user recognition, first user tracking, classification, or a combination thereof. In some examples, some sensors 230 face the (two) eyes of the user 215, and the face detector 240 can detect the face in the sensor data 235 based on gaze detection of the gaze of the (two) eyes of the user 215. In some examples, the user device 210 and / or the (multiple) servers 205 can receive one or more communications from a user device of the first user in the sensor data 235, and the face detector 240 can use the communication as an indicator that the face of the first user (or more generally, the first user) may be present in the sensor data 235. Figure 2A 2, a graphic representation of a face detector 240 illustrates sensor data 235 with a bounding box around the face of a first user depicted in sensor data 235, with a magnified version of the first user's face illustrated as extending from the box. The face appears to be that of a young man with short hair and a beard.

[0079] Server(s) 205 include profile identifier 245. In some examples, profile identifier 245 receives at least some sensor data 235 from user device 210. Profile identifier 245 retrieves a first user (e.g., user 215) whose face is detected by face detector 240 and / or a profile of user 215 from data store 290 based on received sensor data 235. In some examples, the first user is user 215, and the profile is a profile of user 215. If the first user does not already have a profile in data store 290, profile identifier 245 can create (e.g., generate) a profile for the first user. In some cases, profile identifier 245 and / or face detector 240 uses facial recognition on the first user's face and / or recognition of the first user on the first user's body to identify an identifier of the first user (e.g., name, email address, phone number, mailing address, number, code, etc.), and then queries data store 290 for the identifier to retrieve a profile of the first user from data store 290 (or to receive an indicator from data store 290 that a profile corresponding to the indicator does not yet exist).

[0080] In some examples, profile recognizer 245 and / or face detector 240 may use at least one of the trained ML models 295 to perform facial recognition on the face of the first user in sensor data 235 to determine an identifier corresponding to the first user (and / or the face of the first user) to query the data store 290 for the identifier. The trained machine learning (ML) models 295 may be trained (e.g., by user device 210, user device 220, server(s) 205, and / or another device) for use in facial recognition by face detector 240 and / or profile recognizer 245 using training data including images including faces for which identifiers of faces are predetermined. In some examples, user device 210 and / or server(s) 205 may receive one or more communications from a user device of the first user in sensor data 235, and profile recognizer 245 may identify an identifier (e.g., name, email address, phone number, mailing address, number, code, etc.) corresponding to the first user based on the one or more communications from the user device of the first user. In some examples, facial recognition may include comparing a face detected in an image by face detector 240 to a data structure of reference faces stored in data store 290, where an identifier is listed for each reference face in data store 290. Figure 2AWithin, a graphic illustration representing the profile identifier 245 illustrates a profile having a facial image of a first user detected by the facial detector 240 (e.g., a young man with short hair and a beard), the name of the same first user ("JohnSmith"), an email address of the same first user ("js@domain.com"), an avatar corresponding to the same first user, and privacy settings of the same first user.

[0081] The data store 290 may include one or more data structures, such as one or more databases, tables, lists, arrays, matrices, heaps, ledgers, distributed ledgers (e.g., blockchain ledgers and / or directed acyclic graph (DAG) ledgers), or combinations thereof. The data store 290 may store profiles of first users, such as profiles of users (e.g., user 215, user 225) detected in image(s) (e.g., image(s) of sensor data 235) captured by sensor(s) (e.g., sensor(s) 230) of a user device (e.g., user device 210, user device 220) . The profile of the first user may include one or more images of the first user (e.g., the real face of the first user), one or more identifiers of the first user (e.g., name, email address, phone number, mailing address, number, code, etc.), one or more avatars of the first user, one or more settings and / or preferences (e.g., conditions detected using condition detector 255) that identify when to use and / or not use different avatars, or combinations thereof. In representing Figure 2A An example of a profile for a first user "John Smith" is illustrated in the graphic of profile identifier 245 in FIG.

[0082] The server(s) 205 include a profile avatar recognizer 250 and a condition detector 255. Once the profile recognizer 245 identifies a profile of a first user whose face the face detector 240 detects in the sensor data 235, the profile avatar recognizer 250 identifies an avatar for the first user. The avatar for the first user may be selected by the first user and / or generated using an avatar generator (e.g., which may use the trained ML model(s) 295). A profile may be associated with one or more conditions and one or more avatars. The conditions may indicate a condition that the first user is represented using a first avatar of the profile, a condition that the first user is represented using a second avatar of the profile, etc. The conditions may indicate a condition that the first user is not represented using a first avatar of the profile, a condition that the first user is not represented using a second avatar of the profile, etc. The profile avatar recognizer 250 may identify which avatar the first user will be represented by based on which conditions are met and which conditions are not met (as determined by the condition detector 255). In some examples, profile avatar recognizer 250 may generate an avatar for the first user, for example, if the profile does not already include any avatar, or if the profile does not include an avatar suitable for representing the first user given which of the condition(s) are met and / or not met as determined by condition detector 255. In some examples, profile avatar recognizer 250 may generate an avatar for the first user in real time based on a request from condition detector 255, profile recognizer 245, data store 290, server(s) 205, user device 210, and / or user device 220. In some examples, profile avatar recognizer 250 may generate an avatar for the first user in advance and store the avatar in data store 290 as part of (or associated with) the profile of the first user. Profile avatar recognizer 250 may retrieve the avatar from data store 290 after identifying the profile using profile recognizer 245. Figure 2A , a graphic illustration representing the profile avatar recognizer 250 illustrates arrows pointing from a facial image of a first user (e.g., a young man named John Smith with short hair and a beard) detected by the facial detector 240 to two corresponding avatars of the first user (e.g., a human avatar and a dog avatar).

[0083] The condition detector 255 may identify which of the conditions associated with the profile (identified by the profile identifier 245) are satisfied and which of the conditions associated with the profile are not satisfied. The condition detector 255 may identify whether the condition is satisfied based on a feature associated with the first user (e.g., user 215). The feature associated with the first user (e.g., user 215) may include a feature associated with a viewer user who will be a viewer of the representation of the first user. For example, in some examples, the condition(s) and / or feature(s) may be associated with the identity of the viewer user who will be a viewer of the representation of the first user. The viewer user is referred to herein as the second user. The condition(s) and / or feature(s) may be associated with the identity of the second user and / or the type of relationship between the first user and the second user. For example, if the second user is a friend and / or family member of the first user (e.g., if the condition that the relationship is a friend or family relationship is satisfied), the first user may wish to be displayed in one way (e.g., a real appearance or a realistic avatar), and displayed in a different way to strangers (e.g., if the condition that the relationship is a friend or family relationship is not satisfied). In some examples, the condition(s) and / or feature(s) may be related to the location of the first user and / or the second user in an environment (e.g., in the real world and / or in an at least partially virtual environment). For example, the first user may wish to appear in one manner (e.g., their real appearance or a realistic avatar) when in a "home" environment (e.g., if the location is "home" condition is met), and appear in a different manner (e.g., a more anonymous avatar) in other environments (e.g., if the location is "home" condition is not met). In some examples, the condition(s) and / or feature(s) may be related to activities performed by the first user and / or the second user in an environment (e.g., in the real world and / or in an at least partially virtual environment). For example, a first user may wish to be presented in one way (e.g., with an avatar associated with an artist or sports team or movie star or movie character) when attending a concert or sporting game or movie premiere (e.g., if one of a set of conditions that the event is a concert or sporting game or movie premiere is satisfied), and to be presented in a different way (e.g., with a more neutral avatar) in other contexts (e.g., if none of the set of conditions that the event is a concert or sporting game or movie premiere is satisfied).

[0084] In some examples, the condition detector 255 may use at least one trained ML model(s) 295 to determine whether a particular condition is satisfied. For example, if the condition is based on whether the first user is performing a particular activity, the trained ML model(s) 295 used by the condition detector 255 may be trained (e.g., by the user device 210, the user device 220, the server(s) 205, and / or another system) to determine whether the first user is performing an activity corresponding to the condition based at least on the sensor data 235. For example, if the activity is dancing, the condition detector 255 may input the sensor data 235 (e.g., one or more images of the first user, gesture data from one or more position sensors coupled to the first user), and the trained ML model(s) 295 may classify whether the representation(s) of the first user (and / or the gestures of the first user) in the sensor data 235 represent a dancing activity of the first user, and the condition detector 255 may use the classification to determine whether the dancing condition is satisfied. In some examples, the trained machine learning model(s) may extract features and / or attributes from the sensor data 235 to identify features associated with the first user. In some examples, the trained machine learning model(s) 295 may determine whether a condition is satisfied based on features associated with the first user. In some examples, the condition detector 255 may provide the features of the first user and / or the second user to the trained ML model(s) 295, and the trained ML model(s) 295 may predict a relationship type (and / or relationship category) between the first user and the second user (e.g., friends, family, co-workers, acquaintances, or strangers) to determine whether a condition related to the relationship type (and / or relationship category) is satisfied. In some examples, the condition detector 255 may provide the trained ML model(s) 295 with images (e.g., from sensor data 235) of the first user and / or the second user, and the trained ML model(s) 295 may attempt to identify the location of the first user and / or the second user in the environment based on environmental recognition of the scene depicted in the image(s). The (plurality) trained machine learning (ML) models 295 may be trained (e.g., by user device 210, user device 220, server(s) 205, and / or another device) for use by condition detector 255 using training data that includes various sensor data (e.g., any of the types described herein with respect to sensor data 235) and pre-generated determinations of whether certain types of conditions corresponding to the sensor data (e.g., with respect to the identity of the second user, a type of relationship (and / or category of relationship) between the first user and the second user, a location of the first user, a location of the second user, an activity of the first user, an activity of the second user, and / or other types of conditions described herein) are satisfied. Figure 2AWithin, a graphic illustration representing condition detector 255 illustrates examples of the types of conditions that condition detector 255 may be configured to detect, such as a condition associated with whether user(s) are at a location (represented by an icon depicting a map marker on a map), a condition associated with the identity of a second user who is viewing a first user (represented by an icon depicting eyes), a condition associated with a type of relationship (and / or relationship category) between a first user and a second user who is viewing the first user (represented by an icon depicting a group of people with ID tags), a condition associated with an activity that the first user or the second user is performing (represented by an icon depicting a bicycle).

[0085] The server(s) 205 include an avatar processor 260. In some examples, the profile avatar identifier 250, the condition detector 255, and / or the avatar processor 260 selects a display avatar from a set of display avatars associated with the profile (identified by the profile identifier 245) based on which conditions of the profile are met and / or not met. The avatar processor 260 can generate display avatar data 265, which can include a representation of the display avatar to be displayed to the second user (e.g., displayed to the user 225 via the user device 220 and / or displayed to the user 215 via the user device 210). In some examples, the avatar processor 260 can generate the representation of the display avatar in the display avatar data 265 by modifying the image from the sensor data 235 to replace at least a portion of the first user in the image with the display avatar selected for the first user (e.g., identified based on the condition(s)). In some examples, the avatar includes at least a face, a head, and / or a body. In some examples, to generate display avatar data 265, avatar processor 260 superimposes an avatar of the first user on at least a portion (e.g., at least a portion of the face and / or head and / or body) of the first user as depicted or otherwise represented in sensor data 235. In some examples, avatar processor 260 generates display avatar data 265 by configuring the display avatar according to features of the first user (e.g., the first user's posture) so that the display avatar matches the features of the first user (e.g., the first user's posture) in display avatar data 265. In some examples, avatar processor 260 generates display avatar data 265 by determining lighting and / or lighting effects (e.g., lighting and / or lighting direction, lighting and / or lighting intensity, lighting and / or lighting color, lighting and / or lighting type) on the first user and applying the lighting and / or lighting effects to the display avatar.

[0086] In some examples, the avatar processor 260 generates the display avatar data 265 using at least one trained machine learning model 295. The trained machine learning model(s) 295 can help ensure that the avatar looks realistic over and blends in with the rest of the first user's body and / or the environment as depicted or otherwise represented in the display avatar data 265 and / or the sensor data 235. The trained machine learning (ML) model(s) 295 can be trained (e.g., by the user device 210, the user device 220, the server(s) 205, and / or another device) for use by the avatar processor 260 using training data including sensor data (e.g., such as the sensor data 235), the selected display avatar, and pre-generated display avatar data (e.g., such as the display avatar data 265) corresponding to the sensor data and the selected display avatar.

[0087] In some examples, the avatar processor 260 realistically blends the avatar into the rest of the first user's body and / or the environment in the sensor data 235, for example, taking into account the lighting in the environment, the head pose (e.g., position and / or orientation) of the first user's head, the facial expressions made by the first user in the sensor data 235, the body pose of the first user's body, the skin color (e.g., skin color), or a combination thereof. For example, in some examples, the avatar processor 260 can determine the ambient lighting of the first user in his or her real or virtual environment (e.g., based on the sensor data 235), and can replicate the same ambient lighting on the avatar. The ambient lighting can include the color of the lighting, the intensity of the lighting, the direction of the lighting (e.g., from the left, from the right, from the top, and / or from the bottom), any shadows cast on the face from (multiple) other objects in the environment, or a combination thereof. This can ensure that immersion is not broken. In some examples, avatar processor 260 may use face detector 240 to determine a facial expression (e.g., smiling, laughing, frowning, crying, surprised, etc.) of the first user's face detected in sensor data 235, and may modify the avatar to apply the same facial expression (e.g., as Figure 7 In some examples, avatar processor 260 may use facial detector 240 to determine a facial expression (e.g., smiling, laughing, frowning, crying, surprised, etc.) of the face of the first user detected in sensor data 235, and may modify or process the avatar to apply the same facial expression, e.g., Figure 7of combined face 730. This can allow a second user (e.g., user 225) viewing displayed avatar data 265 representing a first user (e.g., user 215) to see in displayed avatar data 265 a representation of the first user's mouth moving as the first user speaks, and / or see other facial expressions on the first user's face, realistically and without breaking immersion, while still maintaining the first user's privacy and not revealing the first user's identity (unless the first user elects to use a real avatar under the current condition(s). Figure 2A Within, a graphic representing avatar processor 260 illustrates the environment depicted in the images of sensor data 235 and displayed avatar data 265, wherein a first user's face has a bounding box around it (as in the graphic representing face detector 240), with an enlarged version of the first user's selected displayed avatar (e.g., having a dog-like head) illustrated as extending from the box.

[0088] In some examples, the server(s) 205 may generate an avatar for the first user (e.g., using the condition detector 255), and / or may modify the avatar for the first user (e.g., using the condition detector 255 and / or the avatar processor 260) such that the skin color of the avatar corresponds to the skin color (e.g., skin color) of the first user. Thus, if the displayed avatar replaces only a portion of the first user (e.g., the face or head of the first user), or if the avatar processor 260 retains a portion of the original representation of the first user in the displayed avatar data 265 (either by mistake or intentionally), the skin color of the avatar will not conflict with the skin color of other parts of the user's body (e.g., hands, arms, legs, neck, feet, etc.) visible in the displayed avatar data 265. This ensures that immersion is not disrupted. In some examples, the condition detector 255 and / or the avatar processor 260 may change the skin color of other parts of the user's body (e.g., hands, arms, legs, neck, feet, etc.) visible in the sensor data 235 and / or the displayed avatar data 265 to match the skin color of the selected displayed avatar. This can further increase privacy, for example, so that the second user does not even know the original skin color of the first user - only the skin color of the selected displayed avatar.

[0089] For example, the avatar processor 260 outputs the display avatar data 265 by sending the display avatar data 265 to the user device 210 for output to the user 215 by the output device(s) 270A of the user device 210 and / or by sending the display avatar data 265 to the user device 220 for output to the user 225 by the output device(s) 270B of the user device 220. Figure 2AWithin, the graphic representing the display avatar data 265 illustrates the display avatar data 265 as depicting the same environment in which two persons are present as in the graphic representing the sensor data 235, but the face of one of the persons is changed to the face of the selected display avatar (e.g., having a dog-like head as illustrated relative to the profile avatar recognizer 250 and avatar processor 260).

[0090] User device 210 includes output device(s) 270A. User device 220 includes output device(s) 270B. Output device(s) 270A-270B may include one or more visual output devices, such as display(s) or connector(s) thereof. Output device(s) 270A-270B may include one or more audio output devices, such as speaker(s), headphone(s), and / or connector(s) thereof. Output device(s) 270A-270B may include one or more of output device 1035 and / or communication interface 1040 of computing system 1000. User device 220 causes the display(s) of output device(s) 270A-270B to display avatar data 265.

[0091] In some examples, output device(s) 270A-270B include one or more transceivers. The transceiver(s) may include a wired transmitter, receiver, transceiver, or a combination thereof. The transceiver(s) may include a wireless transmitter, receiver, transceiver, or a combination thereof. The transceiver(s) may include one or more of output devices 1035 and / or communication interface 1040 of computing system 1000. In some examples, user device 210 and / or user device 220 causes the transceiver(s) to send display avatar data 265 to a receiving device. The receiving device may include a display, and the data sent from the transceiver(s) of output device(s) 270A-270B to the receiving device may cause the display of the receiving device to display the display avatar data 265.

[0092] In some examples, the display(s) of the output device(s) 270A-270B of the imaging system 200A function as optical "see-through" displays(s) that allow light from the real-world environment (scene) surrounding the imaging system 200A to pass through (e.g., through) the display(s) of the output device(s) 270A-270B to reach one or both eyes of the user. For example, the display(s) of the output device(s) 270A-270B may be at least partially transparent, semi-transparent, light-permitting, light-transmissive, or a combination thereof. In an illustrative example, the display(s) of the output device(s) 270A-270B include a transparent, semi-transparent, and / or light-transmissive lens and a projector. The display(s) of the output device(s) 270A-270B may include a projector that projects virtual content (e.g., display(s) of the display avatar data 265) onto the lens. For example, the lens can be a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector can project virtual content (e.g., (multiple) display avatars displaying avatar data 265) onto the lens so that from the perspective of one or both eyes of the user, the virtual content appears to be overlaid on the user's view of the environment. In some examples, the projector can project virtual content onto one or both retinas of one or both eyes of the user, rather than onto the lens, which can be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display.

[0093] In some examples, the display(s) of the output device(s) 270A-270B of the imaging system 200A are digital "pass-through" displays that allow a user of a user device of the imaging system 200A (e.g., user 215 of user device 210 or user 225 of user device 220) to view a view of the environment by displaying the view of the environment on the display(s) of the output device(s) 270A-270B. The view of the environment displayed on the digital pass-through display may be a view of the real-world environment surrounding the imaging system 200A, e.g., based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by one or more environment-facing sensors of the sensor(s) 230, modified (e.g., displaying avatar data 265) in some cases using the avatar processor 260. The view of the environment displayed on the digital pass-through display may be a virtual environment (e.g., as in VR), which in some cases may include elements based on the real-world environment (e.g., the boundaries of a room). The environmental view displayed on the digital pass-through display may be an augmented environment based on a real-world environment (e.g., as in AR). The environmental view displayed on the digital pass-through display may be a hybrid environment based on a real-world environment (e.g., as in MR). The environmental view displayed on the digital pass-through display may include virtual content (e.g., display avatar(s) of display avatar data 265) overlaid on other content otherwise incorporated into the environmental view.

[0094] The trained ML model(s) 295 may include one or more neural networks (NNs) (e.g., neural network 800), one or more convolutional neural networks (CNNs), one or more trained time delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (cGANs), one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, or combinations thereof.

[0095] exist Figure 2A, a graph representing (multiple) trained ML models 295 illustrates a set of circles connected to each other. Each circle can represent a node (e.g., node 816), a neuron, a perceptron, a layer, a portion thereof, or a combination thereof. The circles are arranged in columns. The leftmost column of white circles represents an input layer (e.g., input layer 810). The rightmost column of white circles represents an output layer (e.g., output layer 814). The two shaded columns of circles between the leftmost column of white circles and the rightmost column of white circles each represent a hidden layer (e.g., hidden layers 812A-812N).

[0096] In some examples, the imaging system 200A includes feedback engine(s) 275A-275B of a user device (e.g., user device 210 and / or user device 220). The feedback engine(s) 275A-275B are illustrated as part of the user device 210 and the user device 220, respectively, but may additionally or alternatively be part of the server(s) 205. The feedback engine(s) 275A-275B may detect feedback received from a user interface of the user device 210, the user device 220, and / or the server(s) 205. The feedback may include feedback on the display avatar data 265 displayed according to the interactive client application 280A-280B and / or the interactive server application 285 (e.g., using the display(s) of the output device(s) 270A-270B). The feedback may include feedback on the display avatar data 265 itself. The feedback may include feedback on the avatar(s) generated and / or selected and / or processed by the profile avatar recognizer 250 and / or the condition detector 255 and / or the avatar processor 260 and incorporated into the display avatar data 265, and / or feedback on the processing of the selected display avatar(s) in the display avatar data 265 by the avatar processor 260 (e.g., blended with the rest of the image). The feedback may include feedback on the facial detection by the facial detector 240 and / or the facial recognition by the facial detector 240 and / or the profile recognizer 245. The feedback may include feedback on the facial detector 240, the profile recognizer 245, the profile avatar recognizer 250, the condition detector 255, the avatar processor 260, or a combination thereof.

[0097] The feedback engine(s) 275A-275B can detect feedback received from another engine of the imaging system 200A about one engine of the imaging system 200A, for example, whether one engine decides to use data from another engine. The feedback received by the feedback engine(s) 275A-275B can be positive feedback or negative feedback. For example, if one engine of the imaging system 200A uses data from another engine of the imaging system 200A, or if positive feedback is received from a user through a user interface, the feedback engine(s) 275A-275B can interpret it as positive feedback. If one engine of the imaging system 200A rejects data from another engine of the imaging system 200A, or if negative feedback is received from a user through a user interface, the feedback engine(s) 275A-275B can interpret it as negative feedback. Positive feedback can also be based on attributes of sensor data from the sensor(s) 230, such as a user smiling, laughing, nodding, saying an affirmative statement (e.g., "yes," "confirm," "ok," "next"), or otherwise reacting positively to the output of one of the engines described herein or its indication. Negative feedback can also be based on attributes of sensor data from sensor(s) 230, such as the user frowning, crying, shaking his head (e.g., in a "no" motion), uttering a negative statement (e.g., "no," "negative," "bad," "not this"), or otherwise reacting negatively to an output of one of the engines described herein or an indication thereof.

[0098] In some examples, the feedback engine(s) 275A-275B provide feedback as training data to one or more ML systems of the imaging system 200A (e.g., to the server(s) 205) to update one or more trained ML models 295 of the imaging system 200A. For example, the feedback engine(s) 275A-275B may provide feedback as training data to the ML system(s) and / or the trained ML model(s) 295 to update the training of the face detector 240, the profile recognizer 245, the profile avatar recognizer 250, the condition detector 255, the avatar processor 260, or a combination thereof. Positive feedback may be used to reinforce and / or enhance weights associated with the outputs of the ML system(s) and / or the trained ML model(s) 295, and / or weaken or remove weights other than weights associated with the outputs of the ML system(s) and / or the trained ML model(s) 295. Negative feedback can be used to weaken and / or remove weights associated with the outputs of the ML system(s) and / or trained ML model(s) 295, and / or to strengthen and / or enhance weights other than weights associated with the outputs of the ML system(s) and / or trained ML model(s) 295.

[0099] In some examples, certain elements of imaging system 200A (e.g., face detector 240, profile recognizer 245, profile avatar recognizer 250, condition detector 255, avatar processor 260, feedback engine(s) 275A-275B, trained ML model(s) 295, interactive client applications 280A-280B, interactive service application 285, data store 290, or a combination thereof) include software elements, such as instruction sets corresponding to programs, running on a processor, such as processor 1010 of computing system 1000, image processor 150, host processor 152, ISP 154, or a combination thereof. In some examples, these elements of imaging system 200A include one or more hardware elements, such as a dedicated processor (e.g., processor 1010 of computing system 1000, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, these elements of imaging system 200A may include a combination of one or more software elements and one or more hardware elements.

[0100] In some examples, certain elements of imaging system 200A (e.g., face detector 240, profile recognizer 245, profile avatar recognizer 250, condition detector 255, avatar processor 260, feedback engine(s) 275A-275B, trained ML model(s) 295, interactive client applications 280A-280B, interactive service application 285, data store 290, or a combination thereof), other instances thereof, or portions thereof are presented as being associated with Figure 2A Parts of equipment different from the equipment shown are included and / or in conjunction with Figure 2A For example, user device 220 may include its own instance of sensor(s) 230 even though these sensors are not running on Figure 2A In addition, examples of elements of server(s) 205 running on user device 210 are shown in FIG. 2B .

[0101] FIG2B is a block diagram illustrating an example architecture of an imaging process performed by an imaging system 200B having two user devices (eg, user device 210, user device 220). The imaging system 200B of FIG2B is similar to Figure 2A The imaging system 200A of FIG. 1 is shown in FIG. 200A , but lacks the server(s) 205 . Figure 2A All components that are part of the server(s) 205 in the imaging system 200A of FIG. 2B are part of the user device 210 in the imaging system 200B of FIG. 2B. Figure 2AAll processes and / or operations performed by the (multiple) servers 205 in the imaging system 200A of FIG. 2B are performed by the user device 210 in the imaging system 200B of FIG. 2B. In some examples, the user device 210 may be used to perform the following operations: Figure 2A An imaging system between the imaging system 200A of FIG. 2 and the imaging system 200B of FIG. 2B , wherein certain components and / or operations are maintained at (multiple) servers 205 and other components and / or operations are maintained at the user device 210 .

[0102] Figure 3A 300 is a perspective view illustrating a head mounted display (HMD) 310 used as part of an imaging system 200. For example, the HMD 310 may be an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 may be an example of a user device (e.g., the user device 210 and / or the user device 220) of an imaging system (e.g., the imaging system 200A and / or the imaging system 200B). The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of the sensor(s) 230 of the imaging systems 200A-200B. When the user's eye(s) face the display(s) 340, the HMD 310 includes a third camera 330C and a fourth camera 330D facing the user's eye(s). The third camera 330C and the fourth camera 330D may be examples of the sensor(s) 230 of the imaging system 200A-200B. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, and the HMD may also include other types of sensor(s) 230 of the imaging system 200. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0103] The HMD 310 may include one or more displays 340 visible to the user 320 wearing the HMD 310 on the user's 320 head. The one or more displays 340 of the HMD 310 may be examples of one or more displays of the output device(s) 270A-270B of the imaging system 200A-200B. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be digital "pass-through" displays or optical "see-through" displays.

[0104] The HMD 310 may include one or more headphones 335, which may be used as speakers and / or headphones to output audio to one or more ears of a user of the HMD 310 and may be an example of output device(s) 270A-270B. Figure 3A and Figure 3B 335, but it should be understood that the HMD 310 may include two earphones, one for each ear (left and right ear) of the user. In some examples, the HMD 310 may also include one or more microphones (not shown). The one or more microphones may be examples of the sensor(s) 230 of the imaging systems 200A-200B. In some examples, the audio output by the HMD 310 to the user through the one or more earphones 335 may include or be based on audio recorded using the one or more microphones.

[0105] Figure 3B The user 320 is wearing Figure 3A310 is a perspective view 350 of a head mounted display (HMD) of the user 320. The user 320 wears the HMD 310 on the head of the user 320, above the eyes of the user 320. The HMD 310 can capture images with a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images to the eyes of the user 320 using (multiple) displays 340. In some examples, the output images may include display avatar data 265. The output images may be based on the images captured by the first camera 330A and the second camera 330B (e.g., sensor data 235), for example, overlaid with virtual content (e.g., (multiple) display avatars of the display avatar data 265). The output images may provide a stereoscopic view of the environment, in some cases, overlaid with virtual content and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on the image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on the image captured by the second camera 330B. For example, the HMD 310 may provide overlaid virtual content in a display image overlaid on the image captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the (multiple) display 340. In this way, sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earphone 335 of the HMD 310 is illustrated in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earphone 335 and / or through another earphone (not illustrated) of the HMD 310 in the other ear (not illustrated) of the user 320.

[0106] Figure 4A 4 is a perspective view 400 illustrating a front surface of a mobile phone 410 that includes a front-facing camera and can be used as part of the imaging system 200. The mobile phone 410 can be an example of a user device (e.g., user device 210 and / or user device 220) of an imaging system (e.g., imaging system 200A and / or imaging system 200B). For example, the mobile phone 410 can be a cellular phone, a satellite phone, a portable game console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0107] The front surface 420 of the mobile phone 410 includes a display 440. The front surface 420 of the mobile phone 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B can be examples of the sensor(s) 230 of the imaging system 200A-200B. When content (e.g., display avatar data 265 output by the avatar processor 260) is displayed on the display 440, the first camera 430A and the second camera 430B can face the user, including the user's (two) eyes. The display 440 can be an example of the display(s) of the output device(s) 270A-270B of the imaging system 200A-200B.

[0108] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be located in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras located between the display 440 and the rest of the mobile phone 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the plane of the front surface 420 of the mobile phone 410. The first camera 430A and the second camera 430B may be two of the one or more cameras of the mobile phone 410. In some examples, the front surface 420 of the mobile phone 410 may have only a single camera.

[0109] In some examples, the display 440 of the mobile phone 410 displays one or more output images to the user using the mobile phone 410. In some examples, the output images may include the display avatar data 265. The output images may be based on images captured by the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D (e.g., sensor data 235), for example, overlaid with virtual content (e.g., display avatar(s) of the display avatar data 265).

[0110] In some examples, the front surface 420 of the mobile phone 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of the sensor(s) 230 of the imaging system 200A-200B. In some examples, the front surface 420 of the mobile phone 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of the sensor(s) 230 of the imaging system 200A-200B. In some cases, the front surface 420 of the mobile phone 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile phone 410 may be examples of the display(s) of the output device(s) 270A-270B of the imaging system 200A-200B. For example, the one or more displays 440 may include one or more touch screen displays.

[0111] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (e.g., earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410. Figure 4A 435A, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of the sensor(s) 230 of the imaging systems 200A-200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile phone 410, where these microphones are examples of the sensor(s) 230 of the imaging systems 200A-200B. In some examples, the audio output to the user by the mobile phone 410 through the one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0112] Figure 4B 4 is a perspective view 450 illustrating a rear surface 460 of a mobile phone that includes a rear camera and can be used as part of the imaging system 200. The mobile phone 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile phone 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear cameras. The third camera 430C and the fourth camera 430D can be Figure 2A- An example of the sensor(s) 230 of the imaging system 200A-200B of FIG2B The third camera 430C and the fourth camera 430D face in a direction perpendicular to the plane of the rear surface 460 of the mobile phone 410 .

[0113] The third camera 430C and the fourth camera 430D may be two of the one or more cameras of the mobile phone 410. In some examples, the rear surface 460 of the mobile phone 410 may have only a single camera. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the (multiple) sensors 230 of the imaging system 200A-200B. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the (multiple) sensors 230 of the imaging system 200A-200B. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0114] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (e.g., earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410. Figure 4B 435B, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio device. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of the sensor(s) 230 of the imaging systems 200A-200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the rear surface 460 of the mobile phone 410, where these microphones are examples of the sensor(s) 230 of the imaging systems 200A-200B. In some examples, the audio output to the user by the mobile phone 410 through the one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using the one or more microphones.

[0115] The mobile phone 410 can use the display 440 on the front surface 420 as a pass-through display. For example, the display 440 can display an output image, such as the display avatar data 265. The output image can be based on the image captured by the third camera 430C and / or the fourth camera 430D (e.g., the sensor data 235), for example, overlaid with virtual content (e.g., (multiple) display avatars of the display avatar data 265) and / or applied with modifications of the avatar processor 260. The first camera 430A and / or the second camera 430B can capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, the sensor data from the first camera 430A and / or the second camera 430B can capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0116] Figure 5A5 is a conceptual diagram 500A illustrating the generation of a modified image 550 from an image 510 by replacing a representation of the user 520 in the image with a first avatar 530 of the user 520. For example, an image 510 is illustrated, which may be an example of sensor data 235 captured by sensor(s) 230. Face detector 240 detects a face 525 of the user 520 in image 510. Profile recognizer 245 recognizes that the face 525 of the user 520 in image 510 belongs to a user 520 named John Smith, and retrieves a profile for the user 520 (e.g., from data store 290) or creates a profile if the profile does not already exist (e.g., stored in data store 290). Profile avatar recognizer 250 identifies avatars that may be used for the user 520 under different conditions based on the profile of the user 520. The condition detector 255 identifies which of the condition(s) associated with the profile (identified by the profile identifier 245) are satisfied, and which of the condition(s) associated with the profile (identified by the profile identifier 245) are not satisfied. The profile avatar identifier 250 and / or the condition detector 255 selects a first avatar 530 to represent the user 520 in the modified image 550. The first avatar 530 may be referred to as a display avatar. The avatar processor 260 generates the modified image 550 by modifying the image 510 to represent the user 520 using the first avatar 530 instead of the representation of the user 520 in the image 510. The first avatar 530 is represented as having a dog-like head. Because the display avatar 530 does not look like the actual appearance of the user 520, the display avatar 530 may be used to protect the privacy of the user 520 in an at least partially virtual environment, such as in an environment corresponding to a network-based interactive system. The first avatar 530 appears different from the user 520 in both the head and body parts. For example, in terms of the torso, the avatar 530 is shown wearing different clothes (a shirt with a football logo and pants with vertical stripes on the sides) than the user 520 is actually wearing in the image 510 (a jacket and pants without stripes).

[0117] Image 510 depicts light from light source 540 (i.e., a window). In some examples, the light source may include a window, a lamp, the sun, a display screen, a light bulb, or a combination thereof. The face 525 (and / or body) of user 520 is illuminated from the left by light source 540 in image 510. Therefore, to generate modified image 550, avatar processor 260 modifies image 510 to replace at least a portion of user 520 with first avatar 530 of user 520, and applies and / or simulates the lighting from light source 540 to first avatar 530 in modified image 550. The avatar processor 260 can apply and / or simulate lighting from the light source 540 to the first avatar 530 by applying and / or simulating lighting from the same direction as the light source 540 (e.g., from the left side of the first avatar 530, similar to the light from the left side of the user 520), lighting having a similar light color as the light from the light source 540 illustrated in the image 510, using a light pattern similar to the light from the light source 540 illustrated in the image 510, or a combination thereof.

[0118] In addition to the exposed face of user 520 (e.g., face 525 or the face of first avatar 530), both image 510 and modified image 550 depict exposed hands of user 520 and / or first avatar 530. Condition detector 255 and / or avatar processor 260 may synchronize skin tones (e.g., skin color) between face 525, avatar 530, and / or other parts of the body of user 520 visible in image 510 and / or modified image 550 (e.g., in the hands of user 520). For example, in some examples, the imaging system may generate avatar 530, or modify avatar 530 after generating avatar 530, so that the skin color of the avatar corresponds to (e.g., matches) the skin color of face 525 and / or other parts of the body of user 520 visible in image 510 (e.g., such as in the hands of user 520). In some examples, the imaging system may change the skin color of other parts of the user's body visible in the image (e.g., the hands of user 520) to match the skin color of the avatar.

[0119] Figure 5B is an illustration of replacing the representation of user 520 in image 510 with second avatar 565 of user 520. Figure 5A Conceptual diagram 500B of generating a modified image 560 from image 510. Figure 5A As indicated, profile avatar recognizer 250 recognizes avatars that are available to user 520 under different conditions based on the profile of user 520. These avatars include at least Figure 5A The first avatar 530 and Figure 5BThe condition detector 255 identifies which of the conditions (multiple) associated with the profile (identified by the profile identifier 245) are satisfied, and which of the conditions (multiple) associated with the profile (identified by the profile identifier 245) are not satisfied, and Figure 5B Produced in Figure 5A The profile avatar recognizer 250 and / or the condition detector 255 selects the second avatar 565 to represent the user 520 in the modified image 560 based on which of the conditions the condition detector 255 determines are met and / or not met. The avatar processor 260 generates the modified image 560 by modifying the image 510 to represent the user 520 using the second avatar 565 instead of the representation of the user 520 in the image 510. The second avatar 565 is represented as being more realistic and more human, and is more similar in appearance to the appearance of the user 520 himself, for example, wearing clothes similar to the user 520. Therefore, the conditions associated with using the second avatar 565 to represent the user 520 may include, for example, the condition that the viewer of the second avatar 565 may have a closer relationship with the user 520 (e.g., a friend or family member) rather than a stranger, in which case the user 520 may wish to use the first avatar 530 instead, which is more privacy-preserving.

[0120] Figure 5C is an illustration of replacing the representation of user 520 in image 510 with a third avatar 575 of user 520. Figure 5A Conceptual diagram 500C of image 510 generating a modified image 570. Figure 5A As indicated, profile avatar recognizer 250 recognizes avatars that are available to user 520 under different conditions based on the profile of user 520. These avatars include at least Figure 5A The first avatar 530, Figure 5B The second avatar 565 and Figure 5C The condition detector 255 identifies which of the conditions (multiple) associated with the profile (identified by the profile identifier 245) are satisfied, and which of the conditions (multiple) associated with the profile (identified by the profile identifier 245) are not satisfied, and Figure 5C Produced in Figure 5A and Figure 5BCompared to different results. The profile avatar recognizer 250 and / or the condition detector 255 selects the third avatar 575 to represent the user 520 in the modified image 570 based on which of the conditions are determined by the condition detector 255 to be met and / or not met. The avatar processor 260 generates the modified image 570 by modifying the image 510 to represent the user 520 using the third avatar 575 instead of the representation of the user 520 in the image 510. The third avatar 575 is represented as having a cartoon smiling face. For example, a condition associated with using the third avatar 575 to represent the user 520 may include a condition that the user 520 has indicated that he / she is very happy. In some examples, an avatar similar to the third avatar 575 may be used with a frowning face, for example, under the condition that the user 520 feels sad.

[0121] Fig. 6A 6 is a conceptual diagram 600 illustrating that a second user 605 uses a head mounted device 610 to view an interactive environment including a first user 620, represented by a first avatar 625, which is displayed to the second user 605 through the head mounted device 610 based on a condition 635 associated with the first user 620 and a relationship indicator 640 indicating a family relationship between the second user 605 and the first user 620. The head mounted device 610 is an example of the HMD 310. The head mounted device 610 displays a field of view (FOV) 615A including the first avatar 625 representing the first user 620 to the second user 605. The data storage 290 stores the condition 635 of the first user 620 in a profile 630 of the first user 620. The profile 630 is an example of a profile identified using the profile identifier 245. The conditions 635 of the first user 620 include an approval list indicating that family members of the first user 620 are presented with a first avatar 625 to represent the first user 620, while others are presented with a second avatar 655 to represent the first user 620. The data store 290 also stores a relationship indicator 640 indicating that the second user 605 and the first user 620 are indeed family. Therefore, the second user 605 is on the approval list of the first user 620. Similar to Figure 5B The second avatar 565 of the first user 620 may be a realistic avatar similar to the real appearance of the first user 620. Figure 5A The first avatar 530 or Figure 5C655 is a less-than-realistic avatar that is different from the actual appearance of the first user 620. Therefore, the first avatar 625 (e.g., a realistic avatar) is presented to the second user 605 in the FOV 615A via the head mounted device 610 as the selected representation of the first user 620 because the condition 635 associated with the first user 620 indicates that different avatars should be presented to different users based on the relationship type (and / or relationship category) and because, as indicated by the relationship indicator 640, the first user 620 and the second user 605 are family members.

[0122] Figure 6B 6 is a conceptual diagram 650 illustrating a second user 605 using a head mounted device 610 to view an interactive environment including a first user 620 represented by a second avatar 655 displayed to the second user 605 via the head mounted device 610 based on a condition 635 associated with the first user 620 and a relationship indicator 660 indicating a lack of a family relationship between the second user 605 and the first user 620. Fig. 6A The relationship indicator 640 is different. Figure 6B The relationship indicator 660 indicates that the second user 605 and the first user 620 are not family. Therefore, under the condition 635 of the first user 620, the second user 605 is not on the approved list of the first avatar 625. Therefore, the second avatar 655 (e.g., a non-realistic avatar) is presented to the second user 605 in the FOV 615B via the head mounted device 610 as the selected representation of the first user 620 because the condition 635 associated with the first user 620 indicates that different avatars should be presented to different users based on the relationship type (and / or relationship category) and because, as indicated by the relationship indicator 660, the first user 620 and the second user 605 are not family.

[0123] In some examples, if condition detector 255 determines that certain conditions (e.g., condition 635) are met or not met, then a true representation of the user (e.g., first user 620) rather than any avatar may be presented (e.g., unmodified or processed but without avatar representation) to the viewing user (e.g., second user 605). Figure 5A-Figure 5C In the context of , under certain conditions, image 510 can be presented to a viewing user (e.g., second user 605), thereby allowing the viewing user (e.g., second user 605) to see a true representation of the user (e.g., first user 620, user 520).

[0124] Figure 77 is a conceptual diagram 700 illustrating combining an identity input 710 and an expression input 720 to generate a combined face 730 having the identity of the identity input 710 and the expression of the expression input 720. Specifically, the combined face 730 is generated using the trained ML model(s) 740 based on using the identity input 710 and the expression input 720 as inputs to the trained ML model(s) 740 and / or as training data for training the trained ML model(s) 740. The trained ML model(s) 740 may be an example of one or more trained ML models 295 used by the condition detector 255 and / or the avatar processor 260. The trained ML model(s) 740 may be part of the condition detector 255 and / or the avatar processor 260.

[0125] exist Figure 7 In the illustrated example, the identity input 710 is an avatar 715 for representing the user. Figure 7 In the illustrated example, the expression input 720 is the real face 725 of the user to be represented by the avatar 715. Various facial features and / or facial attributes representing the identity of the identity input 710 (e.g., the avatar 715) appear in the combined face 730. For example, the combined face 730 includes the hairstyle, eyebrow thickness, eye shape, facial feature proportions, and overall artistic style of the identity input 710 (e.g., the avatar 715). Various facial features and / or facial attributes representing the expression of the expression input 720 appear in the combined face 730. For example, the combined face 730 includes the mouth shape and expression, facial wrinkles, eyebrow arches, and / or eyebrow orientation of the expression input 720.

[0126] In some examples, the trained ML model(s) 740 may be trained (e.g., by user device 210, user device 220, server(s) 205, and / or another system) to generate a composite face (e.g., composite face 730) using training data. In some examples, the training includes identity input images (e.g., identity input 710) and expression input images (e.g., expression input 720) of real and / or artistic faces and composite faces (e.g., composite face 730) pre-generated using identity features from the corresponding identity input images and expression features from the corresponding expression input images. In some examples, the trained machine learning model(s) 740 may include one or more generative adversarial networks (GANs) and / or other deep learning (DL) based generative ML models. In some examples, the identity input is an example of an avatar 715, which can be an example of an avatar recognized by the profile avatar recognizer 250 in the user's profile, such as the first avatar 530, the second avatar 565, the third avatar 575, the first avatar 625, the second avatar 655, or a combination thereof. In some examples, the expression input is an example of a real face 725 of the user to be represented by a version of the avatar 715, such as the face 525 or another face detected using the face detector 240. The combination of the face 730 using the identity with the identity input 710 (e.g., the avatar 715) and the expression of the expression input 720 (e.g., the real face 725) can allow the viewing user to see the mouth movement of the user represented by the avatar, see when the user represented by the avatar smiles or frowns, see other facial expressions on the face of the user represented by the avatar, etc. This enhances the immersion, flexibility, customization, personalization, and expressiveness of the network-based interactive system, while increasing privacy and security by not revealing the identity of the user (unless the user chooses to do so).

[0127] Figure 8 800 may be used for media processing operations. The neural network 800 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 800 may be an example of one of the trained ML model(s) 295, the trained ML model(s) 740, the trained ML model(s) 740, one or more trained ML models, or a combination thereof used in the process 900. The neural network 800 may be used by the face detector 240, the profile recognizer 245, the profile avatar recognizer 250, the condition detector 255, the avatar processor 260, or a combination thereof.

[0128] The input layer 810 of the neural network 800 includes input data. The input data of the input layer 810 may include data representing pixels of one or more input image frames. In some examples, the input data of the input layer 810 includes data representing pixels of image data (e.g., an image captured by the image capture and processing system 100, an image of sensor data 235 captured by the sensor(s) 230, an image captured by one of the cameras 330A-330D, an image captured by one of the cameras 430A-430D, an image 510, an image of the user 620 processed to include the first avatar 625 or the second avatar 655, an identity input 710, an expression input 720, an image of a representation of the first user having operation 905, or a combination thereof).

[0129] The image may include image data from an image sensor, the image data including raw pixel data (including, for example, a single color per pixel based on a Bayer filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). The neural network 800 includes a plurality of hidden layers 812A, 812B through 812N. The hidden layers 812A, 812B through 812N include "N" hidden layers, where "N" is an integer greater than or equal to 1. The number of hidden layers may include as many layers as desired for a given application. The neural network 800 also includes an output layer 814 that provides an output resulting from the processing performed by the hidden layers 812A, 812B through 812N.

[0130] In some examples, output layer 814 may provide output images, such as display avatar data 265, avatar 530, modified image 550, modified image 560, modified image 570, FOV 615A with first avatar 625, FOV 615B with second avatar 655, avatar 715, combined face 730, display avatar of operations 910-915, or a combination thereof. In some examples, output layer 814 may also provide other types of data, such as face detection data of face detector 240, face recognition data of face detector 240 and / or profile recognizer 245, condition detection of condition detector 255, or a combination thereof.

[0131] Neural network 800 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with the filters is shared between different layers, and each layer retains information as it is processed. In some cases, neural network 800 may include a feedforward network, in which case there is no feedback connection where the output of the network is fed back into itself. In some cases, network 800 may include a recursive neural network, which may have loops that allow information to be carried across nodes while reading in input.

[0132] In some cases, information can be exchanged between layers by node-to-node interconnections between different layers. In some cases, the network may include a convolutional neural network that may not link each node in a layer to each other node in the next layer. In a network that exchanges information between layers, the nodes of the input layer 810 can activate a set of nodes in the first hidden layer 812A. For example, as shown in the figure, each input node of the input layer 810 can be connected to each node of the first hidden layer 812A. The nodes of the hidden layer can transform the information by applying an activation function (e.g., a filter) to the information of each input node. Then, the information derived from the transformation can be passed to the nodes of the next hidden layer 812B, and the nodes of the next hidden layer 812B can be activated, and the nodes can perform their own specified functions. Example functions include convolution functions, downscaling, upscaling, data transformations, and / or any other suitable functions. Then, the output of the hidden layer 812B can activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 812N can activate one or more nodes of the output layer 814, which provides a processed output image. In some cases, although a node in neural network 800 (e.g., node 816) is shown as having multiple output lines, the node has a single output and all lines shown as outputting from the node represent the same output value.

[0133] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from the training of the neural network 800. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable digital weight that can be tuned (e.g., based on a training data set), thereby allowing the neural network 800 to adapt to the input and be able to learn as more and more data is processed.

[0134] The neural network 800 is pre-trained to process features of data from the input layer 810 using different hidden layers 812A, 812B to 812N to provide an output through the output layer 814.

[0135] Fig. 9 900. The user role management process 900 may be performed by a user role management system. In some examples, for example, the user role management system may include the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200A, the imaging system 200B, the server (s) 205, the user device 210, the user device 220, the HMD 310, the mobile phone 410, Figure 5A-Figure 5C (multiple) imaging systems, head mounted device 610, (multiple) trained ML models 740, neural network 800, computing system 1000, processor 1010, or a combination thereof.

[0136] At operation 905, the user role management system is configured and capable of identifying a profile associated with a first user. The profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar in the plurality of avatars. Examples of the first user include user 215, user 225, a user whose face is detected in sensor data 235 using face detector 240, a user whose profile is identified using profile recognizer 245, user 320 of HMD 310, user 410 of mobile phone 410, user 520, first user 620, second user 605, a user represented by avatar 715 in identity input 710, a user whose real face 725 is depicted in expression input 720, a user depicted in an input image provided to input layer 810 of NN 800, any other (multiple) user described herein, or a combination thereof. Examples of profiles include profiles identified using profile recognizer 245, profiles stored in data storage 290, profile 630 of first user 620, any other (multiple) profiles described herein, or a combination thereof. Examples of the plurality of avatars include avatar(s) identified using profile avatar recognizer 250 corresponding to a profile identified using profile recognizer 245, avatar(s) processed using avatar processor 260, avatar(s) represented in display avatar data 265, avatar(s) stored in data store 290, first avatar 530, second avatar 565, third avatar 575, first avatar 625, second avatar 655, avatar 715, combined face 730, avatars input into input layer 810 of NN 800, avatars output via output layer 814 of NN 800, any other avatar(s) described herein, or a combination thereof. Examples of conditions include conditions detected and / or detectable by condition detector 255, conditions associated with a profile identified using profile recognizer 245, conditions identified in data store 290, condition 635 of profile 630, any other condition(s) described herein, or a combination thereof.

[0137] In some examples, identifying the profile associated with the first user includes identifying the profile associated with the first user based on one or more communications from a user device associated with the first user. Examples of user devices associated with the first user include image capture and processing system 100, user device 210, user device 220, HMD 310, mobile handset 410, computing system 1000, or a combination thereof. In some examples, the one or more communications include wireless communications, such as communications or wireless local area network (WLAN) communications.

[0138] In some examples, the user role management system is configured and capable of receiving an image of the environment. In some examples, the user role management system includes an image sensor connector that couples and / or connects the image sensor to the rest of the user role management system (e.g., including a processor and / or memory of the user role management system), and in some examples, the user role management system receives image data from the image sensor by receiving the image data from, through, and / or using the image sensor connector.

[0139] Examples of image sensors include image sensor 130, sensor(s) 230, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor that captures image 510, an image sensor for capturing an image used as input data for input layer 810 of NN 800, input device 1045, another image sensor described herein, or a combination thereof.

[0140] Examples of image data include image data captured using the image capture and processing system 100, sensor data 235 captured using image sensor(s) of the sensor(s) 230, image data captured using the first camera 330A, image data captured using the second camera 330B, image data captured using the third camera 330C, image data captured using the fourth camera 330D, image data captured using the first camera 430A, image data captured using the second camera 430B, image data captured using the third camera 430C, image data captured using the fourth camera 430D, image 510, an image of a real face 725 used as expression input 720, an image used as input data for the input layer 810 of the NN 800, an image captured using the input device 1045, another image described herein, another set of image data described herein, or a combination thereof.

[0141] In some examples, the user persona management system is configured and capable of detecting at least a portion of a first user in an image (e.g., at least a portion of a face of the first user). The one or more processors may identify a profile associated with the first user based on at least a portion of the first user detected in the image (e.g., based on facial detection and / or facial recognition) (as in operation 905).

[0142] At operation 910, the user role management system is configured and capable of selecting a display avatar from a plurality of avatars based on a first condition in one or more conditions, wherein the first condition is associated with one or more characteristics associated with a first user. Examples of the first condition include any of the one or more conditions listed above.

[0143] For example, examples of one or more characteristics associated with a first user include a location of the first user in a real environment, a location of a representation (e.g., an avatar) of the first user in an at least partially virtual environment, activities performed by the first user in the real environment, activities performed by the representation (e.g., an avatar) of the first user in the at least partially virtual environment, a person (e.g., a second user) who is viewing the first user in the real environment, a person (e.g., a second user) who is viewing the representation (e.g., an avatar) of the first user in the at least partially virtual environment, a location of someone (e.g., a second user) who is viewing the first user in the real environment and / or who is viewing the representation (e.g., an avatar) of the first user in the at least partially virtual environment, activities performed by someone (e.g., a second user) who is viewing the first user in the real environment and / or who is viewing the representation (e.g., an avatar) of the first user in the at least partially virtual environment, other feature(s) described herein, or combinations thereof.

[0144] In some examples, one or more features associated with the first user include the identity of a second user to whom an avatar is to be presented, and the first condition corresponds to whether the identity of the second user is identified in a predetermined data structure corresponding to the first user. For example, if the second user has a first specified identity (e.g., a friend or family member of the first user), a first avatar representing the first user may be presented to the second user, and if the second user has a second specified identity (e.g., a stranger to the first user), a second avatar representing the first user may be presented to the second user. For example, condition 635 may be understood to be related to the identity of second user 605, for example, based on whether second user 605 is a family member of first user 605.

[0145] In some examples, one or more features associated with the first user include a relationship category between a second user to whom an avatar is to be presented and the first user, and the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user. For example, if the first user and the second user have a first specified relationship type (e.g., family members), a first avatar representing the first user may be presented to the second user, and if the first user and the second user have a second specified relationship type (e.g., colleagues), a second avatar representing the first user may be presented to the second user. For example, condition 635 may be understood to be related to a relationship type between the first user 620 and the second user 605, for example, based on whether the first user 620 and the second user 605 have a specified relationship category (e.g., family members).

[0146] In some examples, one or more features associated with the first user include the first user's location in the environment, and the first condition corresponds to whether the first user's location in the environment falls within a predetermined area, a predetermined area type, a predetermined location, a predetermined location type, and / or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user. The environment can be a real environment or a virtual environment. For example, if the first user is located in a first area of ​​a real or virtual environment (e.g., the first user's virtual or real home), a first avatar representing the first user can be presented to the second user, and if the first user is located in a second area of ​​the real or virtual environment (e.g., a virtual or real concert venue), a second avatar representing the first user can be presented to the second user.

[0147] In some examples, one or more features associated with the first user include a location of the second user in an environment, and the first condition corresponds to whether the location of the second user in the environment falls within a predetermined range of at least one of a predetermined location and / or a predetermined location type in the environment identified in a predetermined data structure corresponding to the first user. The environment can be a real environment or a virtual environment. For example, if the second user is within the range of a first location of a real or virtual environment (e.g., the center of the first user's virtual or real home), a first avatar representing the first user may be presented to the second user, and if the second user is within the range of a second location of the real or virtual environment (e.g., the stage of a virtual or real concert venue), a second avatar representing the first user may be presented to the second user.

[0148] In some examples, the one or more features associated with the first user include an activity performed by the first user, and the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user. For example, if the first user is performing a first activity in a real or virtual environment (e.g., playing a game), a first avatar representing the first user may be presented to the second user, and if the first user is in a second area of ​​the real or virtual environment (e.g., watching a video), a second avatar representing the first user may be presented to the second user.

[0149] In some examples, the one or more features associated with the first user include an activity performed by the second user, and the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user. For example, if the second user is performing the first activity in a real or virtual environment (e.g., playing a game), a first avatar representing the first user may be presented to the second user, and if the second user is in a second area of ​​the real or virtual environment (e.g., watching a video), a second avatar representing the first user may be presented to the second user.

[0150] At operation 915, the imaging system is configured and capable of outputting a display avatar for presentation to a second user, wherein the display avatar is to be presented according to one or more characteristics associated with the first user. Examples of the second user include any of the examples of the first user listed above.

[0151] In some examples, the user role management system is configured and capable of identifying a second profile associated with a third user. The second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar in the second plurality of avatars. The user role management system may select a second display avatar from the second plurality of avatars based on a second condition in the second set of one or more conditions. The second condition is associated with a second set of one or more features associated with the second user. The user role management system may output a second display avatar for presentation to the second user. The second display avatar will be presented according to a second set of one or more features associated with the third user. Examples of the third user include any examples of the first user and / or the second user listed above. Examples of the second profile include any examples of the profiles listed above. Examples of the second plurality of avatars include any examples of the plurality of avatars listed above. Examples of the second set of one or more conditions include any examples of the one or more conditions listed above. Examples of the second set of one or more features include any examples of the one or more features listed above.

[0152] In some examples, the user role management system is configured and capable of identifying a second profile associated with a third user. The second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar in the second plurality of avatars. The user role management system may select a second display avatar from the second plurality of avatars based on the fact that none of the conditions in the second set of one or more conditions are satisfied. The user role management system may output the second display avatar for presentation to the second user. Examples of the third user include any examples of the first user and / or the second user listed above. Examples of the second profile include any examples of the profiles listed above. Examples of the second plurality of avatars include any examples of the plurality of avatars listed above. Examples of the second set of one or more conditions include any examples of the one or more conditions listed above.

[0153] In some examples, the user role management system is configured and capable of receiving an image of an environment and detecting at least a portion (e.g., a face) of a first user in the image. Identifying a profile associated with the first user can be based on at least a portion of the first user detected in the image. The user role management system can generate a modified image at least in part by modifying the image according to one or more features of the first user to replace at least a portion of the first user in the image with a display avatar. Outputting the display avatar for presentation to a second user can include outputting the modified image for presentation to the second user. Examples of images include image data captured using the image capture and processing system 100, sensor data 235 captured using image sensor(s) of the sensor(s) 230, image data captured using the first camera 330A, image data captured using the second camera 330B, image data captured using the third camera 330C, image data captured using the fourth camera 330D, image data captured using the first camera 430A, image data captured using the second camera 430B, image data captured using the third camera 430C, image data captured using the fourth camera 430D, image 510, an image of a real face 725 used as expression input 720, an image used as input data for the input layer 810 of the NN 800, an image captured using the input device 1045, another image described herein, another set of image data described herein, or a combination thereof. Examples of modified images include display avatar data 265, modified image(s) displayed using display(s) 340, modified image(s) displayed using display(s) 440, modified image 550, modified image 560, modified image 570, FOV 615A, FOV 615B, an image including avatar 715, an image including combined face 730, an image output using output layer 814 of NN 800, or a combination thereof.

[0154] In some examples, at least a portion of the environment in the image includes one or more virtual elements. In some examples, at least a portion of the image of the environment is captured by an image sensor, or is based on content captured by an image sensor (e.g., image sensor 130, sensor(s) 230). In some examples, generating the modified image includes using a displayed avatar and at least a portion of the first user in the image as input to a trained machine learning model (e.g., trained ML model(s) 295, trained ML model(s) 740, NN 800), which modifies the image to replace at least a portion of the first user with a displayed avatar based on one or more characteristics of the first user.

[0155] In some examples, the portion of the first user includes one or more facial features of the first user. In some examples, identifying the profile associated with the first user includes using facial recognition (e.g., facial detector 240) to identify the identity of the one or more facial features of the first user in the image.

[0156] In some examples, the user role management system is configured and capable of modifying the display avatar based on one or more display preferences associated with the second user before outputting the display avatar for presentation to the second user. For example, the display preferences of the second user may indicate a visual impairment of the second user, such as color blindness or dyslexia, and the user role management system may modify the display avatar to use colors distinguishable by color blind users, and / or use words, characters, and / or symbols readable by dyslexic users. In some examples, the display preferences of the second user may indicate a preference for not viewing a specified type of avatar, such as a shirtless avatar, or an avatar that includes obscene images, blood, or specified text or design. In some examples, the display preferences of the second user may indicate a preference for viewing a specified type of avatar, or color, or design, etc.

[0157] In some examples, the user role management system is configured to and can select a second display avatar from a plurality of avatars based on a change from a first condition to a second condition in one or more conditions. The user role management system can change from outputting a display avatar to present to a second user to outputting a second display avatar to present to the second user. The second display avatar will be presented according to one or more features associated with the first user. Examples of the second condition include any of the examples listed above for one or more conditions or the first condition. For example, the first user can change their position in the real world or the virtual world, and the user role management system can change which avatar represents the first user based on the position change. Similarly, the first user can change their activities in the real world or the virtual world, and the user role management system can change which avatar represents the first user based on the change in the activity. Similarly, different second users may be watching the first user in the real world or the virtual world, and the user role management system can change which avatar represents the first user based on this change in the first user's audience type (viewership).

[0158] In some examples, the user role management system is configured and capable of generating a display avatar before outputting the display avatar for presentation to the second user. Generating the display avatar may include providing one or more inputs associated with the first user to a trained machine learning (ML) model, which generates the display avatar based on the one or more inputs associated with the first user. Examples of trained ML models include (multiple) trained ML model 295, (multiple) trained ML model 740, NN 800, or a combination thereof. Examples of one or more inputs include identity input 710 and / or expression input 720.

[0159] In some examples, the user character management system is configured and capable of recognizing a facial expression of the first user and, before outputting the display avatar for presentation to the second user, modifying the display avatar to apply the facial expression to the display avatar. For example, a facial expression may be recognized as part of expression input 720, and (multiple) trained ML models 740 may use the expression input to modify the display avatar (e.g., avatar 715) to provide the facial expression to the display avatar to generate a combined face 730.

[0160] In some examples, the user character management system is configured and capable of recognizing a head pose of a first user and, before outputting the display avatar for presentation to a second user, modifying the display avatar to apply the head pose to the display avatar. The head pose can be applied similarly to facial expressions (e.g., using (multiple) trained ML models 740), or can be mimicked (e.g., matching the head pose) by appropriately rendering and / or rotating the display avatar based on the head pose. The head pose can include head position, head orientation (e.g., pitch, yaw, and / or roll), facial expression, or a combination thereof.

[0161] In some examples, the user character management system is configured and capable of identifying a lighting condition for the first user and, before outputting the display avatar for presentation to the second user, modifying the display avatar to apply the lighting condition to the display avatar. The lighting condition can be applied similarly to facial expressions (e.g., using (multiple) trained ML models 740), or can be mimicked by rendering the display avatar applying similar lighting conditions. Examples of lighting conditions include lighting from light source 540.

[0162] In some examples, the user role management system is configured and capable of causing the display avatar to be displayed using a display. In some examples, the user role management system includes a display (e.g., (multiple) output device 270A, (multiple) output device 270B, output device 1035).

[0163] In some examples, the user character management system is configured and capable of sending at least the display avatar to at least one user device associated with the second user using at least one communication interface. In some examples, the user character management system includes a communication interface (e.g., (multiple) output device 270A, (multiple) output device 270B, output device 1035, communication interface 1040). Examples of user devices associated with the second user include image capture and processing system 100, user device 210, user device 220, HMD 310, mobile phone 410, computing system 1000, or a combination thereof.

[0164] In some examples, the user role management system is or includes a head mounted display (HMD) 310, a mobile phone 410, or a wireless communication device. In some examples, the user role management system is configured and capable of outputting a display avatar for presentation to the second user at least in part by sending the display avatar (e.g., using communication interface 1040) to a user device associated with the second user. In some examples, the user role management system may include one or more network servers (e.g., server(s) 205, computing system 1300).

[0165] In some examples, the user role management system is configured and capable of generating a modified image at least in part by modifying the image based on one or more characteristics of the first user to replace at least a portion of the first user in the image with the display avatar. To output the display avatar for presentation to the second user (as in operation 925), the user role management system is configured and capable of outputting the modified image for presentation to the second user.

[0166] In some examples, the user character management system includes a display, and outputting the display avatar for presentation to the second user includes displaying at least the display avatar (in some examples, as part of the modified image) using the display. Examples of displays include output device(s) 270A-270B, display(s) 340 of HMD 310, display 440 of mobile handset 410, output device 1035, or a combination thereof.

[0167] In some examples, a user role management system may include: a component for identifying a profile associated with a first user, wherein the profile includes data defining multiple avatars that each represent the first user and one or more conditions for displaying corresponding avatars from the multiple avatars; a component for selecting a display avatar from the multiple avatars based on a first condition from the one or more conditions, wherein the first condition is associated with one or more characteristics associated with the first user; and a component for outputting the display avatar for presentation to a second user, wherein the display avatar is presented.

[0168] In some examples, components for identifying a profile associated with a first user include an image capture and processing system 100, an image capture device 105A, an image processing device 105B, an image processor 150, an ISP 154, a host processor 152, an image sensor 130, (multiple) servers 205, a user device 210, a user device 220, (multiple) sensors 230, a face detector 240, a profile recognizer 245, a data store 290, (multiple) trained ML models 295, a first camera 330A, a second camera 330B, a third camera 330C, a fourth camera 330D, a first camera 430A, a second camera 430B, a third camera 430C, a fourth camera 430D, an image sensor that captures an image 510, (multiple) trained ML models 740, a NN 800, an input device 1045, or a combination thereof.

[0169] In some examples, components for selecting an avatar to display based on a first condition include (multiple) servers 205, user device 210, user device 220, profile identifier 245, profile avatar identifier 250, condition detector 255, avatar processor 260, data storage 290, (multiple) trained ML model 295, profile 630, condition 635, (multiple) trained ML model 740, NN 800, or a combination thereof.

[0170] In some examples, components for outputting a display avatar for presentation to a second user include server(s) 205, user device 210, user device 220, profile avatar recognizer 250, avatar processor 260, output device(s) 270A, output device(s) 270B, data storage 290, HMD 310, display(s) 340, headphones 335, display 440, speakers 435A-435B, trained ML model(s) 295, trained ML model(s) 740, NN 800, output device 1035, communication interface 1040, or a combination thereof.

[0171] In some examples, the processes described herein (e.g., Figure 1 , Figure 2A -2B, Figure 5A-Figure 5C , Figure 6A-6B , Figure 7 , Figure 8 The corresponding process Fig. 9 In some examples, the processes described herein may be performed by the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200A, the imaging system 200B, the server(s) 205, the user device 210, the user device 220, the HMD 310, the mobile phone 410, Figure 5A-Figure 5C The process 900 is performed by (multiple) imaging systems, head mounted device 610, (multiple) trained ML models 740, neural network 800, user role management system of execution process 900, computing system 1000, processor 1010, or a combination thereof.

[0172] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a networked watch or smart watch, or other wearable device), a server computer, an automated vehicle or automated vehicle computing device, a robotic device, a television, and / or any other computing device with resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other (multiple) components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other (multiple) components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0173] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform the various operations described herein.

[0174] The processes described herein are illustrated as logical flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that perform the operations when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0175] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors by hardware or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0176] Fig.10 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.10 An example of a computing system 1000 is illustrated, which may be, for example, any computing device constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1005. The connection 1005 may be a physical connection using a bus, or a direct connection to the processor 1010, such as in a chipset architecture. The connection 1005 may also be a virtual connection, a network connection, or a logical connection.

[0177] In some aspects, computing system 1000 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality described for that component. In some aspects, a component may be a physical device or a virtual device.

[0178] The example system 1000 includes at least one processing unit (CPU or processor) 1010 and connections 1005 that couple various system components including system memory 1015, such as read-only memory (ROM) 1020 and random access memory (RAM) 1025, to the processor 1010. The computing system 1000 may include a cache 1012 of high-speed memory that is directly connected to the processor 1010, close to the processor 1110, or integrated as part of the processor 1110.

[0179] Processor 1010 may include any general purpose processor and hardware or software services, such as services 1032, 1034, and 1036 stored in storage device 1030, that are configured to control processor 1010 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1010 may essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0180] To enable user interaction, the computing system 1000 includes an input device 1045, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1000 can also include an output device 1035, which can be one or more of a plurality of output mechanisms. In some cases, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 1000. The computing system 1000 can include a communication interface 1040, which can generally control and manage user input and system output. The communication interface can perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those using audio jacks / plugs, microphone jacks / plugs, universal serial bus (USB) ports / plugs, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1002.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1040 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1000 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), my country's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operation on any particular hardware configuration, so the basic features of this article can be easily replaced with improved hardware or firmware configurations as they are developed.

[0181] The storage device 1030 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic cassette, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk read-only memory (CD-ROM) disc, a rewritable compact disk (CD) disc, a digital video disk (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0182] Storage device 1030 may include software services, servers, services, etc. that enable the system to perform functions when code defining such software is executed by processor 1010. In some aspects, hardware services that perform a particular function may include software components stored in a computer-readable medium that are combined with necessary hardware components such as processor 1010, connection 1005, output device 1035, etc. to perform the function.

[0183] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying (multiple) instructions and / or data. Computer-readable media may include non-transitory media that can store data, and does not include carrier waves and / or temporary electronic signals that are propagated wirelessly or through wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent any combination of processes, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent in any suitable manner (including memory sharing, message passing, token passing, network sending, etc.).

[0184] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0185] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those of ordinary skill in the art that these aspects can be practiced without these specific details. For the sake of explanation, in some cases, the present technology may be presented as including a separate functional block, which includes a method embodied in software or a combination of hardware and software and a functional block including a device, device component, step or routine. In addition to the components shown in the figure and / or described herein, other components may also be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure these aspects in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid obscuring these aspects.

[0186] Various aspects may be described above as processes or methods, which are depicted as flow charts, flowcharts, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0187] The process and method according to the above-mentioned example can be implemented using computer executable instructions stored in or otherwise obtained from a computer readable medium. These instructions may include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Some of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions (such as assembly language, firmware, source code, etc.). Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the example include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0188] The equipment according to these public implementation processes and methods may include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and may take any of various form factors. When implemented in software, firmware, middleware or microcode, the program code or code segment (e.g., computer program product) that performs the necessary tasks may be stored in a computer-readable or machine-readable medium. (Multiple) processors may perform the necessary tasks. Typical form factor examples include laptop computers, smart phones, mobile phones, tablet computer devices or other small personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. By further example, this function may also be implemented on circuit boards between different chips, or implemented on different processes performed in a single device.

[0189] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0190] In the foregoing description, various aspects of the present application are described with reference to the specific aspects of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it should be understood that the concepts of the present disclosure may be embodied and applied differently in other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the above-mentioned application may be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, these aspects may be used in any number of environments and applications outside those described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in alternative aspects, these methods may be performed in an order different from the described order.

[0191] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0192] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor, or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0193] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or that communicates with another component (e.g., connected to other components via a wired or wireless connection, and / or other suitable communication interfaces).

[0194] "At least one of" a set and / or "one or more" in a set as recited in claim language or other language indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, "at least one of A and B" recited in claim language means A, B, or A and B. In another example, "at least one of A, B, and C" recited in claim language means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" in a set does not limit the set to the items listed in the set. For example, "at least one of A and B" recited in claim language may mean A, B, or A and B, and may additionally include items not listed in the A and B sets.

[0195] The various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functionality. Whether these functions are implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the application.

[0196] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices (such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium, which includes a program code, which includes instructions for executing one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer (such as a propagated signal or wave).

[0197] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor can be a microprocessor, but in an alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein. In addition, in some aspects, the functions described herein may be provided in a dedicated software module or a hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0198] Illustrative aspects of the disclosure include:

[0199] Aspect 1: A device for user role management, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: identify a profile associated with a first user, wherein the profile comprises data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar among the plurality of avatars; select a display avatar among the plurality of avatars based on a first condition among the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and output a display avatar for presentation to a second user, wherein the display avatar is presented based on the one or more features associated with the first user.

[0200] Aspect 2: An apparatus according to Aspect 1, wherein the one or more characteristics associated with the first user include the identity of a second user to whom an avatar will be presented, wherein the first condition corresponds to whether the identity of the second user is identified in a predetermined data structure corresponding to the first user.

[0201] Aspect 3: An apparatus according to any one of Aspects 1 to 2, wherein one or more characteristics associated with the first user include a relationship category between a second user to whom a display avatar will be presented and the first user, wherein the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user.

[0202] Aspect 4: An apparatus according to any of Aspects 1 to 3, wherein the one or more characteristics associated with the first user include the location of the first user in the environment, wherein the first condition corresponds to whether the location of the first user in the environment falls within at least one of a predetermined area, a predetermined area type, or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user.

[0203] Aspect 5: An apparatus according to any one of Aspects 1 to 4, wherein the one or more characteristics associated with the first user include an activity performed by the first user, wherein the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user.

[0204] Aspect 6: An apparatus according to any one of Aspects 1 to 5, wherein, in order to identify the profile associated with the first user, the one or more processors are configured to identify the profile associated with the first user based on one or more communications from a user device associated with the first user.

[0205] Aspect 7: An apparatus according to any one of Aspects 1 to 6, wherein one or more processors are configured to: identify a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying corresponding avatars in the second plurality of avatars; select a second display avatar from the second plurality of avatars based on a second condition in the second set of one or more conditions, wherein the second condition is associated with a second set of one or more characteristics associated with the second user; and output a second display avatar for presentation to the second user, wherein the second display avatar is presented based on the second set of one or more characteristics associated with the third user.

[0206] Aspect 8: An apparatus according to any one of Aspects 1 to 7, wherein one or more processors are configured to: identify a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying corresponding avatars in the second plurality of avatars; select a second display avatar from the second plurality of avatars based on none of the conditions in the second set of one or more conditions being met; and output the second display avatar for presentation to the second user.

[0207] Aspect 9: An apparatus according to any of Aspects 1 to 8, wherein one or more processors are configured to: receive an image of an environment; detect at least a portion of a first user in the image, wherein the one or more processors are configured to identify a profile associated with the first user based on at least a portion of the first user detected in the image; and generate a modified image at least in part by modifying the image based on one or more characteristics of the first user to replace at least a portion of the first user in the image with a display avatar, wherein in order to output the display avatar for presentation to a second user, the one or more processors are configured to output the modified image for presentation to the second user.

[0208] Aspect 10: The apparatus according to aspect 9, wherein at least a portion of the environment in the image includes one or more virtual elements.

[0209] Aspect 11: The apparatus according to any one of aspects 9 to 10, wherein at least a portion of the environment image is captured by an image sensor.

[0210] Aspect 12: An apparatus according to any of Aspects 9 to 11, wherein, in order to generate a modified image, one or more processors are configured to use a displayed avatar and at least a portion of a first user in the image as input to a trained machine learning model, which modifies the image based on one or more characteristics of the first user to use the displayed avatar to replace at least a portion of the first user.

[0211] Aspect 13: The apparatus according to any one of aspects 9 to 12, wherein the portion of the first user comprises one or more facial features of the first user.

[0212] Aspect 14: The apparatus of aspect 13, wherein, to identify the profile associated with the first user, the one or more processors are configured to use facial recognition to identify the identity of one or more facial features of the first user in the image.

[0213] Aspect 15: The apparatus according to any one of aspects 1 to 14, wherein the one or more processors are configured to: modify the display avatar based on one or more display preferences associated with the second user before outputting the display avatar for presentation to the second user.

[0214] Aspect 16: An apparatus according to any one of Aspects 1 to 15, wherein one or more processors are configured to: select a second display avatar from multiple avatars based on a change in one or more conditions from a first condition to a second condition; and transition from outputting the display avatar for presentation to the second user to outputting a second display avatar for presentation to the second user, wherein the second display avatar is presented based on one or more features associated with the first user.

[0215] Aspect 17: An apparatus according to any of Aspects 1 to 16, wherein one or more processors are configured to: generate a display avatar before outputting the display avatar for presentation to a second user, wherein, in order to generate the display avatar, the one or more processors are configured to provide one or more inputs associated with the first user to a trained machine learning model, and the trained machine learning model generates the display avatar based on the one or more inputs associated with the first user.

[0216] Aspect 18: An apparatus according to any one of Aspects 1 to 17, wherein the one or more processors are configured to: recognize a facial expression of a first user; and modify the displayed avatar to apply the facial expression to the displayed avatar before outputting the displayed avatar for presentation to a second user.

[0217] Aspect 19: An apparatus according to any one of Aspects 1 to 18, wherein the one or more processors are configured to: identify a head gesture of a first user; and modify the displayed avatar to apply the head gesture to the displayed avatar before outputting the displayed avatar for presentation to a second user.

[0218] Aspect 20: An apparatus according to any one of Aspects 1 to 19, wherein the one or more processors are configured to: identify a lighting condition of the first user; and modify the display avatar to apply the lighting condition to the display avatar before outputting the display avatar for presentation to the second user.

[0219] Aspect 21: The apparatus according to any one of aspects 1 to 20, further comprising: a display, wherein the one or more processors are configured to display the avatar using at least the display.

[0220] Aspect 22: The apparatus according to any one of Aspects 1 to 21 further comprises: a communication interface, wherein the one or more processors are configured to send at least the display avatar to at least one user device associated with the second user using at least the communication interface.

[0221] Aspect 23: The apparatus according to any one of aspects 1 to 22, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

[0222] Aspect 24: An apparatus according to any one of Aspects 1 to 23, wherein the apparatus comprises one or more network servers, wherein one or more processors are configured to output a display avatar for presentation to a second user at least in part by sending the display avatar to a user device associated with the second user.

[0223] Aspect 25: A method for user role management, the method comprising: identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar among the plurality of avatars; selecting a display avatar among the plurality of avatars based on a first condition among the one or more conditions, wherein the first condition is associated with one or more characteristics associated with the first user; and outputting the display avatar for presentation to a second user, wherein the display avatar is presented based on the one or more characteristics associated with the first user.

[0224] Aspect 26: A method according to Aspect 25, wherein the one or more characteristics associated with the first user include the identity of a second user to whom the avatar will be presented, wherein the first condition corresponds to whether the identity of the second user is identified in a predetermined data structure corresponding to the first user.

[0225] Aspect 27: A method according to any one of Aspects 25 to 26, wherein one or more characteristics associated with the first user include a relationship category between a second user to whom a display avatar is to be presented and the first user, wherein the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user.

[0226] Aspect 28: A method according to any one of Aspects 25 to 27, wherein the one or more characteristics associated with the first user include the location of the first user in the environment, wherein the first condition corresponds to whether the location of the first user in the environment falls within at least one of a predetermined area, a predetermined area type, or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user.

[0227] Aspect 29: A method according to any one of Aspects 25 to 28, wherein the one or more characteristics associated with the first user include an activity performed by the first user, wherein the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user.

[0228] Aspect 30: The method according to any one of aspects 25 to 29, wherein identifying the profile associated with the first user comprises identifying the profile associated with the first user based on one or more communications from a user device associated with the first user.

[0229] Aspect 31: The method according to any one of Aspects 25 to 30 further includes: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying corresponding avatars in the second plurality of avatars; selecting a second display avatar from the second plurality of avatars based on a second condition in the second set of one or more conditions, wherein the second condition is associated with a second set of one or more characteristics associated with the second user; and outputting a second display avatar for presentation to the second user, wherein the second display avatar will be presented based on the second set of one or more characteristics associated with the third user.

[0230] Aspect 32: The method according to any one of Aspects 25 to 31 further includes: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar in the second plurality of avatars; selecting a second display avatar from the second plurality of avatars based on none of the conditions in the second set of one or more conditions being met; and outputting the second display avatar to present to the second user.

[0231] Aspect 33: The method according to any one of Aspects 25 to 32 further includes: receiving an image of an environment; detecting at least a portion of a first user in the image, wherein identifying a profile associated with the first user is based on at least a portion of the first user detected in the image; and generating a modified image at least in part by modifying the image based on one or more characteristics of the first user to replace at least a portion of the first user in the image with a display avatar, wherein outputting the display avatar for presentation to a second user includes outputting the modified image for presentation to the second user.

[0232] Aspect 34: The method according to Aspect 33, wherein at least a portion of the environment in the image includes one or more virtual elements.

[0233] Aspect 35: The method according to any one of aspects 33 to 34, wherein at least a portion of the environment image is captured by an image sensor.

[0234] Aspect 36: A method according to any one of Aspects 33 to 35, wherein generating the modified image includes using a displayed avatar and at least a portion of a first user in the image as input to a trained machine learning model, the trained machine learning model modifying the image to replace at least a portion of the first user with the displayed avatar based on one or more characteristics of the first user.

[0235] Aspect 37: The method according to any one of aspects 33 to 36, wherein the portion of the first user includes one or more facial features of the first user.

[0236] Aspect 38: The method of Aspect 37, wherein identifying the profile associated with the first user comprises using facial recognition to identify the identity of one or more facial features of the first user in the image.

[0237] Aspect 39: The method according to any one of aspects 25 to 38, further comprising: before outputting the display avatar for presentation to the second user, modifying the display avatar based on one or more display preferences associated with the second user.

[0238] Aspect 40: The method according to any one of Aspects 25 to 39 further includes: selecting a second display avatar from multiple avatars based on a change in one or more conditions from a first condition to a second condition; and transitioning from outputting the display avatar for presentation to the second user to outputting a second display avatar for presentation to the second user, wherein the second display avatar is presented based on one or more features associated with the first user.

[0239] Aspect 41: The method according to any one of Aspects 25 to 40 further includes: generating a display avatar before outputting the display avatar for presentation to a second user, wherein generating the display avatar includes providing one or more inputs associated with the first user to a trained machine learning model, and the trained machine learning model generates the display avatar based on the one or more inputs associated with the first user.

[0240] Aspect 42: The method according to any one of Aspects 25 to 41 further includes: recognizing a facial expression of the first user; and modifying the display avatar to apply the facial expression to the display avatar before outputting the display avatar for presentation to the second user.

[0241] Aspect 43: The method according to any one of Aspects 25 to 42 further includes: identifying a head gesture of the first user; and modifying the display avatar to apply the head gesture to the display avatar before outputting the display avatar for presentation to the second user.

[0242] Aspect 44: The method according to any one of aspects 25 to 43 further includes: identifying a lighting condition of the first user; and modifying the display avatar to apply the lighting condition to the display avatar before outputting the display avatar for presentation to the second user.

[0243] Aspect 45: The method according to any one of aspects 25 to 44, further comprising: causing the avatar to be displayed using a display.

[0244] Aspect 46: The method according to any one of aspects 25 to 45, further comprising: causing the display avatar to be sent to at least one user device associated with the second user using at least one communication interface.

[0245] Aspect 47: A method according to any one of aspects 25 to 46, wherein an apparatus is configured to perform the method, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

[0246] Aspect 48: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to: identify a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar from the plurality of avatars; select a display avatar from the plurality of avatars based on a first condition from among the one or more conditions, wherein the first condition is associated with one or more characteristics associated with the first user; and output a display avatar for presentation to a second user, wherein the display avatar is presented based on the one or more characteristics associated with the first user.

[0247] Aspect 49: The non-transitory computer-readable medium according to Aspect 48, further comprising operations according to any one of Aspects 2 to 24 and / or any one of Aspects 26 to 47.

[0248] Aspect 50: An apparatus for image processing, the apparatus comprising: a component for identifying a profile associated with a first user, wherein the profile comprises data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar among the plurality of avatars; a component for selecting a display avatar among the plurality of avatars based on a first condition among the one or more conditions, wherein the first condition is associated with one or more features associated with the first user; and a component for outputting a display avatar for presentation to a second user, wherein the display avatar is presented based on one or more features associated with the first user.

[0249] Aspect 51: The apparatus according to aspect 50, further comprising means for performing the operations according to any one of aspects 2 to 24 and / or any one of aspects 26 to 47.

Claims

1. A device for user role management, the device comprising: at least one memory; as well as one or more processors coupled to the at least one memory, the one or more processors configured to: identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar of the plurality of avatars; selecting a display avatar from among the plurality of avatars based on a first condition among the one or more conditions, wherein the first condition is associated with one or more characteristics associated with the first user; and The display avatar is output for presentation to a second user, wherein the display avatar is to be presented according to the one or more characteristics associated with the first user.

2. The device according to claim 1, wherein: The one or more characteristics associated with the first user include an identity of the second user to whom the display avatar is to be presented, wherein the first condition corresponds to whether the identity of the second user is recognized in a predetermined data structure corresponding to the first user.

3. The device according to claim 1, wherein: The one or more characteristics associated with the first user include a relationship category between the second user to whom the display avatar is to be presented and the first user, wherein the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user.

4. The device according to claim 1, wherein: The one or more characteristics associated with the first user include a location of the first user in an environment, wherein the first condition corresponds to whether the location of the first user in the environment falls within at least one of a predetermined area, a predetermined area type, or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user.

5. The device according to claim 1, wherein: The one or more characteristics associated with the first user include an activity performed by the first user, wherein the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user.

6. The device according to claim 1, wherein: To identify the profile associated with the first user, the one or more processors are configured to identify the profile associated with the first user based on one or more communications from a user device associated with the first user.

7. The device according to claim 1, wherein: The one or more processors are configured to: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar of the second plurality of avatars; selecting a second displayed avatar from the second plurality of avatars based on a second condition from the second set of one or more conditions, wherein the second condition is associated with a second set of one or more characteristics associated with the second user; and The second display avatar is output for presentation to the second user, wherein the second display avatar is to be presented according to the second set of one or more characteristics associated with the third user.

8. The device according to claim 1, wherein: The one or more processors are configured to: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar of the second plurality of avatars; selecting a second display avatar from the second plurality of avatars based on none of the second set of one or more conditions being satisfied; and The second display avatar is output to be presented to the second user.

9. The device according to claim 1, wherein: The one or more processors are configured to: receiving an image of the environment; detecting at least a portion of the first user in the image, wherein the one or more processors are configured to identify the profile associated with the first user based on detecting at least a portion of the first user in the image; and A modified image is generated at least in part by modifying the image based on the one or more characteristics of the first user to replace at least a portion of the first user in the image with the display avatar, wherein in order to output the display avatar for presentation to the second user, the one or more processors are configured to output the modified image for presentation to the second user.

10. The device according to claim 9, wherein: At least a portion of the environment in the image includes one or more virtual elements.

11. The device according to claim 1, wherein: The one or more processors are configured to: Prior to outputting the display avatar for presentation to the second user, the display avatar is modified based on one or more display preferences associated with the second user.

12. The device according to claim 1, wherein: The one or more processors are configured to: selecting a second display avatar from among the plurality of avatars based on a change in the one or more conditions from the first condition to a second condition; as well as The method further comprises: switching from outputting the display avatar for presentation to the second user to outputting the second display avatar for presentation to the second user, wherein the second display avatar is to be presented according to the one or more characteristics associated with the first user.

13. The device according to claim 1, wherein: The one or more processors are configured to: The display avatar is generated before being output for presentation to the second user, wherein, to generate the display avatar, the one or more processors are configured to provide one or more inputs associated with the first user to a trained machine learning model, and the trained machine learning model generates the display avatar based on the one or more inputs associated with the first user.

14. The device according to claim 1, wherein: The one or more processors are configured to: recognizing a facial expression of the first user; and Before outputting the display avatar for presentation to the second user, the display avatar is modified to apply the facial expression to the display avatar.

15. The apparatus according to claim 1, further comprising: A display, wherein the one or more processors are configured to cause the display avatar to be displayed using at least the display.

16. The apparatus according to claim 1, further comprising: A communication interface, wherein the one or more processors are configured to send at least the display avatar to at least one user device associated with the second user using at least the communication interface.

17. The device according to claim 1, wherein: The apparatus includes at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

18. The device according to claim 1, wherein: The apparatus comprises one or more network servers, wherein the one or more processors are configured to output the display avatar for presentation to the second user at least in part by transmitting the display avatar to a user device associated with the second user.

19. A method for user role management, the method comprising: identifying a profile associated with a first user, wherein the profile includes data defining a plurality of avatars each representing the first user and one or more conditions for displaying a corresponding avatar of the plurality of avatars; selecting a display avatar from among the plurality of avatars based on a first condition among the one or more conditions, wherein the first condition is associated with one or more characteristics associated with the first user; and The display avatar is output for presentation to a second user, wherein the display avatar is to be presented according to the one or more characteristics associated with the first user.

20. The method according to claim 19, wherein: The one or more characteristics associated with the first user include an identity of the second user to whom the display avatar is to be presented, wherein the first condition corresponds to whether the identity of the second user is recognized in a predetermined data structure corresponding to the first user.

21. The method according to claim 19, wherein: The one or more characteristics associated with the first user include a relationship category between the second user to whom the display avatar is to be presented and the first user, wherein the first condition corresponds to whether the relationship category is identified in a predetermined data structure corresponding to the first user.

22. The method according to claim 19, wherein: The one or more characteristics associated with the first user include a location of the first user in an environment, wherein the first condition corresponds to whether the location of the first user in the environment falls within at least one of a predetermined area, a predetermined area type, or a predetermined environment type in the environment identified in a predetermined data structure corresponding to the first user.

23. The method according to claim 19, wherein: The one or more characteristics associated with the first user include an activity performed by the first user, wherein the first condition corresponds to whether the activity is identified in a predetermined data structure corresponding to the first user.

24. The method of claim 19, further comprising: identifying a second profile associated with a third user, wherein the second profile includes data defining a second plurality of avatars each representing the third user and a second set of one or more conditions for displaying a corresponding avatar of the second plurality of avatars; selecting a second displayed avatar from the second plurality of avatars based on a second condition from the second set of one or more conditions, wherein the second condition is associated with a second set of one or more characteristics associated with the second user; and The second display avatar is output for presentation to the second user, wherein the second display avatar is to be presented according to the second set of one or more characteristics associated with the third user.

25. The method of claim 19, further comprising: Prior to outputting the display avatar for presentation to the second user, the display avatar is modified based on one or more display preferences associated with the second user.

26. The method of claim 19, further comprising: selecting a second display avatar from among the plurality of avatars based on a change in the one or more conditions from the first condition to a second condition; as well as The method further comprises: switching from outputting the display avatar for presentation to the second user to outputting the second display avatar for presentation to the second user, wherein the second display avatar is to be presented according to the one or more characteristics associated with the first user.

27. The method of claim 19, further comprising: The display avatar is generated before being output for presentation to the second user, wherein generating the display avatar comprises providing one or more inputs associated with the first user to a trained machine learning model, and the trained machine learning model generates the display avatar based on the one or more inputs associated with the first user.

28. The method of claim 19, further comprising: recognizing a facial expression of the first user; as well as Before outputting the display avatar for presentation to the second user, the display avatar is modified to apply the facial expression to the display avatar.

29. The method of claim 19, further comprising: The display avatar is displayed using a display.

30. The method of claim 19, further comprising: The display avatar is caused to be sent to at least one user device associated with the second user using at least one communication interface.

Images