System and method for privacy management image processing

Abstract

Systems and techniques for processing one or more images are described. For example, an imaging system may receive an image of an environment and detect a face of a person in the image. The imaging system may identify profile data associated with the person and retrieve a synthesized face based on the profile data. The imaging system may generate a modified image of the environment at least in part by modifying the image to replace the person's face with the synthesized face for the person. The imaging system may also output a modified image of the environment. In some examples, an imaging system generates a modified image using a trained machine learning model.

Description

Technical Field

[0001] The present application relates to image processing. For example, aspects of the present application relate to systems and techniques for image processing to modify an image depicting an environment of a person to replace the person's real face in the image with a synthesized face corresponding to a profile of the person. Background Art

[0002] A network-based interactive system allows users to interact with each other over a network, in some cases even when those users are geographically far from each other. A network-based interactive system may include video conferencing technology. In a video conference, each user is connected by capturing the user's video and / or audio and transmitting the video and / or audio to a user device of other users in the video conference, so that each of the users in the video conference can see and hear each other. A network-based interactive system may include a network-based multiplayer game, such as a massively multiplayer online (MMO) game. A network-based interactive system may include extended reality (XR) technology, such as virtual reality (VR) or augmented reality (AR). At least a portion of the XR environment displayed to a user of an XR device may be virtual, including in some examples representations of other users with whom the user can interact in the XR environment.

[0003] In some examples, the network-based interactive system can use a camera to obtain image data of a user and / or a portion of a real-world environment that the user is in. In some examples, the network-based interactive system transmits the image data to other users. However, the transmission of image data, which may include representations of the user and / or other people in the environment, increases privacy issues because some of those people may not want their images captured and / or shared. Summary of the invention

[0004] In some examples, systems and techniques for image processing are described. An imaging system receives an image of an environment and detects a face of a person in the image. The imaging system identifies a profile associated with the person, for example based on facial recognition and / or communications from the person's device. The imaging system retrieves a synthesized face that corresponds to the person and is different from the person's face from the person's profile. In some examples, the imaging system generates the synthesized face using a trained machine learning model and stores the synthesized face in the profile prior to the retrieval. The imaging system generates a modified image at least in part by modifying the image to replace the person's face with the synthesized face for the person, and outputs the modified image. In some examples, the imaging system generates the modified image using a trained machine learning model, for example to ensure that the synthesized face looks realistic on the rest of the person's body as depicted in the image and blends into the rest.

[0005] In one example, an apparatus for processing one or more images is provided. The apparatus includes at least one memory and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory. The at least one processor is configured to: receive an image of an environment; detect a face of a person in the image; identify profile data associated with the person; retrieve a synthesized face based on the profile data; generate a modified image of the environment at least in part by modifying the image to replace the face of the person with a synthesized face for the person; and output the modified image of the environment.

[0006] In another example, a method of processing one or more images is provided. The method includes: receiving an image of an environment; detecting a face of a person in the image; identifying profile data associated with the person; retrieving a synthesized face based on the profile data; generating a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; and outputting the modified image of the environment.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which, when executed by one or more processors, causes the one or more processors to: receive an image of an environment; detect a face of a person in the image; identify profile data associated with the person; retrieve a synthesized face based on the profile data; generate a modified image of the environment at least in part by modifying the image to replace the face of the person with a synthesized face for the person; and output the modified image of the environment.

[0008] In another example, an apparatus for image processing is provided. The apparatus includes: means for receiving an image of an environment; means for detecting a face of a person in the image; means for identifying profile data associated with the person; means for retrieving a synthesized face based on the profile data; means for generating a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; and means for outputting the modified image of the environment.

[0009] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or a component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes one or more cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Exemplary aspects of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples;

[0014] Figure 2A is a block diagram illustrating an example architecture of an imaging process performed by an imaging system having one or more servers and two user devices according to some examples;

[0015] Figure 2B is a block diagram illustrating an example architecture of an imaging process performed by an imaging system having two user devices according to some examples;

[0016] Figure 3A is a perspective view illustrating a head mounted display (HMD) used as part of an imaging system according to some examples;

[0017] Figure 3B is an example based on some examples Figure 3AA perspective view of a head mounted display (HMD) being worn by a user;

[0018] Figure 4A is a perspective view illustrating a front surface of a mobile phone that includes a front-facing camera and can be used as part of an imaging system according to some examples;

[0019] Figure 4B is a perspective view illustrating a rear surface of a mobile phone that includes a rear camera and can be used as part of an imaging system according to some examples;

[0020] Figure 5 is a conceptual diagram illustrating generating a modified image by replacing a user's face with a synthesized face of the user according to some examples;

[0021] Fig. 6A is a conceptual diagram illustrating generation of a modified image by replacing a user's face with a pixelated face of the user according to some examples;

[0022] Figure 6B is a conceptual diagram illustrating generation of a modified image by replacing a user's face with a review block according to some examples;

[0023] Figure 6C is a conceptual diagram illustrating generation of a modified image by replacing a user's face with an avatar face for the user according to some examples;

[0024] Fig.6D is a conceptual diagram illustrating generation of a modified image by using image inpainting instead of a user according to some examples;

[0025] Figure 7 is a conceptual diagram illustrating generation of a synthesized face based on features of multiple input faces according to some examples;

[0026] Fig. 8A is a conceptual diagram illustrating a first user using a head mounted device to view an interactive environment including a second user, the second user's face being displayed to the first user through the head mounted device based on the second user's privacy settings and a relationship indicator indicating a family relationship between the first user and the second user, according to some examples;

[0027] Figure 8B is a conceptual diagram illustrating a first user using a head mounted device to view an interactive environment including a second user, a synthesized face of the second user being displayed to the first user through the head mounted device based on the second user's privacy settings and a relationship indicator indicating a lack of a familial relationship between the first user and the second user, according to some examples;

[0028] Fig. 9is a conceptual diagram illustrating combining identity input and expression input according to some examples to generate a combined face having an identity of the identity input and an expression of the expression input;

[0029] Fig.10 is a block diagram illustrating an example of a neural network that may be used for image processing operations according to some examples;

[0030] Fig.11 is a flow chart illustrating an imaging process according to some examples; and

[0031] Fig.12 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0032] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0033] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0034] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop (aperture number), shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor used to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes in contrast, brightness, saturation, sharpness, level, curve or color. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) used to process one or more image frames captured by an image sensor.

[0035] An extended reality (XR) system or device may provide virtual content to a user and / or may combine a real-world view of a physical environment (scene) and a virtual environment (including virtual content). The XR system facilitates user interaction with such a combined XR environment. The real-world view may include real-world objects (also referred to as physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world objects or physical objects. The XR system or device may facilitate interaction with different types of XR environments (e.g., a user may use an XR system or device to interact with an XR environment). The XR system may include a VR system that facilitates interaction with a virtual reality (VR) environment, an AR system that facilitates interaction with an augmented reality (AR) environment, an MR system that facilitates interaction with a mixed reality (MR) environment, and / or other XR systems. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses, and the like. In some cases, the XR system may track a user's location (e.g., a user's hands and / or fingertips) to allow the user to interact with items of virtual content.

[0036] Video conferencing is a network-based technology that allows multiple users who may be in different locations to use corresponding user devices to connect to a video conference through a network, and each user device generally includes a display and a camera. In a video conference, each camera of each user device captures image data representing the user who is using the user device, and transmits the image data to other user devices connected to the video conference to display on the displays of other users using those other user devices. At the same time, the user device displays image data representing other users in the video conference, and the image data is captured by the corresponding cameras of other user devices used by these other users to connect to the video conference. When users are in different locations, a group of users can use video conferencing to speak face to face virtually. Despite travel restrictions (such as those related to the pandemic), video conferencing can be a valuable way for users to meet each other virtually. Video conferencing can be performed using user devices connected to each other by one or more servers in some cases. In some examples, the user device may include a laptop, a phone, a tablet computer, a mobile phone, a video game console, a vehicle computer, a desktop computer, a wearable device, a television, a media center, an XR system, or other computing devices discussed herein.

[0037] A network-based interactive system allows users to interact with each other over a network, in some cases even when those users are geographically distant from each other. A network-based interactive system may include video conferencing technology, such as those described above. In a video conference, each user is connected by a user device that captures the user's video and / or audio and transmits the video and / or audio to the other users in the video conference, so that each of the users in the video conference can see and hear each other. A network-based interactive system may include extended reality (XR) technology, such as those described above. At least a portion of the XR environment displayed to a user of the XR device may be virtual, including in some examples representations of other users with whom the user can interact in the XR environment. A network-based interactive system may include a network-based multiplayer game, such as a massively multiplayer online (MMO) game. A network-based interactive system may include a network-based interactive environment, such as a "metaverse" environment.

[0038] In some examples, the web-based interactive system can use a camera to obtain image data of a user and / or a portion of the real-world environment in which the user is located. In some examples, the web-based interactive system transmits the image data to other users. However, the transmission of image data that may include representations of users and / or other people in the environment increases privacy issues, because some of those people may not want to be captured and / or share their images (e.g., their faces and / or other parts of their respective bodies) by the web-based interactive system. For example, the user may be using the web-based interactive system outdoors, in a coffee shop, a store, a home, an office, or a school. In each of these cases, the camera of the web-based interactive system may eventually capture image data of the user and other people, such as when those people may eventually walk into the camera's field of view, or the camera's field of view may move (e.g., when the user moves their head while wearing an HMD as a part of the web-based interactive system) to include these people. For some people (such as children), the privacy laws of some countries or regions may prohibit or otherwise regulate the use of such web-based interactive system image capture and / or sharing. In some network-based interactive systems (such as those that include head mounted display (HMD) devices), it may be difficult for a user to control the field of view of a camera to prevent capturing image data of others. In some cases, the user himself / herself may not want to share his / her images using the network-based interactive system, for example, if the user is not getting ready for the day, has a bad hair day, is feeling sick or under the weather, is wearing clothing that they are not willing to share (e.g., pajamas), is eating while using the network-based interactive system, or some combination thereof.

[0039] In some examples, the web-based interactive system may modify the image data to protect the privacy of the user and / or other persons depicted or otherwise represented in the image data. In some examples, the web-based interactive system may blur or pixelate the faces of persons in the images to protect the privacy of the persons (e.g., see Fig. 6A ). In some examples, the web-based interactive system can cover the face of a person in an image (e.g., with a black box or cartoon avatar face) to protect the person's privacy (e.g., see FIG. 6B to FIG. 6C ). In some examples, the web-based interactive system can use image restoration to effectively remove people and / or their faces from images to protect the privacy of the people (see, e.g., Fig.6D ). However, all of these solutions may cause the people and the interactive environment they are in to no longer look realistic, thereby destroying the immersion of the user of the network-based interactive system. For example, if the people in the environment have faces that are blurred, pixelated, covered in black boxes, or covered in cartoon avatar faces such as smiley faces, the image of the environment will not look like the real environment. Similarly, image restoration often produces visual artifacts and may look inconsistent from one video frame to the next, similarly destroying the user's immersion in the environment.

[0040] In some examples, one or more machine learning models (producing one or more trained machine learning models) can be trained using training data of numerous input images of human faces to generate realistic synthesized faces. In some cases, the trained machine learning model can extract features and / or attributes from various faces in the input image, and combine the extracted features and / or attributes in a new way to generate a realistic synthesized face. In some examples, the synthesized face generated by the trained machine learning model can look like a real human face, with the correct features and proportions, but is different from any face in the human face in the training data, and does not represent any real person. In some examples, the trained machine learning model may include one or more generative adversarial networks (GANs) and / or other deep learning (DL)-based generative ML models. In some aspects, in some examples, pre-generated synthetic faces can be used for replacement. Synthesized faces may also be referred to as synthetic faces, artificial faces, replacement faces, secondary faces, or combinations thereof.

[0041] In some examples, systems and techniques for image processing are described. An imaging system receives an image of an environment and detects a face of a person in the image. The imaging system identifies a profile associated with the person, for example based on facial recognition and / or communications from the person's device. The imaging system retrieves a synthesized face that corresponds to the person and is different from the person's face from the person's profile. In some examples, the imaging system generates the synthesized face using a trained machine learning model and stores the synthesized face in the profile prior to the retrieval. The imaging system generates a modified image at least in part by modifying the image to replace the person's face with the synthesized face for the person, and outputs the modified image. In some examples, the imaging system generates the modified image using a trained machine learning model, for example to ensure that the synthesized face looks realistic on the rest of the person's body as depicted in the image and blends into the rest.

[0042] In an illustrative example, a user of an imaging system (e.g., an XR system) may travel to a store, a concert, or another area in an at least partially virtual environment. Other people may be in the area in addition to the user themselves, and pedestrians may pass through the area. However, not all of these users want to publicly display their real faces to all users of the imaging system. For each person that the user sees in the area in the environment, the imaging system generates a synthetic face, for example using a GAN. The imaging system replaces the person's real face with a synthetic face, for example using a trained ML model to replace or superimpose the synthetic face on the person's real face and realistically blend the synthetic face into the rest of the person's body and / or into the environment, for example to take into account the lighting in the environment, the orientation of the person's head, the facial expressions the person is making, etc. The imaging system displays a modified view of the environment to the user of the imaging system. In the illustrative example discussed above, the user of the imaging system can see a modified view of the area in the environment, where other people in the store have synthetic faces. Thus, other people in the interactive environment retain their privacy, but the sense of immersion is not broken for the user of the imaging system because the people in the interactive environment still look like real humans, rather than people with faces that are unnaturally blurred, pixelated, or covered.

[0043] In some examples, the imaging system may generate a synthesized face so that the skin color of the synthesized face corresponds to the natural skin color of the person. This ensures that the skin color of the synthesized face does not conflict with the skin color of other parts of the user's body that may be visible in the image (e.g., hands, arms, legs, neck, feet, etc.). This can ensure that the sense of immersion is not broken. In some examples, the imaging system may change the skin color of other parts of the user's body that are visible in the image (e.g., hands, arms, legs, neck, feet, etc.) to match the skin color of the synthesized face. This can further increase privacy, such as making the user not even aware of the person's original skin color, but may increase the risk of visual artifacts, such as if the imaging system incorrectly classifies an object in the image as a body part of a person (e.g., causing the imaging system to incorrectly modify the color and / or tone of the object), or if the imaging system fails to identify a body part of a person (e.g., causing the imaging system to fail to modify the skin color of the body part).

[0044] In some examples, a profile of a user or other person may include preferences corresponding to how and when their synthetic face "disguise" is applied. In some examples, the user's preferences may include an approved list (whitelist) of certain users of the interactive environment (e.g., family and / or friends) who are allowed to see their real face, while the user appears to be disguised with a synthetic face (as described above) to anyone else. In some examples, the user's preferences may include a blocked list (blacklist) of certain users of the interactive environment who are not allowed to see their real face (e.g., people they dislike or wish to hide), for whom the user appears to be disguised with a synthetic face (as described above), while the user appears to be their real face to anyone else.

[0045] In some examples, the user's settings may also be based on where the user is in the interactive environment. For example, if the user's settings may include an approved list (whitelist) of certain locations in the interactive environment (e.g., their home in the interactive environment), where anyone visiting is allowed to see their real face, while the user appears disguised with a synthetic face (e.g., as described herein) anywhere else in the interactive environment. In some examples, the user's settings may include a blocked list (blacklist) of certain locations in the interactive environment (e.g., very public areas) where the user is not allowed to show their real face, for which the user appears disguised with a synthetic face (e.g., as described herein), while the user appears with their real face anywhere else in the interactive environment.

[0046] In some examples, each person (user or bystander) in the interactive environment can have a profile. The imaging system can generate a synthesized face for each person (e.g., as described above), which can be stored in the person's profile. The person's settings (if any) can be stored in the profile. In some examples, when a new person is detected, a profile can be created for the person by the imaging system, where the synthesized face corresponds to the person and / or the person's default settings. The person's real face or features extracted from it can be included so that the system can consistently detect, identify and / or track people throughout the interactive environment. In this way, the same person can have the same synthesized face regardless of when and where they are seen. This can preserve immersion for users of the interactive environment, for example, if the user briefly looks away from the person and then looks back, the person will still have the same synthesized face because their profile stores the synthesized face for them, rather than having their face suddenly change to a different synthesized face, which would destroy the immersion. In some examples, the default settings may indicate that the imaging system will always replace a person's face with a corresponding synthesized face (e.g., to maximize privacy), never replace a person's face with a corresponding synthesized face (e.g., to minimize image processing time, energy, and computing resources), or have a predetermined approval list and / or block list that indicates when to replace a person's face with a corresponding synthesized face and / or when not to replace a person's face. In some cases, a new synthesized face may be regenerated periodically (e.g., every week, every day, every few hours, every few minutes) for each person's profile to prevent malicious users from trying to track the person in an interactive environment based on another person's consistent synthesized face. In some examples, a new synthesized face may be regenerated aperiodically for each person's profile. For example, if a person spends a long period of time in an immersive environment (e.g., five hours a day, two hours a day for three consecutive days, etc.), the imaging system may regenerate faces more frequently (e.g., on a particular day, every day for three consecutive days, etc.). In another example of non-periodic regeneration of new synthesized faces, if a person has been inactive for a period of time (e.g., inactive for at least two, three, or other number of days, inactive for a week or more, etc.), the imaging system may regenerate the face less frequently (e.g., once a week, once every two weeks, etc.).

[0047] The imaging systems and techniques described herein provide many technical improvements over existing imaging systems. For example, the imaging systems and techniques described herein provide increased privacy and security for network-based interactive systems and / or other imaging systems, at least in part, by alternate faces (e.g., synthesized faces) for people depicted in image data. Compared to other privacy-enhancing techniques such as face blurring, face pixelation, covering faces with black boxes, covering faces with cartoon avatar faces, image restoration, or combinations thereof, the imaging systems and techniques described herein provide increased immersion and / or do not detract from immersion.

[0048] Various aspects of the application will be described with respect to the accompanying drawings. Figure 1 1 is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of one or more scenes (e.g., images of scene 110). The image capture and processing system 100 can capture independent images (or photos) and / or can capture a video including multiple images (or video frames) in a particular sequence. A lens 115 of the system 100 faces the scene 110 and receives light from the scene 110. The lens 115 bends the light toward the image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130. In some examples, the scene 110 is a scene in an environment. In some examples, the scene 110 is a scene of at least a portion of a user. For example, the scene 110 can be a scene of one or both of the user's eyes and / or at least a portion of the user's face.

[0049] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include control mechanisms other than those illustrated and additional control mechanisms, such as control mechanisms to control analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0050] A focus control mechanism 125B of the control mechanism 120 may obtain a focus setting. In some examples, the focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, the focus control mechanism 125B may adjust the position of the lens 115 relative to the position of the image sensor 130. For example, based on the focus setting, the focus control mechanism 125B may move the lens 115 closer to the image sensor 130 or further away from the image sensor 130 by actuating a motor or a servo, thereby adjusting the focus. In some cases, an additional lens may be included in the system 100, such as one or more micro lenses above each photodiode of the image sensor 130, each of which bends the light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0051] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or f / stop), the duration that the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0052] A zoom control mechanism 125C of the control mechanism 120 may obtain a zoom setting. In some examples, the zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, the zoom control mechanism 125C may control the focal length of a lens element assembly (lens assembly) including the lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses relative to each other. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, the focusing lens may be the lens 115) that first receives light from the scene 110, wherein the light then passes through an afocal zoom system between the focusing lens (e.g., the lens 115) and the image sensor 130 before the light reaches the image sensor 130. An afocal zoom system may, in some cases, include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0053] The image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a particular pixel in the image generated by the image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus may measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes a red filter, a blue filter, and a green filter, where each pixel of an image is generated based on red light data from at least one photodiode covered in a red filter, blue light data from at least one photodiode covered in a blue filter, and green light data from at least one photodiode covered in a green filter. Other types of color filters may use yellow, magenta, and / or cyan (also called "emerald") color filters instead of or in addition to red, blue, and / or green filters. Some image sensors may have no color filters at all, and may instead use different photodiodes (in some cases vertically stacked) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to light of different wavelengths. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0054] In some cases, image sensor 130 may alternatively or additionally include an opaque and / or reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier to amplify the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) to convert the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed for one or more of control mechanisms 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge coupled device (CCD) sensor, an electron multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal oxide semiconductor (CMOS), an N-type metal oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0055] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processors 1210 discussed for computing system 1200. Host processor 152 may be a digital signal processor (DSP) and / or other type of processor. In some implementations, image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes host processor 152 and ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth TM, global positioning system (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In an illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0056] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in random access memory (RAM) 140 and / or 1220, read-only memory (ROM) 145 and / or 1225, cache, memory unit, another storage device, or some combination thereof.

[0057] Various input / output (I / O) devices 160 may be connected to the image processor 150. The I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touch pad, a touch-sensitive surface, a printer, any other output device 1235, any other input device 1245, or some combination thereof. In some cases, subtitles may be input into the image processing device 105B through a physical keyboard or keypad of the I / O device 160, or through a virtual keyboard or keypad of a touch screen of the I / O device 160. The I / O 160 may include one or more ports, jacks, or other connectors that implement a wired connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. The I / O 160 may include one or more wireless transceivers that implement a wireless connection between the system 100 and one or more peripheral devices, through which the system 100 may receive data from the one or more peripheral devices and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0058] In some cases, the image capture and processing system 100 may be a single device. In some cases, the image capture and processing system 100 may be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0059] like Figure 1 As shown, the vertical dotted line will Figure 1 1 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120 and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145 and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0060] The image capture and processing system 100 may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 1202.11 wi-fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some specific implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile phone, a desktop computer, or other computing devices.

[0061] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include electronic circuits or other electronic hardware, and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0062] Figure 2A1 is a block diagram illustrating an example architecture of an imaging process performed by an imaging system 200A having one or more servers 205 and two user devices. Specifically, the imaging system 200A includes one or more servers 205, a user device 210 associated with a user 215, and a user device 220 associated with a user 225. Each of the server 205, the user device 210, and / or the user device 220 may include at least one computing system 1200. Each of the server 205, the user device 210, and / or the user device 220 may include, for example, one or more laptop computers, phones, tablet computers, mobile phones, video game consoles, transportation computers, desktop computers, wearable devices, televisions, media centers, XR systems, head-mounted display (HMD) devices, or other types of computing devices discussed herein, or combinations thereof. In some examples, the user device 210 includes components illustrated and / or described herein as included in the user device 220. In some examples, the user device 210 may perform operations illustrated and / or described herein as performed by the user device 220. In some examples, user device 220 includes components illustrated and / or described herein as included in user device 210. In some examples, user device 220 can perform operations illustrated and / or described herein as performed by user device 210. In some examples, user device 210 and / or user device 220 include components illustrated and / or described herein as included in server 205. In some examples, user device 210 and / or user device 220 can perform operations illustrated and / or described herein as performed by server 205. In some examples, server 205 includes components illustrated and / or described herein as included in user device 210 and / or user device 220. In some examples, server 205 can perform operations illustrated and / or described herein as performed by user device 210 and / or user device 220.

[0063] The imaging system 200A and the corresponding imaging process can be used in a network-based interactive system application, such as those used for video conferencing, extended reality (XR), video games, metaverse environments, or combinations thereof. For example, the user device 210 includes an interactive client application 280A, the user device 220 includes an interactive client application 280B, and the server 205 includes an interactive server application 285. In some examples, the interactive client application 280A and the interactive client application 280B can be client instances of a software application for a network-based interactive system application, such as those used for video conferencing, extended reality (XR), video games, metaverse environments, or combinations thereof. In some examples, the interactive server application 285 can be a server instance of a software application for a network-based interactive system application, such as those used for video conferencing, extended reality (XR), video games, metaverse environments, or combinations thereof. In some examples, the interaction client application 280A, the interaction client application 280B, and / or the interaction server application 285 can generate a virtual environment, virtual elements to be incorporated into a real-world environment (e.g., as represented in image data captured using the sensor 230), or a combination thereof. In some examples, a representation of the user 215 and / or a representation of the user 225 is positioned within and / or is able to move throughout an at least partially virtual environment, where the virtual elements of the environment are generated using the interaction client application 280A, the interaction client application 280B, and / or the interaction server application 285. For example, the at least partially virtual environment can be an environment for a video game, a VR environment, an AR environment, an MR environment, an XR environment, a metaverse environment, a video conferencing environment, a teleconferencing environment, or a combination thereof. Figure 2A 2, a graphic representing interaction client application 280A and interaction client application 280B illustrates a user (e.g., user 215 or user 225) wearing an HMD device (e.g., user device 210 or user device 220) and seeing representations (e.g., virtual representations) of two people (e.g., two people depicted in image 235 and / or modified image 265) in an XR environment (e.g., which may be at least partially virtual). The XR environment may be generated at least in part using interaction client application 280A, interaction client application 280B, and / or interaction server application 285.

[0064] User device 210 of user 215 includes one or more sensors 230. In some examples, sensor 230 includes one or more image sensors or one or more cameras. Image sensor capture may include image data of one or more images, one or more videos, portions thereof, or a combination thereof. In some examples, at least one of sensors 230 may be oriented toward user 215 (e.g., may face user 215), and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of user 215. In some examples, at least one of sensors 230 may be facing away from user 215 (e.g., may be facing away from user 215) and / or toward an environment in which user 215 is located, and thus may capture (e.g., depict or otherwise represent) sensor data (e.g., image data) of at least a portion of the environment. In some examples, sensor data captured by at least one of sensors 230 that is facing away from user 215 and / or toward user 215 may have a field of view (FoV) that includes, is included by, overlaps, and / or otherwise corresponds to a field of view (FoV) of an eye of user 215. In Figure 2A , a graphic representing sensor 230 illustrates sensor 230 as including a camera facing the environment in which two persons are located.

[0065] In some examples, sensor 230 captures sensor data that measures and / or tracks information about various aspects of user 215's body and / or user 215's behavior. In some examples, sensor 230 includes one or more image sensors of one or more cameras facing at least a portion of the user (e.g., at least a portion of the face and / or head of user 215). One or more cameras may include one or more image sensors that capture image data, and the image data includes one or more images of at least a portion of user 215. For example, sensor 230 may include one or more image sensors focused on one or both eyes (and / or eyelids) of user 215, wherein the image sensors of the cameras capture image data of one or both eyes of user 215. One or more image sensors may also be referred to as eye capture sensors. In some implementations, one or more image sensors may capture image data including a series of images over time, and in some examples, the images may be arranged together in a time sequence, such as to form a video. This series of images may depict or otherwise indicate, for example, movement of the user's 215 eyes, pupil dilation, blinking (using eyelids), squinting (using eyelids), glances, gazes, eye moisture levels, optokinetic reflexes or reactions, vestibulo-ocular reflexes or reactions, accommodation reflexes or reactions, other attributes related to the eyes and / or eyelids described herein, or combinations thereof.

[0066] Sensors 230 may include one or more sensors that track information about user 215 and / or the environment, including posture (e.g., position and / or orientation), the body of user 215, and / or the behavior of user 215. For example, sensors 230 may include one or more cameras, image sensors, microphones, heart rate monitors, blood oximeters, biometric sensors, positioning receivers, global navigation satellite system (GNSS) receivers, inertial measurement units (IMUs), accelerometers, gyroscopes, gyrometers, barometers, thermometers, altimeters, depth sensors, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, sound detection and ranging (SODAR) sensors, sound navigation and ranging (SONAR) sensors, time of flight (ToF) sensors, structured light sensors, other sensors discussed herein, or combinations thereof. In some examples, one or more sensors 230 include at least one image capture and processing system 100, image capture device 105A, image processing device 105B, or combinations thereof. In some examples, one or more sensors 230 include at least one input device 1245 of computing system 1200. In some implementations, one or more of sensors 230 can supplement or refine sensor readings from other sensors 230. For example, an inertial measurement unit (IMU), accelerometer, gyroscope, or other sensor can be used to identify the pose (e.g., position and / or orientation) of user device 210 and / or user 215 in an environment, and / or the gaze of user 215 through user device 210.

[0067] Sensors 230 of user device 210 include an image sensor that captures images 235 of the environment (and / or user 215). Figure 2A 2. The graphic representing image 235 illustrates image 235 as depicting an environment with two people therein. User device 210 transmits image 235 to server 205.

[0068] The server 205 receives the image 235. The face detector 240 of the server 205 detects the face of the person depicted in the image 235. The face detector 240 can detect, extract, recognize and / or track features of parts of the face, body, object and / or environment in order to detect the face of the person. In some examples, the face detector 240 detects the face of the person by first detecting the body of the person and then detecting the face based on the expected position of the face within the body structure. In some examples, the face detector 240 detects the face of the person by inputting the image 235 into one or more models of one or more trained machine learning (ML) models 277 discussed herein and receiving an output indicating the position and / or orientation of the face. The trained machine learning (ML) model 277 can be trained (e.g., by the user device 210, the user device 220, the server 205 and / or another device) using training data including images for use by the face detector 240, and the image includes a face whose position and / or orientation is a predetermined face. In some examples, face detector 240 detects a position (e.g., pixel coordinates) of a face within image 235, a position (e.g., 3D coordinates within a 3D volume of the environment) of a face within image 235, an orientation (e.g., pitch, yaw, and / or roll) of a face within image 235 (e.g., along an axis about which rotation is visible in image 235), and / or an orientation (e.g., pitch, yaw, and / or roll) of a face within the environment. For example, the pose (e.g., position and / or orientation) of a face in the environment may be based on a comparison of a distance (e.g., inter-ocular distance) between two features on a face in image 235 with a reference distance (e.g., inter-ocular distance) for a typical human being. In some examples, face detector 240 detects a face of a person using feature detection, feature extraction, feature recognition, feature tracking, object detection, object recognition, object tracking, face detection, face recognition, face tracking, person detection, person recognition, person tracking, classification, or a combination thereof. In some examples, some of sensors 230 face the eyes of user 215, and face detector 240 may detect a face in image 235 based on gaze detection of a gaze of the eyes of user 215. In some examples, user device 210 and / or server 205 may receive one or more communications from a user device of a person in image 235, which face detector 240 may use as an indicator that the face of the person (or more generally, a person) may be present in image 235. Figure 2A 2, a graphic representing a face detector 240 illustrates an image 235 having a bounding box around the face of a person depicted in the image 235, with an enlarged version of the illustrated person's face extending from the box. The face appears to be that of a young man with short hair and a beard.

[0069] Server 205 includes profile identifier 245. Profile identifier 245 retrieves a profile of a person whose face is detected by face detector 240 from data repository 290. If the person does not already have a profile in data repository 290, profile identifier 245 may create (e.g., generate) a profile for the person. In some cases, profile identifier 245 and / or face detector 240 use facial recognition of the person's face and / or person recognition of the person's body to identify an identifier (e.g., name, email address, phone number, mailing address, number, code, etc.) for the person and then query data repository 290 for the identifier to retrieve a profile for the person from data repository 290 (or receive an indicator from data repository 290 that a profile corresponding to the indicator does not yet exist). In some examples, profile identifier 245 and / or face detector 240 may use at least one of trained ML models 277 to perform facial recognition on the face of the person in image 235 to determine an identifier corresponding to the person (and / or the face of the person) for querying data repository 290. Trained machine learning (ML) model 277 may be trained (e.g., by user device 210, user device 220, server 205, and / or another device) for use in facial recognition by face detector 240 and / or profile identifier 245 using training data including images, the images including identifiers for faces being faces for which identifiers are predetermined. In some examples, user device 210 and / or server 205 may receive one or more communications from a user device of the person in image 235, and profile identifier 245 may identify an identifier (e.g., a name, email address, phone number, mailing address, number, code, etc.) corresponding to the person based on the one or more communications from the user device of the person. In some examples, facial recognition may include comparing a face detected in an image by face detector 240 to a data structure of reference faces stored in data repository 290, wherein an identifier is listed for each of the reference faces in data repository 290. Figure 2A Within, a graphic representing a profile identifier 245 illustrates an image having a face of a person detected by the face detector 240 (e.g., a young man with short hair and a beard), a name for the same person ("John Smith"), an email address for the same person ("js@domain.com"), a synthesized face corresponding to the same person, and a profile of the privacy settings for the same person.

[0070] Data repository 290 may include one or more data structures, such as one or more databases, tables, lists, arrays, matrices, heaps, ledgers, distributed ledgers (e.g., blockchain ledgers and / or directed acyclic graph (DAG) ledgers), or combinations thereof. Data repository 290 may store profiles for people, such as profiles for users (e.g., user 215, user 225) and / or other people detected in images (e.g., image 235) captured by sensors (e.g., sensor 230) of user devices (e.g., user device 210, user device 220). A profile for a person may include one or more images of the person (e.g., images of the person's real face), one or more identifiers for the person (e.g., name, email address, phone number, mailing address, number, code, etc.), one or more synthesized faces for the person (e.g., generated by synthesized face generator 255), one or more privacy settings for the person (e.g., privacy settings 835), or combinations thereof. An example of a profile for person "John Smith" is shown in FIG. Figure 2A The profile identifier 245 is shown in the graphic.

[0071] The server 205 includes a profile synthesized face identifier 250 and a synthesized face generator 255. Once the profile identifier 245 identifies a profile for a person whose face is detected in the image 235 by the face detector 240, the profile synthesized face identifier 250 identifies a synthesized face for the person. The synthesized face for the person may be generated by the synthesized face generator 255 and stored in the data repository 290. In some examples, the profile may not yet include a synthesized face for the person, such as if the profile is a new profile, or if a previous synthesized face has been used for a duration longer than an expiration threshold duration and is scheduled to be replaced by a new synthesized face. In some examples, the synthesized face generator 255 may generate a synthesized face for the person in real time as requested by the profile synthesized face identifier 250, the profile identifier 245, the data repository 290, the server 205, the user device 210, and / or the user device 220. In some examples, synthetic face generator 255 may generate a synthetic face for the person in advance and store the synthetic face in data repository 290 along with (or in association with) the person's profile, and profile synthetic face identifier 250 may retrieve the synthetic face from data repository 290 after identifying the profile using profile identifier 245. Figure 2AWithin, a graphic representing profile synthesized facial identifiers 250 illustrates an arrow pointing from an image of a person's face detected by face detector 240 (e.g., a young man named John Smith with short hair and a beard) to a corresponding image of a synthesized face for that person (e.g., the synthesized face looks like an older person with longer hair and a clean-shaven chin without a beard).

[0072] In some examples, synthetic face generator 255 can use at least one of trained ML models 277 to generate a synthetic face for the person. In some examples, trained ML model 277 used by synthetic face generator 255 can be trained (e.g., by user device 210, user device 220, server 205, and / or another system) to generate realistic synthetic faces using training data that includes numerous input images of human faces (e.g., see Figure 7 and / or Fig. 9 ). In some examples, the trained machine learning model can extract features and / or attributes from various faces in the input image, and combine the extracted features and / or attributes in new ways to generate a realistic synthesized face. The features and / or attributes may include, for example, eye shape, eye color, eyelash length, nose shape, nose width, nose length, nose height, mouth shape, mouth size, lip shape, lip size, lip color, tooth arrangement, tooth color, fillers, cheek shape, eyebrow shape, eyebrow color, eyebrow length, eyebrow width, hairstyle, hair color, hair thickness, hair type (e.g., straight or curly), baldness, facial hair style, facial hair color, facial hair thickness, skin color (e.g., skin color), forehead size, face shape, head shape, chin shape, or a combination thereof. In some examples, the synthesized face generated by the trained machine learning model may look like a real human face, with the correct features and proportions, but is different from any of the human faces in the training data and does not represent any real person. In some examples, the trained machine learning model may include one or more generative adversarial networks (GANs) and / or other deep learning (DL) based generative ML models. The trained machine learning (ML) model 277 may be trained (e.g., by user device 210, user device 220, server 205, and / or another device) for use by the synthetic face generator 255 using training data including various input images of faces and synthesized faces pre-generated using various features and / or attributes from those input images. Figure 2AWithin, a graphic illustration of a synthetic face generator 255 illustrates two arrows pointing from two input images of real human faces to a single image of a synthesized face, respectively, the single image of the synthesized face combining features and / or attributes from the two input images of real human faces into a new synthesized face that is different from either of the real human faces in the input images.

[0073] Server 205 includes image modifier 260. Image modifier 260 generates modified image 265 at least in part by modifying image 235 to replace the face of the person in image 235 with a synthesized face for the person (e.g., generated by synthesized face generator 255 and identified by profile synthesized face identifier 250). In some examples, the synthesized face includes a synthesized head, and image modifier 260 generates modified image 265 at least in part by modifying image 235 to replace the head of the person in image 235 with at least a portion of a synthesized head for the person (e.g., generated by synthesized face generator 255 and identified by profile synthesized face identifier 250). In some examples, to generate modified image 265, image modifier 260 superimposes the synthesized face and / or head for the person on the face and / or head of the person in image 235. In some examples, to generate modified image 265, image modifier 260 replaces the face and / or head of the person in image 235 with the synthesized face and / or head for the person.

[0074] In some examples, image modifier 260 uses at least one of trained machine learning models 277 to generate modified image 265. Trained machine learning model 277 can help ensure that the synthesized face looks realistic on and blends into the rest of the person's body as depicted in image 235. Trained machine learning (ML) model 277 (e.g., by user device 210, user device 220, server 205, and / or another device) is trained for use by image modifier 260 using training data, including images of an environment with faces therein, images of a replacement face to replace one of the faces in the image of the environment, and a pre-modified version of the image of the environment with the replacement face replacing the original face.

[0075] In some examples, image modifier 260 realistically blends the synthesized face into the rest of the person's body and / or into the environment in image 235, for example to take into account the lighting in the environment, the head pose (e.g., position and / or orientation) of the person's head, the facial expressions made by the person in image 235, the person's skin color (e.g., skin color), or a combination thereof. For example, in some examples, image modifier 260 may determine the ambient lighting on the original face, and may replicate the same ambient lighting on the synthesized face. The ambient lighting may include the color of the lighting, the intensity of the lighting, the direction of the lighting (e.g., from the left, from the right, from above and / or from below), any shadows cast on the face from other objects in the environment, or a combination thereof. This may ensure that the sense of immersion is not broken. In some examples, image modifier 260 may determine the facial expression (e.g., smiling, laughing, frowning, crying, surprised, etc.) of the face of the person detected in image 235 using face detector 240, and may modify the synthesized face to apply the same facial expression (e.g., as in Fig. 9 In some examples, image modifier 260 may determine a facial expression (e.g., smiling, laughing, frowning, crying, surprised, etc.) of the face of the person detected in image 235 using face detector 240, and may modify the synthesized face to apply the same facial expression, e.g., as in Fig. 9 This may allow user 215 of user device 210 to realistically see the person's mouth move as the person speaks and / or see other facial expressions on the person's face without breaking immersion, while still maintaining the person's privacy and not revealing his / her identity to user 215. Figure 2A Within, the graphic representing image modifier 260 illustrates the environment depicted in image 235 and modified image 265, where the images have a bounding box around a person's face (as in the graphic representing face detector 240) with an enlarged version of the synthesized face of the illustrated person extending from the box.

[0076] In some examples, server 205 may generate a synthesized face for a person (e.g., using synthesized face generator 255), and / or may modify a synthesized face for a person (e.g., using synthesized face generator 255 and / or image modifier 260) so that the skin tone of the synthesized face corresponds to the skin tone (e.g., skin color) of the person. This ensures that once the synthesized face is incorporated into modified image 265, the skin tone of the synthesized face does not conflict with the skin tone of other parts of the user's body (e.g., hands, arms, legs, neck, feet, etc.) that may be visible in image 235. This can ensure that the sense of immersion is not broken. In some examples, synthesized face generator 255 and / or image modifier 260 may change the skin tone of other parts of the user's body (e.g., hands, arms, legs, neck, feet, etc.) visible in the image to match the skin tone of the synthesized face. This can further increase privacy, for example by making it so that the user does not even know the original skin color of a person, but may increase the risk of visual artifacts, for example if synthetic face generator 255 and / or image modifier 260 misclassifies an object in an image as being a body part of a person (e.g., causing the imaging system to incorrectly modify the color and / or tone of the object), or if synthetic face generator 255 and / or image modifier 260 fails to identify a body part of a person (e.g., causing the imaging system to fail to modify the skin color of the body part).

[0077] Image modifier 260 outputs modified image 265, for example, by transmitting modified image 265 to user device 210 for output to user 215 by output device 270A of user device 220 and / or by transmitting modified image 265 to user device 220 for output to user 225 by output device 270B of user device 210. Figure 2A , the graphic representing modified image 265 illustrates modified image 265 as depicting the same environment as in the graphic representing image 235 with two people therein, but with the face of one of the people changed to a synthesized face.

[0078] User device 210 includes output device 270A. User device 220 includes output device 270B. Output devices 270A-270B may include one or more visual output devices, such as a display or a connector therefor. Output devices 270A-270B may include one or more audio output devices, such as speakers, headphones, and / or connectors therefor. Output devices 270A-270B may include one or more of output device 1235 and / or communication interface 1240 of computing system 1200. User device 220 causes the display of output devices 270A-270B to display modified image 265.

[0079] In some examples, output devices 270A-270B include one or more transceivers. The transceiver may include a wired transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include a wireless transmitter, a receiver, a transceiver, or a combination thereof. The transceiver may include one or more of the output device 1235 and / or the communication interface 1240 of the computing system 1200. In some examples, user device 210 and / or user device 220 causes the transceiver to transmit the modified image 265 to the recipient device. The recipient device may include a display, and the data transmitted from the transceiver of the output devices 270A-270B to the recipient device may cause the display of the recipient device to display the modified image 265.

[0080] In some examples, the display of the output device 270A-270B of the imaging system 200A is used as an optical "see-through" display that allows light from the real-world environment (scene) surrounding the imaging system 200A to traverse (e.g., pass through) the display of the output device 270A-270B to one or both eyes of the user. For example, the display of the output device 270A-270B may be at least partially transparent, translucent, light-transmissive, light-transmissive, or a combination thereof. In an illustrative example, the display of the output device 270A-270B includes a transparent, translucent and / or light-transmissive lens and a projector. The display of the output device 270A-270B may include a projector that projects virtual content (e.g., a synthesized face of the modified image 265) onto the lens. The lens may be, for example, a lens of a pair of glasses, a lens of goggles, a contact lens, a lens of a head-mounted display (HMD) device, or a combination thereof. Light from the real-world environment passes through the lens and reaches one or both eyes of the user. The projector may project virtual content (e.g., a synthesized face of modified image 265) onto the lens so that the virtual content appears overlaid on the user's view of the environment from the perspective of one or both of the user's eyes. In some examples, the projector may project the virtual content onto one or both retinas of one or both of the user's eyes, rather than onto the lens, which may be referred to as a virtual retinal display (VRD), a retinal scanning display (RSD), or a retinal projector (RP) display.

[0081] In some examples, the display of the output device 270A-270B of the imaging system 200A is a digital "transparent" display that allows a user of a user device of the imaging system 200A (e.g., user 215 of user device 210 or user 225 of user device 220) to see a view of the environment by displaying the view of the environment on the display of the output device 270A-270B. For example, based on sensor data (e.g., images, videos, depth images, point clouds, other depth data, or a combination thereof) captured by one or more environment-facing sensors in the sensor 230, the view of the environment displayed on the digital transparent display can be a view of the real-world environment surrounding the imaging system 200A, which in some cases is modified using the image modifier 260 (e.g., modified image 265). The view of the environment displayed on the digital transparent display can be a virtual environment (e.g., as in VR), which in some cases can include elements based on the real-world environment (e.g., the boundaries of a room). The view of the environment displayed on the digital transparent display can be an augmented environment based on the real-world environment (e.g., as in AR). The view of the environment displayed on the digital pass-through display may be a hybrid environment based on a real-world environment (e.g., as in MR). The view of the environment displayed on the digital pass-through display may include virtual content (e.g., a synthesized face of the modified image 265) overlaid on or otherwise incorporated into the view of the environment.

[0082] The trained ML model 277 may include one or more neural networks (NNs) (e.g., neural network 1000), one or more convolutional neural networks (CNNs), one or more trained time-delay neural networks (TDNNs), one or more deep networks, one or more autoencoders, one or more deep belief networks (DBNs), one or more recurrent neural networks (RNNs), one or more generative adversarial networks (GANs), one or more conditional generative adversarial networks (cGANs), one or more other types of neural networks, one or more trained support vector machines (SVMs), one or more trained random forests (RFs), one or more computer vision systems, one or more deep learning systems, or a combination thereof.

[0083] In some examples, the imaging system 200A includes a feedback engine 275A-275B of a user device (e.g., user device 210 and / or user device 220). The feedback engine 275A-275B is illustrated as part of the user device 210 and the user device 220, respectively, but may additionally or alternatively be part of the server 205. The feedback engine 275A-275B may detect feedback received from a user interface of the user device 210, the user device 220, and / or the server 205. The feedback may include feedback about the modified image 265 as displayed (e.g., using a display of the output device 270A-270B) according to the interactive client application 280A-280B and / or the interactive server application 285. The feedback may include feedback about the modified image 265 itself. Feedback may include feedback about the synthesized face generated by synthesized face generator 255 and incorporated into modified image 265, and / or the blending of the synthesized face with the rest of the image in modified image 265 by image modifier 260. Feedback may include feedback about face detection by face detector 240 and / or face recognition by face detector 240 and / or profile identifier 245. Feedback may include feedback about face detector 240, profile identifier 245, profile synthesized face identifier 250, synthesized face generator 255, image modifier 260, or a combination thereof. Feedback engines 275A-275B may detect feedback received from another engine of imaging system 200A about one engine of imaging system 200A, such as whether one engine decides to use data from another engine. Feedback received by feedback engines 275A-275B may be positive feedback or negative feedback. For example, if one engine of the imaging system 200A uses data from another engine of the imaging system 200A, or if positive feedback is received from a user through a user interface, the feedback engine 275A-275B may interpret this as positive feedback. If one engine of the imaging system 200A rejects data from another engine of the imaging system 200A, or if negative feedback is received from a user through a user interface, the feedback engine 275A-275B may interpret this as negative feedback. Positive feedback may also be based on attributes of sensor data from the sensor 230, such as the user smiling, laughing, nodding, saying a positive statement (e.g., "yes," "confirm," "ok," "next"), or otherwise reacting positively to the media. Negative feedback may also be based on attributes of sensor data from the sensor 230, such as the user frowning, crying, shaking his head (e.g., making a "no" gesture), saying a negative statement (e.g., "no," "reject," "bad," "not this"), or otherwise reacting negatively to the virtual content.

[0084] In some examples, the feedback engines 275A-275B provide the feedback as training data to one or more ML systems of the imaging system 200A (e.g., to the server 205) to update one or more trained ML models 277 of the imaging system 200A. For example, the feedback engines 275A-275B can provide the feedback as training data to the ML system and / or the trained ML model 277 to update the training for the face detector 240, the profile identifier 245, the profile synthesized face identifier 250, the synthesized face generator 255, the image modifier 260, or a combination thereof. Positive feedback can be used to reinforce and / or enhance the weight associated with the output of the ML system and / or the trained ML model 277. Negative feedback can be used to weaken and / or remove the weight associated with the output of the ML system and / or the trained ML model 277.

[0085] In some examples, certain elements of imaging system 200A (e.g., face detector 240, profile identifier 245, profile synthesized face identifier 250, synthesized face generator 255, image modifier 260, feedback engine 275A-275B, trained ML model 277, interactive client applications 280A-280B, interactive server application 285, data repository 290, or a combination thereof) include software elements, such as instruction sets corresponding to programs, which are executed on a processor (such as processor 1210 of computing system 1200, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, these elements of imaging system 200A include one or more hardware elements, such as a dedicated processor (e.g., processor 1210 of computing system 1200, image processor 150, host processor 152, ISP 154, or a combination thereof). In some examples, these elements of imaging system 200A may include a combination of one or more software elements and one or more hardware elements.

[0086] In some examples, certain elements of imaging system 200A (e.g., face detector 240, profile identifier 245, profile synthesized face identifier 250, synthesized face generator 255, image modifier 260, feedback engine 275A-275B, trained ML model 277, interactive client applications 280A-280B, interactive server application 285, data repository 290, or a combination thereof), other instances thereof, or portions thereof are included as part of the present invention. Figure 2A For example, user device 220 may include its own instances of sensor 230, even though these are not illustrated in FIG. Figure 2A Example in.

[0087] Figure 2B is a block diagram illustrating an example architecture of an imaging process performed by an imaging system 200B having two user devices (eg, user device 210 , user device 220 ). Figure 2B The imaging system 200B is similar to Figure 2A The imaging system 200A of FIG. 1 is shown in FIG. 20A , but lacks the server 205 . Figure 2A All components of the server 205 that is part of the imaging system 200A are Figure 2B A portion of the user device 210 in the imaging system 200B. Figure 2A All of the processes and / or operations performed by the server 205 in the imaging system 200A are performed by Figure 2B In some examples, the user device 210 in the imaging system 200B may be used Figure 2A The imaging system 200A and Figure 2B An imaging system between the imaging systems 200B of FIG. 200B , wherein certain components and / or operations are maintained at the server 205 and other components and / or operations are maintained at the user device 210 .

[0088] Figure 3A300 is a perspective view illustrating a head mounted display (HMD) 310 used as part of an imaging system 200A or 200B. The HMD 310 may be, for example, an augmented reality (AR) headset, a virtual reality (VR) headset, a mixed reality (MR) headset, an extended reality (XR) headset, or some combination thereof. The HMD 310 may be an example of a user device (e.g., user device 210 and / or user device 220) of an imaging system (e.g., imaging system 200A and / or imaging system 200B). The HMD 310 includes a first camera 330A and a second camera 330B along the front of the HMD 310. The first camera 330A and the second camera 330B may be examples of sensors 230 of the imaging systems 200A-200B. The HMD 310 includes a third camera 330C and a fourth camera 330D that face the user's eyes when the user's eyes face the display 340. The third camera 330C and the fourth camera 330D may be examples of sensors 230 of the imaging systems 200A-200B. In some examples, the HMD 310 may have only a single camera with a single image sensor. In some examples, the HMD 310 may include one or more additional cameras in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D. In some examples, the HMD 310 may include one or more additional sensors in addition to the first camera 330A, the second camera 330B, the third camera 330C, and the fourth camera 330D, which may also include other types of sensors 230 of the imaging system 200A. In some examples, the first camera 330A, the second camera 330B, the third camera 330C, and / or the fourth camera 330D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0089] The HMD 310 may include one or more displays 340 visible to a user 320 (wearing the HMD 310 on the head of the user 320). The one or more displays 340 of the HMD 310 may be examples of one or more displays of the output devices 270A-270B of the imaging systems 200A-200B. In some examples, the HMD 310 may include one display 340 and two viewfinders. The two viewfinders may include a left viewfinder for the left eye of the user 320 and a right viewfinder for the right eye of the user 320. The left viewfinder may be oriented so that the left eye of the user 320 sees the left side of the display. The right viewfinder may be oriented so that the right eye of the user 320 sees the right side of the display. In some examples, the HMD 310 may include two displays 340, including a left display that displays content to the left eye of the user 320 and a right display that displays content to the right eye of the user 320. The one or more displays 340 of the HMD 310 may be digital "pass-through" displays or optical "see-through" displays.

[0090] The HMD 310 may include one or more earpieces 335, which may be used as speakers and / or headphones to output audio to one or both ears of a user of the HMD 310 and may be examples of output devices 270A-270B. Figure 3A and Figure 3B 310, but it should be understood that the HMD 310 may include two earpieces, one for each ear of the user (left ear and right ear). In some examples, the HMD 310 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 230 of the imaging systems 200A-200B. In some examples, the audio output by the HMD 310 to the user through the one or more earpieces 335 may include or be based on audio recorded using the one or more microphones.

[0091] Figure 3B is an example Figure 3A310 is a perspective view 350 of a head mounted display (HMD) being worn by a user 320. The user 320 wears the HMD 310 on the head of the user 320 above the eyes of the user 320. The HMD 310 can capture images through a first camera 330A and a second camera 330B. In some examples, the HMD 310 displays one or more output images toward the eyes of the user 320 using a display 340. In some examples, the output image may include a modified image 265. The output image may be based on an image (e.g., image 235) captured by the first camera 330A and the second camera 330B, for example, overlaid with virtual content (e.g., a synthesized face of the modified image 265). The output image may provide a stereoscopic view of the environment, in some cases with virtual content superimposed and / or with other modifications. For example, the HMD 310 may display a first display image to the right eye of the user 320, the first display image being based on the image captured by the first camera 330A. The HMD 310 may display a second display image to the left eye of the user 320, the second display image being based on an image captured by the second camera 330B. For example, the HMD 310 may provide in the display image an overlay virtual content superimposed on the image captured by the first camera 330A and the second camera 330B. The third camera 330C and the fourth camera 330D may capture images of the eyes before, during, and / or after the user views the display image displayed by the display 340. In this way, sensor data from the third camera 330C and / or the fourth camera 330D may capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content. The earpiece 335 of the HMD 310 is illustrated as being in the ear of the user 320. The HMD 310 may output audio to the user 320 through the earpiece 335 and / or through another earpiece (not shown) of the HMD 310 in the other ear (not shown) of the user 320.

[0092] Figure 4A 4 is a perspective view 400 illustrating a front surface of a mobile handset 410 that includes a front-facing camera and can be used as part of imaging system 200A or 200B. Mobile handset 410 can be an example of a user device (e.g., user device 210 and / or user device 220) of an imaging system (e.g., imaging system 200A and / or imaging system 200B). Mobile handset 410 can be, for example, a cellular phone, a satellite phone, a portable game console, a music player, a health tracking device, a wearable device, a wireless communication device, a laptop device, a mobile device, any other type of computing device or computing system discussed herein, or a combination thereof.

[0093] The front surface 420 of the mobile phone 410 includes a display 440. The front surface 420 of the mobile phone 410 includes a first camera 430A and a second camera 430B. The first camera 430A and the second camera 430B may be examples of sensors 230 of the imaging systems 200A-200B. The first camera 430A and the second camera 430B may face the user, including the user's eyes, while content (e.g., the modified image 265 output by the image modifier 260) is displayed on the display 440. The display 440 may be an example of a display of the output device 270A-270B of the imaging systems 200A-200B.

[0094] The first camera 430A and the second camera 430B are illustrated in a bezel around the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be located in a notch or cutout cut out of the display 440 on the front surface 420 of the mobile phone 410. In some examples, the first camera 430A and the second camera 430B may be under-display cameras located between the display 440 and the rest of the mobile phone 410 so that light passes through a portion of the display 440 before reaching the first camera 430A and the second camera 430B. The first camera 430A and the second camera 430B of the perspective view 400 are front-facing cameras. The first camera 430A and the second camera 430B face a direction perpendicular to the planar surface of the front surface 420 of the mobile phone 410. The first camera 430A and the second camera 430B may be two cameras of one or more cameras of the mobile phone 410. In some examples, the front surface 420 of the mobile phone 410 may have only a single camera.

[0095] In some examples, display 440 of mobile phone 410 displays one or more output images to a user using mobile phone 410. In some examples, the output images may include modified image 265. The output images may be based on images (e.g., image 235) captured by first camera 430A, second camera 430B, third camera 430C, and / or fourth camera 430D, for example, with virtual content (e.g., a synthesized face of modified image 265) superimposed.

[0096] In some examples, the front surface 420 of the mobile phone 410 may include one or more additional cameras in addition to the first camera 430A and the second camera 430B. The one or more additional cameras may also be examples of sensors 230 of the imaging systems 200A-200B. In some examples, the front surface 420 of the mobile phone 410 may include one or more additional sensors in addition to the first camera 430A and the second camera 430B. The one or more additional sensors may also be examples of sensors 230 of the imaging systems 200A-200B. In some cases, the front surface 420 of the mobile phone 410 includes more than one display 440. The one or more displays 440 of the front surface 420 of the mobile phone 410 may be examples of displays of output devices 270A-270B of the imaging systems 200A-200B. For example, the one or more displays 440 may include one or more touch screen displays.

[0097] The mobile handset 410 may include one or more speakers 435A and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4A 435A, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 230 of the imaging systems 200A-200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the front surface 420 of the mobile phone 410, where these microphones are examples of sensors 230 of the imaging systems 200A-200B. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435A and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0098] Figure 4B 4 is a perspective view 450 illustrating a rear surface 460 of a mobile phone that includes a rear camera and can be used as part of the imaging system 200A or 200B. The mobile phone 410 includes a third camera 430C and a fourth camera 430D on the rear surface 460 of the mobile phone 410. The third camera 430C and the fourth camera 430D of the perspective view 450 are rear-facing. The third camera 430C and the fourth camera 430D can be FIG. 2A to FIG. 2B The third camera 430C and the fourth camera 430D face in a direction perpendicular to the planar surface of the rear surface 460 of the mobile phone 410.

[0099] The third camera 430C and the fourth camera 430D may be two cameras of the one or more cameras of the mobile phone 410. In some examples, the rear surface 460 of the mobile phone 410 may have only a single camera. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional cameras in addition to the third camera 430C and the fourth camera 430D. The one or more additional cameras may also be examples of the sensor 230 of the imaging system 200A-200B. In some examples, the rear surface 460 of the mobile phone 410 may include one or more additional sensors in addition to the third camera 430C and the fourth camera 430D. The one or more additional sensors may also be examples of the sensor 230 of the imaging system 200A-200B. In some examples, the first camera 430A, the second camera 430B, the third camera 430C, and / or the fourth camera 430D may be examples of the image capture and processing system 100, the image capture device 105A, the image processing device 105B, or a combination thereof.

[0100] The mobile handset 410 may include one or more speakers 435B and / or other audio output devices (eg, earphones or headphones or connectors thereof) that may output audio to one or more ears of a user of the mobile handset 410 . Figure 4B 435B, but it should be understood that the mobile phone 410 may include more than one speaker and / or other audio devices. In some examples, the mobile phone 410 may also include one or more microphones (not shown). The one or more microphones may be examples of sensors 230 of the imaging systems 200A-200B. In some examples, the mobile phone 410 may include one or more microphones along and / or adjacent to the rear surface 460 of the mobile phone 410, where these microphones are examples of sensors 230 of the imaging systems 200A-200B. In some examples, the audio output to the user by the mobile phone 410 through one or more speakers 435B and / or other audio output devices may include or be based on audio recorded using one or more microphones.

[0101] The mobile phone 410 can use the display 440 on the front surface 420 as a transparent display. For example, the display 440 can display an output image, such as the modified image 265. The output image can be based on the image captured by the third camera 430C and / or the fourth camera 430D (e.g., image 235), for example, with virtual content (e.g., a synthesized face of the modified image 265) superimposed and / or the modification made by the image modifier 260 applied. The first camera 430A and / or the second camera 430B can capture images of the user's eyes (and / or other parts of the user) before, during, and / or after the output image with virtual content is displayed on the display 440. In this way, the sensor data from the first camera 430A and / or the second camera 430B can capture the reaction of the user's eyes (and / or other parts of the user) to the virtual content.

[0102] Figure 5 5 is a conceptual diagram 500 illustrating generation of a modified image 550 by replacing a face 525 of user 520 with a synthesized face 530 of user 520. For example, an image 510 is illustrated, which may be an example of an image 235 captured by sensor 230. Face detector 240 detects face 525 of user 520 in image 510. Profile identifier 245 identifies that face 525 of user 520 in image 510 belongs to user 520 named John Smith, and retrieves a profile for user 520 (e.g., from data repository 290) or creates the profile if it does not already exist (e.g., to be stored in data repository 290). Profile synthesized face identifier 250 identifies a synthesized face 530 to be used for user 520 based on the profile for user 520. Image modifier 260 generates modified image 550 by modifying image 510 to replace face 525 of user 520 with synthesized face 530 for user 520.

[0103] Image 510 depicts light from light source 540 (i.e., a window). In some examples, light source may include a window, a lamp, the sun, a display screen, a light bulb, or a combination thereof. Face 525 of user 520 is illuminated from the left by light source 540 in image 510. Therefore, to generate modified image 550, image modifier 260 modifies image 510 to replace face 525 of user 520 with synthesized face 530 for user 520, and applies and / or simulates lighting from light source 540 to synthesized face 530 in modified image 550. Image modifier 260 may apply and / or simulate lighting from light source 530 to synthesized face 540 by applying and / or simulating lighting from the same direction as light source 540 (e.g., from the left side of synthesized face 530, similar to light from the left side of face 525), having a similar light color as light from light source 540 (as shown in image 510), using a similar light pattern as light from light source 540 (as shown in image 510), or a combination thereof.

[0104] In addition to the exposed face of user 520 (e.g., face 525 or synthesized face 530), image 510 and modified image 550 also each depict an exposed hand of user 520. Synthesized face generator 255 and / or image modifier 260 may synchronize the skin tone (e.g., skin color) between face 525, synthesized face 530, and / or other portions of user 520's body visible in image 510 and / or modified image 550 (e.g., such as in the hands of user 520). For example, in some examples, the imaging system may generate synthesized face 530, or modify synthesized face 530 after generating synthesized face 530, so that the skin tone of the synthesized face corresponds to (e.g., matches) the skin tone of face 525 and / or other portions of user 520's body visible in image 510 (e.g., such as in the hands of user 520). In some examples, the imaging system may change the skin color of other parts of the user's body visible in the image (eg, the hands of user 520 ) to match the skin color of the synthesized face.

[0105] Fig. 6A is a conceptual diagram 600A illustrating generation of a modified image 615 by replacing face 525 of user 520 with pixelated face 617 of the user. Fig. 6A The conceptual diagram 600A includes Figure 5510 of the user 520. The face detector 240 detects the face 525 of the user 520 in the image 510. The profile identifier 245 identifies that the face 525 in the image 510 belongs to the user 520, and identifies (e.g., retrieves or creates) a profile for the user 520. The image modifier 610 of the imaging system (e.g., of the server 205 and / or the user device 210) generates a modified image 615 by modifying the image 510 to replace the face 525 of the user 520 with a pixelated face 617 for the user 520. For example, the image modifier 610 may pixelate the face 525 to generate the pixelated face 617. In some cases, the image modifier 610 may generate and / or use a blurred face (e.g., a blurred instance of the face 525 of the user 520) to replace the pixelated face 617.

[0106] Although Fig. 6A The use of pixelated face 617 (or blurred face) in modified image 615 protects the privacy of user 520, but this privacy protection is not as good as that of Figure 5 5. Image processing techniques used to depixelate, deblur, and / or sharpen images may reproduce face 525 (or a close approximation) from pixelated face 617 (or blurred face) in modified image 615, thereby providing a security hole in such privacy protection and potentially allowing a malicious party to discover the identity of user 520 from modified image 615. The security issues of using pixelated face 617 (or blurred face) may be compounded in video, as a larger number of pixelated or blurred faces from different video frames may be layered to reconstruct more detailed information about the different facial feature structures of face 525 of user 520. In addition, as Fig. 6A The use of pixelated face 617 (or blurred face) in modified image 615 as in illustrative example destroys the sense of immersion in the environment of the network-based interactive system because the pixelated face and / or blurred face (e.g., pixelated face 617) does not appear in the natural environment and therefore looks unnatural and unrealistic, which in turn makes the modified image 615 as a whole look unnatural and unrealistic.

[0107] Figure 6B is a conceptual diagram 600B illustrating generation of a modified image 625 by replacing the face 525 of the user 520 with a review block 627 . Figure 6B The conceptual diagram 600B includes Figure 5510 of the user 520. The face detector 240 detects the face 525 of the user 520 in the image 510. The profile identifier 245 identifies that the face 525 in the image 510 belongs to the user 520, and identifies (e.g., retrieves or creates) a profile of the user 520. The image modifier 620 of the imaging system (e.g., of the server 205 and / or the user device 210) generates a modified image 625 by modifying the image 510 to replace the face 525 of the user 520 with a review block 627. The review block 627 is illustrated as a black rectangular box, but can be any shape, any color, and / or any pattern. In some examples, the review block 627 can include an image. For example, the image modifier 620 can overlay the review block 627 on the face 525 of the user 520 to generate the modified image 625.

[0108] Although Figure 6B The use of review block 627 in modified image 625 as in FIG. 1 protects the privacy of user 520, but Figure 6B The use of censorship block 627 in modified image 625 as described above breaks the sense of immersion in the environment for the network-based interactive system because the censorship block (e.g., censorship block 627) does not appear in the natural environment and therefore looks unnatural and unrealistic, which in turn makes the modified image 625 as a whole look unnatural and unrealistic.

[0109] Figure 6C is a conceptual diagram 600C illustrating that a modified image 635 is generated by replacing the face 525 of the user 520 with an avatar face 637 for the user. Figure 6C Conceptual diagram of the 600C includes Figure 5 510 of the user 520. The face detector 240 detects the face 525 of the user 520 in the image 510. The profile identifier 245 identifies that the face 525 in the image 510 belongs to the user 520, and identifies (e.g., retrieves or creates) a profile of the user 520. The image modifier 630 of the imaging system (e.g., of the server 205 and / or the user device 210) generates a modified image 635 by modifying the image 510 to replace the face 525 of the user 520 with an avatar face 637. The avatar face 637 is illustrated as a smiling face, but can be another type of cartoon, non-realistic, or stylized face. In some examples, the avatar face 637 can include an image. For example, the image modifier 630 can overlay the avatar face 637 on the face 525 of the user 520 to generate the modified image 635.

[0110] Although Figure 6C Using avatar face 637 in modified image 635 as in FIG. 6 protects the privacy of user 520, but Figure 6CUsing avatar face 637 in modified image 635 as in illustrative example destroys the sense of immersion in the environment for the network-based interactive system because the avatar face (e.g., avatar face 637) does not appear in the natural environment and therefore looks unnatural and unrealistic, which in turn makes the modified image 635 as a whole look unnatural and unrealistic.

[0111] Fig.6D is a conceptual diagram 600D illustrating generation of a modified image by using image restoration 647 instead of user 520 . Fig.6D Conceptual image of the 600D includes Figure 5 510 of the image 510. The face detector 240 detects the face 525 of the user 520 in the image 510. The profile identifier 245 identifies that the face 525 in the image 510 belongs to the user 520, and identifies (e.g., retrieves or creates) a profile of the user 520. The image modifier 640 of the imaging system (e.g., of the server 205 and / or the user device 210) generates a modified image 645 by modifying the image 510 to replace the region (or a portion thereof, such as the face 525) in the image 510 depicting the user 520 with the image inpainting 647 based on the background region around the region depicting the user 520 to infer the content (e.g., the background) behind the user 520. For example, the image modifier 640 may perform the image inpainting 647 using pixel interpolation and (in some examples) a trained ML model (e.g., the trained ML model 277). In the modified image 645, the inpainted line is drawn as a dashed line.

[0112] In some cases, image restoration 647 can produce distracting visual artifacts. For example, in the area where image restoration 647 is performed, the line connecting the wall and the floor behind user 520 appears to bend in an unnatural manner. In addition, although image restoration 647 "completes" the body of another person who is partially behind user 520 in image 510 by adding a second leg to the other person, the orientation of the foot of the added second leg exactly matches the orientation of the other foot of the other person, making the body posture of the other person appear unnatural. Fig.6D Using image restoration 647 in modified image 645 as in Fig.6DUsing image restoration 647 in modified image 645 as in , breaks the sense of immersion in the environment for the network-based interactive system because the visual artifacts produced by image restoration (e.g., image restoration 647) do not appear in the natural environment and therefore appear unnatural and unrealistic, making the modified image 645 as a whole appear unnatural and unrealistic. The visual artifacts produced by image restoration 647 can be compounded in the video because slightly different visual artifacts may appear in different video frames, sometimes creating a "ghost" of user 520 in modified image 645 at the location where user 520 was located in image 510.

[0113] Figure 7 7 is a conceptual diagram 700 illustrating generating a synthesized face 730 based on features of multiple input faces. In particular, a trained ML model 740 is used to generate a synthesized face 730 based on using input faces 710 and input faces 720 as inputs to a trained ML model 740 and / or as training data for training the trained ML model 740. The trained ML model 740 may be an example of one or more trained ML models in the trained ML models 277 used by the synthesized face generator 255. The trained ML model 740 may be part of the synthesized face generator 255.

[0114] Various facial feature structures and / or facial attributes of the input face 710 and the input face 720 appear in the synthesized face 730, but the synthesized face 730 is visually different in identity from both the input face 710 and the input face 720. For example, the facial expression, facial shape, nose, and mouth of the synthesized face 730 are similar to those of the input face 710. However, the eyes, eyebrows, hairstyle, and chin shape of the synthesized face 730 are similar to those of the input face 710. In some examples, the trained ML model 740 may be trained (e.g., by the user device 210, the user device 220, the server 205, and / or another system) to generate realistic synthesized faces using training data of an input image including a human face and a pre-generated synthesized face using features from the input image. In some examples, the trained machine learning model 740 may include one or more generative adversarial networks (GANs) and / or other deep learning (DL)-based generative ML models.

[0115] Fig. 8A8 is a conceptual diagram 800 illustrating a first user 805 using a head mounted device 810 to view an interactive environment including a second user 820, whose face is displayed to the first user 805 through the head mounted device 810 based on a privacy setting 835 of the second user 805 and a relationship indicator 840 indicating a family relationship between the first user 805 and the second user 820. The head mounted device 810 is an example of an HMD 310. The head mounted device 810 displays a field of view (FOV) 815A including a face 825 of the user 820 to the first user 805. The data repository 290 stores the privacy setting 835 of the second user 820 in a profile 830 of the second user 820. The profile 830 is an example of a profile identified using a profile identifier 245. The privacy setting 835 of the second user 820 includes an approval list indicating that family members of the second user 820 are allowed to view the face 825 of the user 820. The data repository 290 also stores a relationship indicator 840 indicating that the first user 805 and the second user 820 are indeed a family. Therefore, the first user 805 is on the approved list of the second user 820. Therefore, the face 825 of the user 805 is shown to the user 820 in the FOV 815A via the head mounted device 810 in an undisguised manner.

[0116] Figure 8B 850 is a conceptual diagram illustrating a first user 805 using a head mounted device 810 to view an interactive environment including a second user 820, a synthesized face 855 of the second user being displayed to the first user 805 via the head mounted device 810 based on the privacy settings 835 of the second user 820 and a relationship indicator 860 indicating a family relationship between the first user 805 and the second user 820. The head mounted device 810 displays a field of view (FOV) 815B to the first user 805, the field of view including the user 820 with the synthesized face 855 of the user 820. Figure 8B In the data repository 290, different Fig. 8A 840 of the head mounted device 810. Relationship indicator 860 indicates that first user 805 and second user 820 are not a family. Therefore, first user 805 is not on the approved list of second user 820. Therefore, face 825 of user 805 is not shown to user 820 in FOV 815B via head mounted device 810. Instead, FOV 815B is modified (e.g., using image modifier 260) to replace face 825 of user 820 with synthesized face 855.

[0117] like FIG. 8A to FIG. 8BAs shown, in some examples, whether a person (e.g., second user 820) has his or her identity disguised (e.g., by replacing the person's face with a synthesized face 855) or undisguised (e.g., by leaving the person's face unmodified) may depend on the identity of the viewer (e.g., first user 805), such as whether the person (e.g., second user 820) approves the viewer (e.g., first user 805) via the person's privacy settings (e.g., privacy settings 835).

[0118] Fig. 9 9 is a conceptual diagram 900 illustrating combining identity input 910 and expression input 920 to generate a combined face 930 having the identity of identity input 910 and the expression of expression input 920. In particular, combined face 930 is generated using trained ML model 940 based on using identity input 910 and expression input 920 as inputs to trained ML model 940 and / or as training data for training trained ML model 940. Trained ML model 940 may be an example of one or more trained ML models 277 used by synthesized face generator 255 and / or image modifier 260. Trained ML model 940 may be part of synthesized face generator 255 and / or image modifier 260.

[0119] Various facial feature structures and / or facial attributes representing the identity of the identity input 910 appear in the combined face 930. For example, the identity input 910 has the identity of a bearded man with glasses, and so does the combined face 930. Various facial feature structures and / or facial attributes representing the expression of the expression input 920 appear in the combined face 930. For example, the expression input 920 has a shouting expression with an open mouth, and so does the combined face 930. In some examples, the trained ML model 940 may be trained (e.g., by the user device 210, the user device 220, the server 205, and / or another system) to generate a realistic combined face using training data including identity input images (e.g., identity input 910) and expression input images (e.g., expression input 920) of human faces and pre-generated combined human faces (e.g., combined face 930) using features from the input images. In some examples, the trained machine learning model 940 may include one or more generative adversarial networks (GANs) and / or other deep learning (DL)-based generative ML models. In some examples, the identity input is an example of a synthesized face 915, such as a synthesized face 915 generated by a synthesized face generator 255. In some examples, the expression input is an example of a real face 925, such as a face detected by a face detector 240. The use of a combined face 930 having the identity of the identity input 910 (synthesized face 915) and the expression of the expression input 920 (real face 925) can allow a user 215 of a user device 210 to realistically see the person's mouth move when the person speaks and / or see other facial expressions on the person's face without breaking the immersion, while still maintaining the person's privacy and not revealing his / her identity to the user 215.

[0120] Fig.10 1100 is a block diagram illustrating an example of a neural network (NN) 1000 that may be used for media processing operations. The neural network 1000 may include any type of deep network, such as a convolutional neural network (CNN), an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), a generative adversarial network (GAN), and / or other types of neural networks. The neural network 1000 may be an example of one of the trained ML model 277, the trained ML model 740, the trained ML model 940, one or more trained ML models used in the process 1100, or a combination thereof. The neural network 1000 may be used by the face detector 240, the profile identifier 245, the profile synthesized face identifier 250, the synthesized face generator 255, the image modifier 260, or a combination thereof.

[0121] An input layer 1010 of neural network 1000 includes input data. The input data of input layer 1010 may include data representing pixels of one or more input image frames. In some examples, the input data of input layer 1010 includes data representing pixels of image data (e.g., an image captured by image capture and processing system 100, an image 235 captured by sensor 230, an image captured by one of cameras 330A-330D, an image captured by one of cameras 430A-430D, image 510, input face 710, input face 720, FOV 815A, identity input 910, expression input 920, manipulated image 1105, manipulated face 1110, manipulated synthesized face 1120, or a combination thereof).

[0122] The image may include image data from an image sensor, including raw pixel data (including a single color per pixel based on, for example, a Bayer color filter) or processed pixel values ​​(e.g., RGB pixels of an RGB image). Neural network 1000 includes a plurality of hidden layers 1012A, 1012B through 1012N. Hidden layers 1012A, 1012B through 1012N include "N" hidden layers, where "N" is an integer greater than or equal to one. The plurality of hidden layers may include as many layers as required for a given application. Neural network 1000 also includes an output layer 1014 that provides output resulting from the processing performed by hidden layers 1012A, 1012B through 1012N.

[0123] In some examples, output layer 1014 may provide an output image, such as a synthesized face (e.g., generated by synthesized face generator 255), modified image 265, synthesized face 530, modified image 550, synthesized face 730, FOV 815B, synthesized face 855, synthesized face 930, synthesized face 915, combined face 930, manipulated synthesized face 1120, manipulated modified image 1125, or a combination thereof. In some examples, output layer 1014 may also provide other types of data, such as face detection data for face detector 240, face recognition data for face detector 240, and / or profile identifier 245, or a combination thereof.

[0124] Neural network 1000 is a multi-layer neural network of interconnected filters. Each filter can be trained to learn features that represent input data. Information associated with these filters is shared between different layers, and each layer retains information as it processes it. In some cases, neural network 1000 may include a feedforward network in which there is no feedback connection in which the output of the network is fed back to itself. In some cases, network 1000 may include a recurrent neural network that may have loops that allow information to be carried across nodes as input is read in.

[0125] In some cases, information can be exchanged between layers by node-to-node interconnection between each layer. In some cases, the network may include a convolutional neural network, which may not link each node in a layer to each other node in the next layer. In a network in which information is exchanged between layers, the nodes of the input layer 1010 may activate a set of nodes in the first hidden layer 1012A. For example, as shown in the figure, each of the input nodes of the input layer 1010 may be connected to each of the nodes of the first hidden layer 1012A. The nodes of the hidden layer may transform the information by applying an activation function (e.g., a filter) to the information of each input node. Then, the information derived from the transformation may be passed to the nodes of the next hidden layer 1012B and may activate these nodes, which may perform their own specified functions. Example functions include convolution functions, reduction, enlargement, data transformation and / or any other suitable functions. Then, the output of the hidden layer 1012B may activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 1012N may activate one or more nodes of the output layer 1014, which provide a processed output image. In some cases, although a node in neural network 1000 (e.g., node 1016) is shown as having multiple output lines, the node has a single output, and all lines shown as output from the node represent the same output value.

[0126] In some cases, each node or interconnection between nodes may have a weight, which is a set of parameters derived from the training of neural network 1000. For example, an interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have a tunable digital weight that can be tuned (e.g., based on a training data set), thereby allowing neural network 1000 to adapt to the input and be able to learn as more and more data is processed.

[0127] The neural network 1000 is pre-trained to process features of data from the input layer 1010 using different hidden layers 1012A, 1012B to 1012N to provide an output through the output layer 1014.

[0128] Fig.111 is a flow chart illustrating an imaging process 1100. The imaging process 1100 may be performed by an imaging system. In some examples, the imaging system may include, for example, the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200A, the imaging system 200B, the server 205, the user device 210, the user device 220, the HMD 310, the mobile phone 410, Figure 5 Imaging system, FIG. 6A to FIG. 6D The imaging system, the trained ML model 740, the head mounted device 810, the trained ML model 940, the neural network 1000, the computing system 1200, the processor 1210, or a combination thereof is performed.

[0129] At operation 1105, the imaging system is configured to receive an image of an environment. In some aspects, at least a portion of the environment includes one or more virtual elements. For example, the one or more virtual elements may be associated with an extended reality (XR) application or other application. In some cases, at least a portion of the image of the environment is captured by an image sensor (or in some cases multiple image sensors). In some examples, the imaging system includes an image sensor connector that couples and / or connects the image sensor to the remainder of the imaging system (e.g., including a processor and / or memory of the imaging system). In some examples, the imaging system receives image data from the image sensor by receiving image data from, on, and / or using the image sensor connector.

[0130] Examples of image sensors include image sensor 130, sensor 230, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor that captures image 510, an image sensor for capturing an image used as input data for input layer 1010 of NN 1000, input device 1245, another image sensor described herein, another sensor described herein, or a combination thereof.

[0131] Examples of image data include image data captured using image capture and processing system 100, image 235 captured using an image sensor of sensor 230, image data captured using first camera 330A, image data captured using second camera 330B, image data captured using third camera 330C, image data captured using fourth camera 330D, image data captured using first camera 430A, image data captured using second camera 430B, image data captured using third camera 430C, image data captured using fourth camera 430D, image 510, FOV 815A, an image used as input data for input layer 1010 of NN1000, an image captured using input device 1245, another image described herein, another set of image data described herein, or a combination thereof.

[0132] At operation 1110, the imaging system is configured to detect a face of a person in an image. At operation 1115, the imaging system is configured to identify profile data associated with the person. In one illustrative example, the profile data does not match a profile of the person stored in a memory of the imaging system or an external memory to which the imaging system has access (e.g., this may indicate that the imaging system does not know the person). In another illustrative example, the profile data may indicate one or more privacy settings for the person. In some cases, to identify the profile data associated with the person, the imaging system is configured to determine the identity of the face of the person in the image using facial recognition. In some cases, to identify the profile data associated with the person, the imaging system is configured to identify the profile data associated with the person based on one or more communications from a user device associated with the person.

[0133] At operation 1120, the imaging system is configured to retrieve a synthesized face based on the profile data. In some examples, the synthesized face is generated to be different from any face of any known real person. In some cases, the synthesized face is generated to have a skin color that matches the skin color of the person. In some aspects, the imaging system is configured to generate the synthesized face before retrieving the synthesized face corresponding to the person from the profile data. For example, to generate the synthesized face, the imaging system can use a trained machine learning model (e.g., a generative adversarial network (GAN) or other trained machine learning model) that combines attributes from multiple different images of multiple different faces into a synthesized face. In some aspects, the imaging system is configured to regenerate the synthesized face. In some cases, the synthesized face changes appearance when it is regenerated.

[0134] At operation 1125, the imaging system is configured to generate a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person. In some aspects, to generate the modified image, the imaging system is configured to use the synthesized face and at least a portion of the image as input to a trained machine learning model that modifies the image to replace the face of the person with the synthesized face for the person. In some cases, to generate the modified image, the imaging system is configured to modify a first skin color of a portion of the person in the image to correspond to a second skin color of the synthesized face of the person.

[0135] In some aspects, the imaging system is configured to detect a second face of a second person in the image. The imaging system is further configured to identify second profile data associated with the second person, such as a second identity based on the second face of the second person in the image. In some cases, the second profile data does not match a stored profile of the person in at least one memory. In some cases, the second profile data indicates one or more privacy settings for the second person. The imaging system is configured to determine not to modify the second face of the second person in the modified image based on the second profile data associated with the second person. In such aspects, to generate the modified image, the imaging system is configured to leave the second face of the second person unmodified in the image in addition to modifying the image to replace the face of the person with a synthesized face for the person.

[0136] In some examples, the imaging system is configured to detect a second face of a second person in an image. The imaging system is further configured to identify second profile data associated with the second person, such as a second identity based on the second face of the second person in the image. In some cases, the second profile data does not match a stored profile of the person in at least one memory. In some cases, the second profile data indicates one or more privacy settings for the second person. The imaging system is configured to retrieve a second synthesized face corresponding to the second person and different from the second face of the second person based on the second profile data. In such aspects, to generate a modified image, the imaging system is configured to modify the image to replace the second face of the second person with the second synthesized face for the second person in addition to modifying the image to replace the face of the person with the synthesized face for the person.

[0137] In some cases, the imaging system is configured to identify a head pose of a person's head in the image. In some cases, to modify the image to replace the person's face with a synthesized face for the person, the imaging system is configured to orient the synthesized face for the person according to the head pose of the person's head in the image.

[0138] In some aspects, the imaging system is configured to identify lighting conditions in the environment in the image. In some cases, to modify the image to replace the face of the person with the synthesized face for the person, the imaging system is configured to illuminate the synthesized face for the person according to the lighting conditions in the environment in the image.

[0139] In some examples, the imaging system is configured to identify at least one privacy setting associated with a person in the image from profile data. In some cases, the at least one privacy setting indicates that the person's face will be avoided from being output. In some examples, the imaging system is configured to modify the image to replace the person's face with a synthesized face for the person in response to identifying at least one privacy setting indicating that the output of the person's face will be avoided. In some cases, the imaging system is configured to determine that a predetermined condition is active. In some cases, the at least one privacy setting indicates that the person's face will be avoided from being output when the predetermined condition is active. In some examples, the predetermined condition is associated with the person's location in the environment, the identity of the viewer of the environment, the time of day, any combination thereof, or other conditions.

[0140] In some aspects, the imaging system is configured to identify an expression on a face of a person in the image. The imaging system is configured to modify the synthesized face before generating the modified image based on a combination of the identity of the synthesized face and the expression identified on the face of the person in the image.

[0141] At operation 1130, the imaging system is configured to output a modified image of the environment. In some examples, the imaging system includes a display, and outputting the modified image data includes using the display to display the modified image data. Examples of displays include output devices 270A-270B, display 340 of HMD 310, display 440 of mobile phone 410, display device 1235, or a combination thereof. In some examples, the imaging system includes a communication interface, and outputting the modified image data includes transmitting or sending the modified image to at least a recipient device using at least the communication interface. In some examples, the imaging system is configured to receive an image from an image sensor of the imaging system, and the imaging system is configured to output the modified image using an output device of the imaging system. In such examples, the imaging system may include an XR device (e.g., ARHMD, VR HMD, etc.), a mobile phone or device, a wireless communication device, or other computing device. In some examples, the imaging system is configured to receive an image from a user device via a network, and the imaging system is configured to transmit the modified image to the user device via the network. In such examples, the imaging system may include one or more servers communicatively coupled to the user device via a network.

[0142] In some examples, an imaging system may include: components for receiving an image of an environment; components for detecting a face of a person in the image; components for identifying profile data associated with the person; components for retrieving a synthesized face based on the profile data; components for generating a modified image of the environment at least in part by modifying the image to replace the face of the person with a synthesized face for the person; and components for outputting the modified image of the environment. In some examples, the components may include image capture and processing system 100, image capture device 105A, image processing device 105B, image processor 150, ISP 154, host processor 152, image sensor 130, sensor 230, first camera 330A, second camera 330B, third camera 330C, fourth camera 330D, first camera 430A, second camera 430B, third camera 430C, fourth camera 430D, an image sensor that captures image 510, an image sensor for capturing an image used as input data for input layer 1010 of NN 1000, input device 1245, another image sensor described herein, one or more of another sensor described herein, or a combination thereof.

[0143] In some examples, the processes described herein (e.g., Figure 1 , FIG. 2A to FIG. 2B , Figure 5 , FIG. 6A to FIG. 6D , Figure 7 , FIG. 8A to FIG. 8B , Fig. 9 , Fig.10 process, Fig.11 In some examples, the processes described herein may be performed by the image capture and processing system 100, the image capture device 105A, the image processing device 105B, the image processor 150, the ISP 154, the host processor 152, the imaging system 200A, the imaging system 200B, the server 205, the user device 210, the user device 220, the HMD 310, the mobile phone 410, Figure 5 Imaging system, FIG. 6A to FIG. 6D The imaging system, the trained ML model 740, the head mounted device 810, the trained ML model 940, the neural network 1000, the computing system 1200, the processor 1210, or a combination thereof is performed.

[0144] The computing device may include any suitable device, such as a mobile device or wireless communication device (e.g., a mobile phone or other mobile handset), a desktop computing device, a tablet computing device, an extended reality (XR) device (e.g., a VR headset or HMD, an AR headset or HMD, an AR glasses or HMD, etc.), a wearable device (e.g., a network-connected watch or smart watch or other wearable device), a server computer, an autonomous vehicle or a computing device of an autonomous vehicle, a robotic device, a television, and / or any other computing device having resource capabilities to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on an Internet Protocol (IP) or other types of data.

[0145] The components of the computing device may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0146] The processes described herein are illustrated as logic flow diagrams, block diagrams, or conceptual diagrams, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer executable instruction stored on one or more computer-readable storage media that performs the described operations when executed by one or more processors. Generally speaking, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0147] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, implemented by hardware, or a combination of the above. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0148] Fig.12 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig.12 An example of a computing system 1200 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 1205. The connection 1205 may be a physical connection using a bus, or a direct connection into the processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a networked connection, or a logical connection.

[0149] In some aspects, computing system 1200 is a distributed system in which the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components that each perform some or all of the functionality that the component is described for. In some aspects, each component may be a physical or virtual device.

[0150] The example system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 that couple various system components including system memory 1215, such as read-only memory (ROM) 1220 and random access memory (RAM) 1225, to the processor 1210. The computing system 1200 may include a cache 1212 of high-speed memory directly connected to, in close proximity to, or integrated as part of the processor 1210.

[0151] Processor 1210 may include any general purpose processor and hardware or software services, such as services 1232, 1234, and 1236 stored in storage device 1230, that are configured to control processor 1210 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1210 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0152] To enable user interaction, the computing system 1200 includes an input device 1245 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1200 may also include an output device 1235 that can be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1200. The computing system 1200 may include a communication interface 1240, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using a wired and / or wireless transceiver, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, Wireless signal transfer, Low Energy (BLE) wireless signal transfer, Wireless signal transfer, radio frequency identification (RFID) wireless signal transfer, near field communication (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 1202.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communication interface 1240 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 1200 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0153] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer-readable memory device, and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0154] Storage device 1230 may include software services, servers, services, etc., which, when the code defining such software is executed by processor 1210, causes the system to perform functions. In some aspects, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1210, connection 1205, output device 1235, etc.

[0155] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent using any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0156] In some aspects, computer readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0157] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can also be practiced without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid confusing these aspects in unnecessary details. In other instances, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0158] Various aspects may be described above as a process or method, which is depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe an operation as a sequential process, many operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to a calling function or a main function.

[0159] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or function group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0160] The equipment implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0161] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0162] In the above description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary aspects of the present application have been described in detail herein, it is to be understood that each inventive concept can be implemented and adopted in various other ways, and the appended claims are not intended to be interpreted as including these variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, various aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that, in alternative aspects, each method can be performed in a different order than described.

[0163] One of ordinary skill in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of this specification.

[0164] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0165] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0166] Claim language or other language that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that states "at least one of A and B" means A, B, or A and B. In another example, claim language that states "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that states "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0167] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with aspects disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on specific applications and the design constraints proposed to the entire system. Technicians may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0168] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. Computer-readable data storage media can form a part of a computer program product, which can include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0169] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0170] Illustrative aspects of the present disclosure include:

[0171] Aspect 1. A device for processing one or more images, the device comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors being configured to: receive an image of an environment; detect a face of a person in the image; identify profile data associated with the person; retrieve a synthesized face based on the profile data; generate a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; and output the modified image of the environment.

[0172] Aspect 2. The apparatus according to aspect 1, wherein the synthesized face is generated to be different from any face of any known real person.

[0173] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein, in order to generate the modified image, the one or more processors are configured to use the synthesized face and at least a portion of the image as input to a trained machine learning model, and the trained machine learning model modifies the image to replace the face of the person with the synthesized face for the person.

[0174] Aspect 4. An apparatus according to any one of aspects 1 to 3, wherein the profile data does not match a stored profile of a person in the at least one memory.

[0175] Aspect 5. An apparatus according to any one of aspects 1 to 4, wherein the profile data indicates one or more privacy settings for the person.

[0176] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein to identify the profile data associated with the person, the one or more processors are configured to use facial recognition to determine the identity of the face of the person in the image.

[0177] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein, in order to identify the profile data associated with the person, the one or more processors are configured to identify the profile data associated with the person based on one or more communications from a user device associated with the person.

[0178] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the one or more processors are configured to: detect a second face of a second person in the image; identify second profile data associated with the second person based on a second identity of the second face of the second person in the image; and determine that the second face of the second person is not modified in the modified image based on the second profile data associated with the second person, wherein in order to generate the modified image, the one or more processors are configured to, in addition to modifying the image to replace the face of the person with the synthesized face for the person, also cause the second face of the second person to be unmodified in the image.

[0179] Aspect 9. An apparatus according to aspect 8, wherein the second profile data does not match a stored profile of a person in the at least one memory.

[0180] Aspect 10. The apparatus of aspect 8, wherein the second profile data indicates one or more privacy settings for the second person.

[0181] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the one or more processors are configured to: detect a second face of a second person in the image; identify second profile data associated with the second person based on a second identity of the second face of the second person in the image; and retrieve a second synthesized face corresponding to the second person and different from the second face of the second person based on the second profile data, wherein in order to generate the modified image, the one or more processors are configured to modify the image to replace the face of the person with the synthesized face for the second person in addition to modifying the image to replace the face of the person with the synthesized face for the second person.

[0182] Aspect 12. An apparatus according to any one of Aspects 1 to 11, wherein the one or more processors are configured to: generate the synthesized face before retrieving the synthesized face corresponding to the person from the profile data, wherein in order to generate the synthesized face, the one or more processors are configured to use a trained machine learning model that combines attributes from multiple different images of multiple different faces into the synthesized face.

[0183] Aspect 13. An apparatus according to Aspect 12, wherein the trained machine learning model is a generative adversarial network.

[0184] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the one or more processors are configured to regenerate the synthesized face, wherein the synthesized face changes in appearance when being regenerated.

[0185] Aspect 15. An apparatus according to any one of Aspects 1 to 14, wherein the synthesized face is generated to have a skin color that matches the skin color of the person.

[0186] Aspect 16. An apparatus according to any one of Aspects 1 to 15, wherein, in order to generate the modified image, the one or more processors are configured to modify a first skin color of a portion of the person in the image to correspond to a second skin color of the synthesized face for the person.

[0187] Aspect 17. An apparatus according to any one of Aspects 1 to 16, wherein the one or more processors are configured to: identify a head posture of the head of the person in the image, wherein in order to modify the image to replace the face of the person with the synthesized face for the person, the one or more processors are configured to orient the synthesized face for the person according to the head posture of the head of the person in the image.

[0188] Aspect 18. An apparatus according to any one of Aspects 1 to 17, wherein the one or more processors are configured to: identify lighting conditions in the environment in the image, wherein in order to modify the image to replace the face of the person with the synthesized face for the person, the one or more processors are configured to illuminate the synthesized face for the person according to the lighting conditions in the environment in the image.

[0189] Aspect 19. An apparatus according to any one of Aspects 1 to 18, wherein the one or more processors are configured to: identify at least one privacy setting associated with the person in the image from the profile data, wherein the at least one privacy setting indicates that output of the face of the person will be avoided, and wherein the one or more processors are configured to modify the image to replace the face of the person with the synthesized face for the person in response to identifying the at least one privacy setting indicating that output of the face of the person will be avoided.

[0190] Aspect 20. An apparatus according to Aspect 19, wherein the one or more processors are configured to: determine that a predetermined condition is active, wherein at least one of the privacy settings indicates that output of the face of the person will be avoided when the predetermined condition is active, wherein the predetermined condition is associated with at least one of the position of the person in the environment or the identity of an observer of the environment.

[0191] Aspect 21. An apparatus according to any one of Aspects 1 to 20, wherein the one or more processors are configured to: identify an expression on the face of the person in the image; and modify the synthesized face before generating the modified image based on a combination of the identity of the synthesized face and the expression identified on the face of the person in the image.

[0192] Aspect 22. An apparatus according to any one of Aspects 1 to 21, wherein at least a portion of the image of the environment is captured by an image sensor.

[0193] Aspect 23. An apparatus according to any one of Aspects 1 to 22, wherein at least a portion of the environment includes one or more virtual elements.

[0194] Aspect 24. The apparatus of aspect 23, wherein the one or more virtual elements are associated with an extended reality (XR) application.

[0195] Aspect 25. An apparatus according to any one of Aspects 1 to 24, the apparatus further comprising: a display, wherein in order to output the modified image, the one or more processors are configured to display the modified image using at least the display.

[0196] Aspect 26. An apparatus according to any one of Aspects 1 to 25, the apparatus further comprising: a communication interface, wherein in order to output the modified image, the one or more processors are configured to use at least the communication interface to transmit the modified image to at least a receiving device.

[0197] Aspect 27. An apparatus according to any one of Aspects 1 to 26, wherein the one or more processors are configured to receive the image from an image sensor of the apparatus, wherein the one or more processors are configured to output the modified image using an output device of the apparatus, and wherein the apparatus includes at least one of a head-mounted display (HMD), a mobile phone, or a wireless communication device.

[0198] Aspect 28. An apparatus according to any one of Aspects 1 to 27, wherein the one or more processors are configured to receive an image from a user device via a network, wherein the one or more processors are configured to transmit the modified image to the user device via the network, and wherein the apparatus includes one or more servers communicatively coupled to the user device via the network.

[0199] Aspect 29. A method for processing one or more images, the method comprising: receiving an image of an environment; detecting a face of a person in the image; identifying profile data associated with the person; retrieving a synthesized face based on the profile data; generating a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; and outputting the modified image of the environment.

[0200] Aspect 30. The method according to aspect 29, wherein the synthesized face is generated to be different from any face of any known real person.

[0201] Aspect 31. A method according to any one of Aspects 29 to 30, wherein generating the modified image includes using the synthesized face and at least a portion of the image as input to a trained machine learning model, the trained machine learning model modifying the image to replace the face of the person with the synthesized face for the person.

[0202] Aspect 32. A method according to any one of Aspects 29 to 31, wherein the profile data does not match a stored profile of a person in at least one memory.

[0203] Aspect 33. A method according to any one of aspects 29 to 32, wherein the profile data indicates one or more privacy settings for the person.

[0204] Aspect 34. A method according to any one of aspects 29 to 33, wherein identifying the profile data associated with the person comprises using facial recognition to determine the identity of the face of the person in the image.

[0205] Aspect 35. A method according to any one of Aspects 29 to 34, wherein identifying the profile data associated with the person includes identifying the profile data associated with the person based on one or more communications from a user device associated with the person.

[0206] Aspect 36. A method according to any one of Aspects 29 to 35, the method further comprising: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; and determining that the second face of the second person is not modified in the modified image based on the second profile data associated with the second person, wherein generating the modified image includes, in addition to modifying the image to replace the face of the person with the synthesized face for the person, leaving the second face of the second person unmodified in the image.

[0207] Aspect 37. The method of aspect 36, wherein the second profile data does not match a stored profile of a person in the at least one memory.

[0208] Aspect 38. The method of aspect 36, wherein the second profile data indicates one or more privacy settings for the second person.

[0209] Aspect 39. A method according to any one of Aspects 29 to 38, the method further comprising: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; and retrieving a second synthesized face corresponding to the second person and different from the second face of the second person based on the second profile data, wherein generating the modified image includes modifying the image to replace the face of the person with the synthesized face for the second person in addition to modifying the image to replace the face of the person with the synthesized face for the second person.

[0210] Aspect 40. A method according to any one of Aspects 29 to 39, the method further comprising: generating the synthesized face before retrieving the synthesized face corresponding to the person from the profile data, wherein in order to generate the synthesized face, the one or more processors are configured to use a trained machine learning model, which combines attributes from multiple different images of multiple different faces into the synthesized face.

[0211] Aspect 41. A method according to Aspect 40, wherein the trained machine learning model is a generative adversarial network.

[0212] Aspect 42. A method according to any one of aspects 29 to 41, the method also comprising regenerating the synthesized face, wherein the synthesized face changes in appearance when being regenerated.

[0213] Aspect 43. A method according to any one of Aspects 29 to 42, wherein the synthesized face is generated to have a skin color that matches the skin color of the person.

[0214] Aspect 44. A method according to any one of Aspects 29 to 43, wherein generating the modified image includes modifying a first skin color of a portion of the person in the image to correspond to a second skin color for the synthesized face of the person.

[0215] Aspect 45. A method according to any one of Aspects 29 to 44, the method further comprising: identifying a head posture of the head of the person in the image, wherein modifying the image to replace the face of the person with the synthesized face for the person includes orienting the synthesized face for the person according to the head posture of the head of the person in the image.

[0216] Aspect 46. A method according to any one of Aspects 29 to 45, the method further comprising: identifying lighting conditions in the environment in the image, wherein modifying the image to replace the face of the person with the synthesized face for the person includes illuminating the synthesized face for the person according to the lighting conditions in the environment in the image.

[0217] Aspect 47. A method according to any one of Aspects 29 to 46, the method further comprising: identifying at least one privacy setting associated with the person in the image from the profile data, wherein the at least one privacy setting indicates that output of the face of the person will be avoided; and modifying the image to replace the face of the person with the synthesized face for the person in response to identifying the at least one privacy setting indicating that output of the face of the person will be avoided.

[0218] Aspect 48. A method according to any one of Aspects 29 to 47, the method further comprising: determining that a predetermined condition is active, wherein at least one privacy setting indicates that output of the face of the person will be avoided when the predetermined condition is active, wherein the predetermined condition is associated with at least one of the position of the person in the environment or the identity of an observer of the environment.

[0219] Aspect 49. A method according to any one of Aspects 29 to 48, the method further comprising: identifying an expression on the face of the person in the image; and modifying the synthesized face before generating the modified image based on a combination of the identity of the synthesized face and the expression identified on the face of the person in the image.

[0220] Aspect 50. The method according to any one of aspects 29 to 49, wherein at least a portion of the image of the environment is captured by an image sensor.

[0221] Aspect 51. A method according to any one of Aspects 29 to 50, wherein at least a portion of the environment includes one or more virtual elements.

[0222] Aspect 52. The apparatus of aspect 51, wherein the one or more virtual elements are associated with an extended reality (XR) application.

[0223] Aspect 53. A method according to any one of aspects 29 to 52, wherein outputting the modified image comprises displaying the modified image using at least one display.

[0224] Aspect 54. The method according to any one of aspects 29 to 53, wherein outputting the modified image comprises transmitting the modified image to at least one recipient device using at least one communication interface.

[0225] Aspect 55. The method according to any one of aspects 29 to 54, further comprising: receiving the image from an image sensor; and outputting the modified image using an output device.

[0226] Aspect 56. The method according to any one of Aspects 29 to 55, further comprising: receiving the image from a user device via a network; and transmitting the modified image to the user device via the network.

[0227] Aspect 57: A non-transitory computer-readable medium is provided, on which instructions are stored, which, when executed by one or more processors, cause the one or more processors to perform the operations according to any one of Aspects 1 to 56.

[0228] Aspect 58: An apparatus for processing one or more images, comprising one or more components for performing the operations according to any one of aspects 1 to 56.

Claims

1. An apparatus for processing one or more images, the apparatus comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors configured to: receiving an image of the environment; detecting a face of a person in the image; identifying profile data associated with the person; retrieving a synthesized face based on the profile data; generating a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; as well as The modified image of the environment is output.

2. An apparatus according to claim 1, wherein to generate the modified image, the one or more processors are configured to use the synthesized face and at least a portion of the image as input to a trained machine learning model, and the trained machine learning model modifies the image to replace the face of the person with the synthesized face for the person.

3. The apparatus of claim 1, wherein the profile data does not match a stored profile of a person in the at least one memory. 4 . The apparatus of claim 1 , wherein the profile data indicates one or more privacy settings for the person.

5. The apparatus of claim 1, wherein to identify the profile data associated with the person, the one or more processors are configured to use facial recognition to determine the identity of the face of the person in the image.

6. The apparatus of claim 1 , wherein to identify the profile data associated with the person, the one or more processors are configured to identify the profile data associated with the person based on one or more communications from a user device associated with the person.

7. The apparatus of claim 1, wherein the one or more processors are configured to: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; and determining, based on the second profile data associated with the second person, that the second face of the second person is not modified in the modified image, wherein, to generate the modified image, the one or more processors are configured to, in addition to modifying the image to replace the face of the person with the synthesized face for the person, also cause the second face of the second person to be not modified in the image.

8. The apparatus of claim 7, wherein the second profile data does not match a stored profile of the person in the at least one memory.

9. The apparatus of claim 7, wherein the second profile data indicates one or more privacy settings for the second person.

10. The apparatus of claim 1, wherein the one or more processors are configured to: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; and retrieving a second synthesized face corresponding to the second person and different from the second face of the second person based on the second profile data, wherein to generate the modified image, the one or more processors are configured to modify the image to replace the second face of the second person with the second synthesized face for the second person in addition to modifying the image to replace the face of the person with the synthesized face for the second person.

11. The apparatus of claim 1 , wherein the one or more processors are configured to: Generating the synthesized face prior to retrieving the synthesized face corresponding to the person from the profile data, wherein to generate the synthesized face, the one or more processors are configured to use a trained machine learning model that combines attributes from multiple different images of multiple different faces into the synthesized face.

12. The apparatus of claim 1, wherein the one or more processors are configured to regenerate the synthesized face, wherein the synthesized face changes in appearance when being regenerated.

13. The apparatus of claim 1, wherein the synthesized face is generated to have a skin color that matches a skin color of the person.

14. The apparatus of claim 1, wherein to generate the modified image, the one or more processors are configured to modify a first skin color of a portion of the person in the image to correspond to a second skin color for the synthesized face of the person.

15. The apparatus of claim 1, wherein the one or more processors are configured to: Identifying a head pose of the person's head in the image, wherein in order to modify the image to replace the person's face with the synthesized face for the person, the one or more processors are configured to orient the synthesized face for the person based on the head pose of the person's head in the image.

16. The apparatus of claim 1, wherein the one or more processors are configured to: At least one privacy setting associated with the person in the image is identified from the profile data, wherein the at least one privacy setting indicates that output of the face of the person is to be avoided, wherein the one or more processors are configured to modify the image to replace the face of the person with the synthesized face for the person in response to identifying the at least one privacy setting indicating that output of the face of the person is to be avoided.

17. The apparatus of claim 1, wherein the one or more processors are configured to: identifying an expression on the face of the person in the image; and The synthesized face is modified based on a combination of the identity of the synthesized face and the expression identified on the face of the person in the image before generating the modified image.

18. The apparatus according to claim 1, further comprising: A display, wherein to output the modified image, the one or more processors are configured to display the modified image using at least the display.

19. The apparatus according to claim 1, further comprising: A communication interface, wherein to output the modified image, the one or more processors are configured to transmit the modified image to at least a recipient device using at least the communication interface.

20. The apparatus of claim 1, wherein the one or more processors are configured to receive the image from an image sensor of the apparatus, wherein the one or more processors are configured to output the modified image using an output device of the apparatus, and wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

21. An apparatus according to claim 1, wherein the one or more processors are configured to receive an image from a user device via a network, wherein the one or more processors are configured to transmit the modified image to the user device via the network, and wherein the apparatus includes one or more servers communicatively coupled to the user device via the network.

22. A method of processing one or more images, the method comprising: receiving an image of the environment; detecting a face of a person in the image; identifying profile data associated with the person; retrieving a synthesized face based on the profile data; generating a modified image of the environment at least in part by modifying the image to replace the face of the person with the synthesized face for the person; as well as The modified image of the environment is output.

23. A method according to claim 22, wherein generating the modified image includes using the synthesized face and at least a portion of the image as input to a trained machine learning model, and the trained machine learning model modifies the image to replace the face of the person with the synthesized face for the person.

24. The method of claim 22, wherein the profile data does not match a stored profile of a person in at least one memory.

25. The method of claim 22, wherein the profile data indicates one or more privacy settings for the person.

26. The method of claim 22, wherein identifying the profile data associated with the person comprises using facial recognition to determine the identity of the face of the person in the image.

27. The method of claim 22, wherein identifying the profile data associated with the person comprises identifying the profile data associated with the person based on one or more communications from a user device associated with the person.

28. The method of claim 22, further comprising: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; as well as and determining, based on the second profile data associated with the second person, that the second face of the second person is not modified in the modified image, wherein generating the modified image includes modifying the image to replace the face of the person with the synthesized face for the person in addition to leaving the second face of the second person unmodified in the image.

29. The method of claim 22, further comprising: detecting a second face of a second person in the image; identifying second profile data associated with the second person based on a second identity of the second face of the second person in the image; as well as A second synthesized face corresponding to the second person and different from the second face of the second person is retrieved based on the second profile data, wherein generating the modified image includes, in addition to modifying the image to replace the face of the person with the synthesized face for the person, also modifying the image to replace the second face of the second person with the second synthesized face for the second person.

30. The method of claim 22, further comprising: identifying, from the profile data, at least one privacy setting associated with the person in the image, wherein the at least one privacy setting indicates that output of the face of the person is to be avoided; and The image is modified to replace the face of the person with the synthesized face for the person in response to identifying the at least one privacy setting indicating that output of the face of the person is to be avoided.

Images