Generative pipeline using a single selfie input

Abstract

The present technology receives an input image, which includes a selfie. The present technology uses a neural network to convert the input image into a latent representation of an identity. The present technology uses a diffusion model to convert text conditions into a second latent representation compatible with the latent representation of the identity. The present technology converts a pose template into a latent feature set for the diffusion model. The present technology generates an intermediate image based on the latent representation of the identity, the second latent representation, and the latent feature set. The present technology uses a face enhancement network to modify the intermediate image based on the input image. The present technology uses a face restoration network to generate a final output image based on the modified intermediate image. The present technology is provided to display the final output image on a display of a client device.

Description

Technology Field

[0001] Claim of priority

[0002] This patent application claims the benefit of priority to U.S. Application No. 18 / 417,541 filed January 19, 2024, the entirety of which is incorporated by reference into this specification. Background Technology

[0003] With the increasing use of digital images, the affordability of portable computing devices, the availability of increased capacity in digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become a part of the daily lives of more and more people. Brief explanation of the drawing

[0004] In drawings that are not necessarily drawn to actual scale, similar numbers may describe similar components within different views. To facilitate the identification of any specific element or act, the top digit or numbers in a reference number indicate the drawing number where the element is first introduced. Some non-limiting examples are illustrated in the drawings of the attached drawings: FIG. 1 is a schematic representation of a networked environment in which the present disclosure may be arranged, according to some examples. Figure 2 is a schematic representation of a messaging system having both client-side and server-side functionality according to some examples. Figure 3 is a schematic representation of a data structure as maintained in a database, according to some examples. Figure 4 is a schematic representation of a message according to some examples. FIG. 5 illustrates a system including a head-wearable device according to some examples. FIG. 6 illustrates an exemplary processing flow of a generative pipeline using a single selfie ("one shot") input according to embodiments of the present technology. FIG. 7 illustrates exemplary interfaces according to embodiments of the present technology. FIG. 8 illustrates exemplary interfaces according to embodiments of the present technology. FIG. 9 illustrates examples of output media content provided by a one-shot personalized generative pipeline according to one embodiment. FIG. 10 illustrates examples of output media content provided by a one-shot personalized generative pipeline according to one embodiment. FIG. 11 illustrates a flowchart illustrating a method according to some exemplary embodiments. FIG. 12 is a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to enable the machine to perform any one or more of the methodologies discussed in this specification, according to some examples. FIG. 13 is a block diagram illustrating a software architecture in which examples can be implemented. Specific details for implementing the invention

[0005] Users with diverse interests from various locations can capture digital images of various subjects and make the captured images available to others through networks such as the Internet. To enhance users' experience with digital images and provide various features, enabling computing devices to perform image processing operations on various objects and / or features captured under a wide range of changing conditions (e.g., changes in image scales, noise, lighting, motion, or geometric distortion) can be a challenging and computationally intensive task.

[0006] Augmented reality technology aims to bridge the gap between virtual and real-world environments by providing enhanced real-world environments augmented with electronic information. As a result, the electronic information appears to be part of the real-world environment as perceived by the user. In one example, augmented reality technology further provides a user interface that allows interaction with electronic information overlaid on the enhanced real-world environment.

[0007] As mentioned above, due to the increased use of digital images, the affordability of portable computing devices, the availability of increased capacity in digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become an increasingly common part of people's daily lives. Users with diverse interests from various locations can capture digital images of various subjects and make the captured images available to others through networks such as the Internet. To enhance users' experience with digital images and provide diverse features, enabling computing devices to perform image processing operations on various objects and / or features captured under a wide range of changing conditions (e.g., changes in image scales, noise, lighting, motion, or geometric distortion) can be a challenging and computationally intensive task.

[0008] Embodiments of the present technology enable the generation of augmented reality content (e.g., images, videos, etc.) using a generative machine learning (ML) / artificial intelligence (AI) pipeline that receives media content representing an individual's head (e.g., a self-photo or "selfie" image), text conditions (e.g., text information or text description), and a pose template as inputs, and provides output media content based on the aforementioned inputs.

[0009] Messaging systems are frequently and increasingly utilized by users of mobile computing devices in various settings to provide different types of functionality in a convenient manner. As described herein, the messaging system includes practical applications that provide improvements in capturing image data and rendering AR content (e.g., images, videos, etc.) based on captured image data by providing at least technical improvements in capturing image data using power and resource-constrained electronic devices. These improvements in capturing image data are made possible by techniques provided by the present technology, which reduce latency and increase efficiency in processing captured image data, thereby also reducing power consumption in capturing devices.

[0010] As further discussed herein, the infrastructure supports the creation and sharing of interactive media, referred herein as messages including 3D content or AR effects, across various components of the messaging system. In the exemplary embodiments described herein, messages may enter the system from a live camera or from storage (e.g., where messages including 3D content and / or AR effects are stored in memory or a database). The system supports motion sensor input and the loading of external effects and asset data.

[0011] As mentioned in this specification, the phrases “augmented reality experience,” “augmented reality content item,” and “augmented reality content generator” include or refer to various image processing operations corresponding to image modification, filters, AR content generators, media overlays, transformations, etc., as further described in this specification, and may additionally include the playback of audio or music content during the presentation of AR content or media content.

[0012] Networked computing environment

[0013] FIG. 1 is a block diagram illustrating an exemplary interaction system (100) for facilitating interactions (e.g., exchange of text messages, execution of text audio and video calls, or play of games) over a network. The interaction system (100) includes a plurality of user systems (102), each of which hosts a plurality of applications including an interaction client (104) and other applications (106). Each interaction client (104) is communically coupled to other instances of the interaction client (104) (e.g., hosted on their respective other user systems (102)), an interaction server system (110), and third-party servers (112) through one or more communication networks including a network (108) (e.g., the Internet). The interaction client (104) may also communicate with locally hosted applications (106) using APIs (Applications Program Interfaces).

[0014] Each user system (102) may include a number of user devices, such as a mobile device (114), a head-wearable apparatus (116), and a computer client device (118), which are connected to communicate to exchange data and messages.

[0015] An interaction client (104) interacts with other interaction clients (104) and an interaction server system (110) through a network (108). Data exchanged between the interaction clients (104) (e.g., interactions (120)) and between the interaction clients (104) and the interaction server system (110) includes functions (e.g., commands that invoke functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0016] The interaction server system (110) provides server-side functionality to interaction clients (104) via a network (108). Although specific functions of the interaction system (100) are described herein as being performed by the interaction client (104) or by the interaction server system (110), the location of specific functions within the interaction client (104) or the interaction server system (110) may be a design choice. For example, it may be technically desirable to initially place specific technologies and functions within the interaction server system (110), but later migrate these technologies and functions to the interaction client (104) when the user system (102) has sufficient processing capacity.

[0017] The interaction server system (110) supports various services and operations provided to the interaction clients (104). These operations include transmitting data to the interaction clients (104), receiving data from them, and processing data generated by them. This data may include message content, client device information, geolocation information, media augmentation and overlays, message content persistence conditions, entity relationship information, and live event information. Data exchanges within the interaction system (100) are invoked and controlled through functions available via the user interfaces (UIs) of the interaction clients (104).

[0018] Now, referring specifically to the interaction server system (110), an API (Application Program Interface) server (122) is coupled to the interaction servers (124) to provide programmatic interfaces, thereby making the functions of the interaction servers (124) accessible to interaction clients (104), other applications (106), and third-party servers (112). The interaction servers (124) are communicably coupled to a database server (126) to facilitate access to a database (128) that stores data associated with interactions processed by the interaction servers (124). Similarly, a web server (130) is coupled to the interaction servers (124) and provides web-based interfaces to the interaction servers (124). To this end, the web server (130) processes incoming network requests via HTTP (Hypertext Transfer Protocol) and various other related protocols.

[0019] An API (Application Program Interface) server (122) receives and transmits interaction data (e.g., commands and message payloads) between interaction servers (124), user systems (102) (and, for example, interaction clients (104) and other applications (106)) and a third-party server (112). Specifically, the API (Application Program Interface) server (122) provides a set of interfaces (e.g., routines and protocols) that can be called or queried by interaction clients (104) and other applications (106) to invoke the functionality of interaction servers (124). The API (Application Program Interface) server (122) includes account registration; login functionality; and transmitting interaction data from a specific interaction client (104) to another interaction client (104) via the interaction servers (124). Exposing various functions supported by the interaction servers (124), including communicating media files (e.g., images or videos) from the interaction client (104) to the interaction servers (124); setting up a collection of media data (e.g., stories); searching for a list of friends of the user of the user system (102); searching for messages and content; adding and deleting entities (e.g., friends) to an entity relationship graph (e.g., entity graph (310)); the location of friends within the entity relationship graph; and opening application events (e.g., related to the interaction client (104)).

[0020] The interactive servers (124) host a number of systems and subsystems described below with reference to FIG. 2.

[0021] System Architecture

[0022] FIG. 2 is a block diagram illustrating additional details regarding an interaction system (100) according to some examples. Specifically, the interaction system (100) is illustrated as including an interaction client (104) and interaction servers (124). The interaction system (100) implements a number of subsystems supported by the interaction client (104) on the client side and by the interaction servers (124) on the server side. In some examples, these subsystems are implemented as microservices. A microservice subsystem (e.g., a microservice application) may have components that enable it to operate independently and communicate with other services. Exemplary components of a microservice subsystem may include the following:

[0023] ● Functional Logic: Functional logic implements the functionality of a microservice subsystem, expressing the specific capabilities or functions provided by the microservice.

[0024] ● API Interface: Microservices can communicate with each other component through well-defined APIs or interfaces using lightweight protocols such as REST or messaging. The API interface defines the inputs and outputs of a microservice subsystem and how it interacts with other microservice subsystems of the interaction system (100).

[0025] ● Data Storage: A microservice subsystem may be responsible for its own data storage, which may be in the form of a database (e.g., using a database server (126) and a database (128)), a cache, or other storage mechanism. This allows the microservice subsystem to operate independently of other microservices of the interaction system (100).

[0026] ● Service Discovery: Microservice subsystems can discover other microservice subsystems of the interacting system (100) and communicate with them. Service discovery mechanisms enable microservice subsystems to locate other microservice subsystems and communicate with them in a scalable and efficient manner.

[0027] ● Monitoring and Logging: Microservice subsystems may need to be monitored and logged to ensure availability and performance. Monitoring and logging mechanisms enable the tracking of the health and performance of microservice subsystems.

[0028] In some examples, the interaction system (100) may use a monolithic architecture, a service-oriented architecture (SOA), a function-as-a-service (FaaS) architecture, or a modular architecture:

[0029] Exemplary subsystems are discussed below.

[0030] The image processing system (202) provides various functions that enable the user to capture and augment media content associated with the message (e.g., annotate or modify or edit in other ways).

[0031] The camera system (204) includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system (102) (e.g., directly or through operating system controls) to modify and augment real-time images captured and displayed through the interaction client (104).

[0032] The augmentation system (206) provides functions related to the creation and publication of augmentations (e.g., media overlays) for images that are captured in real time by the cameras of the user system (102) or retrieved from the memory of the user system (102). For example, the augmentation system (206) operatively selects, presents, and displays media overlays (e.g., image filters or image lenses) to the interactive client (104) for augmenting real-time images received through the camera system (204) or stored images retrieved from the memory (502) of the user system (102). These augmentations are selected by the augmentation system (206) and presented to the user of the interactive client (104) based on a number of inputs and data, for example, as follows:

[0033] ● Geographic location of the user system (102); and

[0034] ● User entity relationship information of the user system (102).

[0035] Augmentation may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlaying. Audio and visual content or visual effects may be applied to media content items (e.g., photos or videos) in the user system (102) for communication in messages, or to video content such as video content streams or feeds transmitted from the interaction client (104). In this way, the image processing system (202) may interact with and support various subsystems of the communication system (208), such as the messaging system (210) and the video communication system (212).

[0036] A media overlay may include text or image data that can be overlaid on top of a photo taken by the user system (102) or a video stream produced by the user system (102). In some examples, the media overlay may be a location overlay (e.g., Venice Beach), the name of a live event, or a vendor name overlay (e.g., Beach Coffee House). In additional examples, an image processing system (202) uses the geolocation of the user system (102) to identify a media overlay containing the name of a vendor located at the geolocation of the user system (102). The media overlay may include other indicia associated with the vendor. Media overlays may be stored in databases (128) and accessed via a database server (126).

[0037] The image processing system (202) provides a user-based publication platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which a specific media overlay should be provided to other users. The image processing system (202) generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0038] The augmentation creation system (214) supports augmented reality developer platforms and includes an application for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) of the interaction client (104). The augmentation creation system (214) provides content creators with a library of built-in features and tools, including, for example, custom shaders, tracking techniques, and templates.

[0039] In some examples, the augmentation generation system (214) provides a merchant-based publication platform that enables merchants to select a specific augmentation associated with a geolocation through a bidding process. For example, the augmentation generation system (214) associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time.

[0040] The communication system (208) is responsible for enabling and processing multiple forms of communication and interaction within the interaction system (100) and includes a messaging system (210), an audio communication system (216), and a video communication system (212). The messaging system (210) is responsible for enforcing temporary or time-limited access to content by the interaction clients (104). The messaging system (210) incorporates multiple timers (e.g., within a short-term timer system) that selectively enable access to messages and associated content (e.g., for presentation and display) through the interaction clients (104) based on duration and display parameters associated with a message or a collection of messages (e.g., a story). The audio communication system (216) enables and supports audio communications (e.g., real-time audio chat) between multiple interaction clients (104). Similarly, the video communication system (212) enables and supports video communications (e.g., real-time video chat) between multiple interactive clients (104).

[0041] The user management system (218) is responsible for managing user data and profiles and maintains entity information regarding users of the interaction system (100) and relationships between users (e.g., stored in an entity table (308), an entity graph (310), and profile data (302)).

[0042] The collection management system (220) is operationally responsible for managing sets or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages including images, videos, text, and audio) may be organized into "event galleries" or "event stories." These collections may be available for a specific period, such as the duration of the event to which the content is related. For example, content related to a music concert may be available as a "story" for the duration of that music concert. The collection management system (220) may also be responsible for posting icons that provide notifications of specific collections on the user interface of the interaction client (104). The collection management system (220) includes a curation function that enables a collection manager to manage and curate collections of specific content. For example, the curation interface enables an event organizer to curate collections of content related to a specific event (e.g., deleting inappropriate content or duplicate messages). Additionally, the collection management system (220) automatically curates content collections using machine vision (or image recognition technology) and content rules. In certain examples, users may be paid a reward to include user-generated content in a collection. In these cases, the collection management system (220) operates to automatically pay these users for using their content.

[0043] The map system (222) provides various geolocation functions (e.g., geolocation) and supports the presentation of map-based media content and messages by the interaction client (104). For example, the map system (222) enables the display of user icons or avatars (e.g., stored in profile data (302)) on the map to display the current or past locations of the user's "friends" as well as media content created by these friends (e.g., collections of messages including photos and videos) within the context of the map. For example, a message posted by a user to the interaction system (100) from a specific geolocation can be displayed to the specific user's "friends" on the map interface of the interaction client (104) within the context of the map at that specific location. Furthermore, the user can share their location and status information with other users of the interaction system (100) through the interaction client (104) (e.g., using an appropriate status avatar), and this location and status information is similarly displayed to selected users within the context of the map interface of the interaction client (104).

[0044] The game system (224) provides various gaming functions within the context of the interaction client (104). The interaction client (104) provides a game interface that provides a list of available games that can be launched by a user within the context of the interaction client (104) and played with other users of the interaction system (100). The interaction system (100) also enables a specific user to invite other users to participate in the play of a specific game by issuing invitations to other users from the interaction client (104). The interaction client (104) also supports audio, video, and text messaging (e.g., chats) within the context of gameplay, provides a leaderboard for games, and supports the provision of in-game rewards (e.g., coins and items).

[0045] The external resource system (226) provides an interface for the interactive client (104) to communicate with remote servers (e.g., third-party servers (112)) to launch or access external resources, namely applications or applets. Each third-party server (112) hosts, for example, a markup language (e.g., HTML5) based application or a small version of an application (e.g., a game, utility, payment, or ride-sharing application). The interactive client (104) can launch a web-based resource (e.g., an application) by accessing an HTML5 file from the third-party servers (112) associated with the web-based resource. Applications hosted by the third-party servers (112) are programmed in JavaScript using a Software Development Kit (SDK) provided by the interactive servers (124). The SDK includes Application Programming Interfaces (APIs) that have functions that can be called or invoked by the web-based application. The interaction servers (124) host a JavaScript library that provides access to a given external resource for specific user data of the interaction client (104). HTML5 is one example of a technology for programming games, but applications and resources programmed based on other technologies may be used.

[0046] To integrate the functions of the SDK into a web-based resource, the SDK is downloaded from the interaction servers (124) by the third-party server (112) or otherwise received by the third-party server (112). Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. Subsequently, the code of the web-based resource may call or invoke specific functions of the SDK to integrate the features of the interaction client (104) into the web-based resource.

[0047] The SDK stored on the interaction server system (110) effectively provides a bridge between external resources (e.g., applications (106) or applets) and the interaction client (104). This provides the user with an experience of communicating seamlessly with other users on the interaction client (104) while preserving the look and feel of the interaction client (104). To bridge communications between the external resources and the interaction client (104), the SDK facilitates communication between third-party servers (112) and the interaction client (104). A bridge script running on the user system (102) establishes two unidirectional communication channels between the external resources and the interaction client (104). Messages are transmitted asynchronously between the external resources and the interaction client (104) through these communication channels. Each SDK function call is transmitted as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and transmitting a message having that callback identifier.

[0048] By using the SDK, not all information from the interaction client (104) is shared with the third-party servers (112). The SDK limits which information is shared based on the needs of the external resources. Each third-party server (112) provides the interaction servers (124) with an HTML5 file corresponding to the web-based external resource. The interaction servers (124) can add a visual representation of the web-based external resource (e.g., box art or other graphics) to the interaction client (104). Once the user selects the visual representation or instructs the interaction client (104) to access the features of the web-based external resource through the GUI of the interaction client (104), the interaction client (104) obtains the HTML5 file and instantiates the resources to access the features of the web-based external resource.

[0049] The interaction client (104) presents a graphical user interface (e.g., a landing page or a title screen) for an external resource. While presenting the landing page or title screen, or before or after, the interaction client (104) determines whether the launched external resource has previously been authorized to access the interaction client (104)'s user data. In response to the determination that the launched external resource has previously been authorized to access the interaction client (104)'s user data, the interaction client (104) presents another graphical user interface of the external resource containing the functions and features of the external resource. In response to the determination that the launched external resource has not previously been authorized to access the interaction client (104)'s user data, after a threshold period (e.g., 3 seconds) of displaying the external resource's landing page or title screen, the interaction client (104) slides up a menu to authorize the external resource to access user data (e.g., animates the menu as it surfaces from the bottom of the screen to the middle or another part of the screen). The menu identifies the type of user data to be authorized for use by an external resource. In response to receiving a user selection of an accept option, the interaction client (104) adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client (104). The external resource is authorized by the interaction client (104) to access user data under the OAuth 2 framework.

[0050] The interaction client (104) controls the type of user data shared with external resources based on the type of external resource authorized. For example, external resources including full-scale applications (e.g., application (106)) are provided with access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics). As another example, external resources including small-scale versions of applications (e.g., web-based versions of applications) are provided with access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the look and feel of the avatar, such as different poses, facial features, clothing, etc.

[0051] The advertising system (228) operatively enables the purchase of advertisements by third parties to be presented to end users through the interacting clients (104) and also handles the delivery and presentation of such advertisements.

[0052] The artificial intelligence and machine learning system (230) provides various services to different subsystems within the interaction system (100). For example, the artificial intelligence and machine learning system (230) works in conjunction with the image processing system (202) and the camera system (204) to analyze images and extract information such as objects, text, or faces. This information can then be used by the image processing system (202) to enhance, filter, or manipulate the images. The artificial intelligence and machine learning system (230) can be used to create augmented content and augmented reality experiences, such as adding virtual objects or animations to real-world images by the augmentation system (206). The communication system (208) and the messaging system (210) can use the artificial intelligence and machine learning system (230) to analyze communication patterns, provide insights into how users interact with each other, and provide intelligent message classification and tagging, such as categorizing messages based on emotions or topics. The artificial intelligence and machine learning system (230) can also provide chatbot functionality for message interactions (120) between user systems (102) and between user systems (102) and interaction server systems (110). The artificial intelligence and machine learning system (230) can also work with an audio communication system (216) to provide speech recognition and natural language processing capabilities, enabling users to interact with the interaction system (100) using voice commands.

[0053] Data Architecture

[0054] FIG. 3 is a schematic diagram illustrating data structures (300) that may be stored in a database (304) of an interactive server system (110) according to specific examples. Although the contents of the database (304) are depicted as including multiple tables, it will be recognized that data may be stored in data structures of other types (e.g., as an object-oriented database).

[0055] The database (304) contains message data stored in the message table (306). For any particular message, this message data includes at least message sender data, message recipient (or receiver) data, and payload. Additional details regarding information that may be included in the message and may be included in the message data stored in the message table (306) are described below with reference to FIG. 3.

[0056] The entity table (308) stores entity data and is linked (e.g., for reference) to the entity graph (310) and profile data (302). Entities for which records are maintained within the entity table (308) may include individuals, legal entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity for which the interaction server system (110) stores data may be a recognized entity. Each entity is provided with an entity type identifier (not shown) as well as a unique identifier.

[0057] The entity graph (310) stores information regarding relationships and associations between entities. Such relationships may be based on interests, for example, social, professional (for example, work in a general corporation or organization), or activity. Certain relationships between entities may be unidirectional, such as an individual user subscribing to digital content from a commercial or publishing user (for example, a newspaper or other digital media outlet, or a brand). Other relationships may be bidirectional, such as "friend" relationships between individual users of the interaction system (100).

[0058] Specific permissions and relationships may be granted to each relationship and also to each direction of the relationship. For example, a bidirectional relationship (e.g., a friend relationship between individual users) may include authorization for the posting of digital content items between individual users, but may impose specific restrictions or filters on the posting of these digital content items (e.g., based on content characteristics, location data, or time of day data). Similarly, a subscription relationship between an individual user and a commercial user may impose varying degrees of restrictions on the posting of digital content from a commercial user to an individual user, and may significantly restrict or block the posting of digital content from an individual user to a commercial user. As an example of an entity, a specific user may record specific restrictions (e.g., by privacy settings) in a record for that entity within the entity table (308). These privacy settings may be applied to all types of relationships within the context of the interaction system (100), or may be applied selectively to specific types of relationships.

[0059] Profile data (302) stores multiple types of profile data for a specific entity. Profile data (302) may be selectively used and presented to other users of the interaction system (100) based on privacy settings specified by the specific entity. If the entity is an individual, profile data (302) includes, for example, a username, phone number, address, settings (e.g., notification and privacy settings), as well as a user-selected avatar representation (or a collection of such avatar representations). Then, a specific user may selectively include one or more of these avatar representations within the content of messages communicated through the interaction system (100) and on map interfaces displayed to other users by the interaction clients (104). A collection of avatar representations may include "state avatars" that present a graphic representation of a state or activity that the user may select to communicate at a specific time.

[0060] If the entity is a group, the profile data (302) for the group may similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the related group.

[0061] The database (304) also stores augmentation data, such as overlays or filters, in the augmentation table (312). The augmentation data is associated with and applied to videos (data about which is stored in the video table (314)) and images (data about which is stored in the image table (316)).

[0062] In some examples, the filters are overlays that are displayed overlaid on an image or video during presentation to the recipient user. The filters may be various types of filters, including user-selected filters from a set of filters presented to the sender user by the interaction client (104) when the sender user is composing a message. Other types of filters include geolocation filters (also known as geo-filters) that may be presented to the sender user based on geolocation. For example, geolocation filters specific to a neighborhood or a particular location may be presented within the user interface by the interaction client (104) based on geolocation information determined by the Global Positioning System (GPS) unit of the user system (102).

[0063] Another type of filter is a data filter that may be optionally presented to the sending user by the interaction client (104) based on other inputs or information collected by the user system (102) during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is moving, the battery life of the user system (102), or the current time.

[0064] Other augmented data that may be stored in the image table (316) includes augmented reality content items (e.g., "lenses" or corresponding to applying augmented reality experiences). Augmented reality content items may be real-time special effects and sounds that can be added to an image or video.

[0065] The collection table (318) stores data regarding collections of messages and associated image, video, or audio data that are compiled into collections (e.g., stories or galleries). The creation of a specific collection may be initiated by a specific user (e.g., each user for whom a record is maintained in the entity table (308)). The user may create a “personal story” in the form of a collection of content created and transmitted / broadcasted by that user. To this end, the user interface of the interaction client (104) may include a user-selectable icon to enable the transmitting user to add specific content to their personal story.

[0066] The collection may also constitute a “live story,” which is a collection of content from multiple users generated manually, automatically, or using a combination of manual and automatic techniques. For example, a “live story” may constitute a curated stream of user-submitted content from various locations and events. Users who have location service-enabled client devices and are at a common location event at a specific time may be presented with an option to contribute content to a specific live story, for example, through the user interface of an interactive client (104). A live story may be identified to the user by the interactive client (104) based on their location. The final result is a “live story” as described in community terms.

[0067] An additional type of content collection is known as a “location story” that enables a user having a user system (102) located within a specific geographic location (e.g., a college or university campus) to contribute to a specific collection. In some examples, contribution to a location story may utilize a second degree of authentication to verify whether the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0068] As mentioned above, the video table (314) stores video data associated with messages for which records are maintained in the message table (306) in some examples. Similarly, the image table (316) stores image data associated with messages for which message data is stored in the entity table (308). The entity table (308) can associate various augmentations from the augmentation table (312) with various images and videos stored in the image table (316) and the video table (314).

[0069] Data communication architecture

[0070] FIG. 4 is a schematic diagram illustrating the structure of a message (400), according to some examples, generated by an interaction client (104) to communicate with an additional interaction client (104) via interaction servers (124). The content of a specific message (400) is used to populate a message table (306) stored in a database (304) accessible by the interaction servers (124). Similarly, the content of the message (400) is stored in memory as "in-transit" or "in-flight" data of the user system (102) or the interaction servers (124). The message (400) is illustrated as including the following exemplary components:

[0071] ● Message identifier (402): A unique identifier that identifies the message (400).

[0072] ● Message text payload (404): Text generated by the user through the user interface of the user system (102) and included in the message (400).

[0073] ● Message image payload (406): Image data included in the message (400) that is captured by the camera component of the user system (102) or retrieved from the memory component of the user system (102). Image data for the transmitted or received message (400) may be stored in an image table (316).

[0074] ● Message video payload (408): Video data captured by a camera component or retrieved from a memory component of the user system (102) and included in the message (400). Video data for the transmitted or received message (400) may be stored in an image table (316).

[0075] ● Message audio payload (410): Audio data captured by a microphone or retrieved from a memory component of the user system (102) and included in the message (400).

[0076] ● Message augmentation data (412): Augmentation data (e.g., filters, stickers, or other annotations or enhancements) representing augmentations to be applied to the message image payload (406), message video payload (408), or message audio payload (410) of the message (400). Augmentation data for a transmitted or received message (400) may be stored in an augmentation table (312).

[0077] ● Message duration parameter (414): A parameter value indicating the amount of time in seconds during which the content of a message (e.g., message image payload (406), message video payload (408), message audio payload (410)) is presented to or made accessible to the user through the interaction client (104).

[0078] ● Message geolocation parameter (416): Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter (416) values ​​may be included in the payload, and each of these parameter values ​​is associated with content items included in the content (e.g., a specific image in the message image payload (406), or a specific video in the message video payload (408)).

[0079] ● Message Story Identifier (418): Identifier values ​​that identify one or more content collections (e.g., “stories”) identified in the collection table (318) to which a specific content item within the message image payload (406) of the message (400) is associated. For example, multiple images within the message image payload (406) may each be associated with multiple content collections using identifier values.

[0080] ● Message Tag (420): Each message (400) may be tagged with multiple tags, each of which indicates the subject of the content included in the message payload. For example, if a specific image included in the message image payload (406) depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tag (420). The tag values ​​may be generated manually based on user input or automatically using, for example, image recognition.

[0081] ● Message sender identifier (422): An identifier representing the user of the user system (102) to which the message (400) was created and the message (400) was sent (e.g., a messaging system identifier, an email address, or a device identifier).

[0082] ● Message recipient identifier (424): An identifier representing the user of the user system (102) to which the message (400) is addressed (e.g., a messaging system identifier, an email address, or a device identifier).

[0083] The contents (e.g., values) of the various components of the message (400) may be pointers to locations within tables where content data values ​​are stored. For example, an image value within the message image payload (406) may be a pointer (or its address) to a location within the image table (316). Similarly, values ​​within the message video payload (408) may point to data stored within the image table (316), values ​​within the message augmentation data (412) may point to data stored within the augmentation table (312), values ​​within the message story identifier (418) may point to data stored within the collection table (318), and values ​​within the message sender identifier (422) and message receiver identifier (424) may point to user records stored within the entity table (308).

[0084] System having a head-wearable device

[0085] FIG. 5 illustrates a system (500) comprising a head-wearable device (116) having a selector input device according to some examples. FIG. 5 is a high-level functional block diagram of an exemplary head-wearable device (116) coupled to a mobile device (114) and various server systems (504) (e.g., an interactive server system (110)) via various networks (108).

[0086] The head-wearable device (116) includes one or more cameras, each of which may be, for example, a visible light camera (506), an infrared emitter (508), and an infrared camera (510).

[0087] The mobile device (114) connects to the head-wearable device (116) using both a low-power wireless connection (512) and a high-speed wireless connection (514). The mobile device (114) is also connected to a server system (504) and a network (516).

[0088] The head-wearable device (116) further includes two image displays of the image display (518) of the optical assembly. The two image displays (518) of the optical assembly include one associated with the left side of the head-wearable device (116) and one associated with the right side. The head-wearable device (116) also includes an image display driver (520), an image processor (522), a low-power circuit (524), and a high-speed circuit (526). The image display (518) of the optical assembly is intended to present images and videos, including images that may include a graphical user interface, to the user of the head-wearable device (116).

[0089] The image display driver (520) commands and controls the image display (518) of the optical assembly. The image display driver (520) can directly transmit image data to the image display (518) of the optical assembly for presentation, or convert the image data into a signal or data format suitable for transmitting to an image display device. For example, the image data may be video data formatted according to compression formats such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, ​​etc., and the still image data may be formatted according to compression formats such as PNG (Portable Network Group), JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), or EXIF ​​(exchangeable image file format).

[0090] The head-wearable device (116) comprises a frame and stems (or temples) extending from the sides of the frame. The head-wearable device (116) further comprises a user input device (528) (e.g., a touch sensor or a push button) comprising an input surface on the head-wearable device (116). The user input device (528) (e.g., a touch sensor or a push button) is intended to receive an input selection from the user for manipulating a graphical user interface of a presented image.

[0091] The components illustrated in FIG. 5 for the head-wearable device (116) are located on one or more circuit boards, e.g., a PCB or a flexible PCB, in the rims or temples. Alternatively or additionally, the described components may be located in chunks, frames, hinges, or bridges of the head-wearable device (116). The left and right visible light cameras (506) may include any other visible or light capturing elements that can be used to capture data, such as digital camera elements, e.g., a CMOS (complementary metal oxide-semiconductor) image sensor, a charge-coupled device, camera lenses, or images of scenes having unknown objects.

[0092] The head-wearable device (116) includes a memory (502) for storing instructions for performing a subset or all of the functions described herein. The memory (502) may also include a storage device.

[0093] As illustrated in FIG. 5, the high-speed circuit (526) includes a high-speed processor (530), memory (502), and a high-speed wireless circuit (532). In some examples, an image display driver (520) is coupled to the high-speed circuit (526) and is operated by the high-speed processor (530) to drive the left and right image displays of the image display (518) of the optical assembly. The high-speed processor (530) may be any processor capable of managing high-speed communication and operation of any general computing system required for the head-wearable device (116). The high-speed processor (530) includes processing resources required to manage high-speed data transmissions in a high-speed wireless connection (514) to a wireless local area network (WLAN) using the high-speed wireless circuit (532). In certain examples, the high-speed processor (530) runs an operating system such as a LINUX operating system or other such operating system of the head-wearable device (116), and the operating system is stored in memory (502) for execution. In addition to any other responsibilities, a high-speed processor (530) executing the software architecture for the head-wearable device (116) is used to manage data transmissions with the high-speed wireless circuit (532). In certain examples, the high-speed wireless circuit (532) is configured to implement the IEEE (Institute of Electrical and Electronic Engineers) 802.11 communication standard, also referred to herein as WI-FI®. In some examples, other high-speed communication standards may be implemented by the high-speed wireless circuit (532).

[0094] The low-power wireless circuit (534) and high-speed wireless circuit (532) of the head-wearable device (116) may include short-range transceivers (Bluetooth™) and wireless wide-area, local area, or wide-area network transceivers (e.g., cellular or WI-FI®). A mobile device (114) including transceivers communicating via the low-power wireless connection (512) and the high-speed wireless connection (514) may be implemented using the details of the architecture of the head-wearable device (116), as well as other elements of the network (516).

[0095] The memory (502) includes any storage device capable of storing various data and applications, including, in particular, camera data generated by the left and right visible light cameras (506), the infrared camera (510), and the image processor (522), as well as images generated by the image display driver (520) to be displayed on the image displays (518) of the optical assembly. Although the memory (502) is depicted as being integrated with the high-speed circuit (526), ​​in some examples, the memory (502) may be an independent, standalone element of the head-wearable device (116). In these specific examples, electrical routing lines may provide a connection from the image processor (522) or the low-power processor (536) to the memory (502) through a chip including the high-speed processor (530). In some examples, the high-speed processor (530) can manage the addressing of memory (502) so that the low-power processor (536) boots the high-speed processor (530) whenever a read or write operation involving memory (502) is required.

[0096] As illustrated in FIG. 5, a low-power processor (536) or a high-speed processor (530) of a head-wearable device (116) may be coupled to a camera (visible light camera (506), infrared emitter (508), or infrared camera (510)), an image display driver (520), a user input device (528) (e.g., a touch sensor or a push button), and a memory (502).

[0097] The head-wearable device (116) is connected to a host computer. For example, the head-wearable device (116) is paired with a mobile device (114) via a high-speed wireless connection (514) or connected to a server system (504) via a network (516). The server system (504) may be one or more computing devices as part of a service or network computing system, for example, including a processor, memory, and a network communication interface for communicating with the mobile device (114) and the head-wearable device (116) via the network (516).

[0098] The mobile device (114) includes a processor and a network communication interface coupled to the processor. The network communication interface allows communication via a network (516), a low-power wireless connection (512), or a high-speed wireless connection (514). The mobile device (114) may additionally store at least a portion of instructions in memory of the mobile device (114) to implement the functionality described herein.

[0099] The output components of the head-wearable device (116) include visual components such as a liquid crystal display (LCD), a plasma display panel (PDP), a light-emitting diode (LED) display, a projector, or a waveguide. The image displays of the optical assembly are driven by an image display driver (520). The output components of the head-wearable device (116) further include acoustic components (e.g., speakers), haptic components (e.g., vibration motors), other signal generators, etc. Input components of a head-wearable device (116), a mobile device (114), and a server system (504), such as a user input device (528), may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing mechanism), haptic input components (e.g., a physical button, a touch screen providing the location and force of a touch or touch gesture, or other haptic input components), audio input components (e.g., a microphone), etc.

[0100] The head-wearable device (116) may also include additional peripheral device elements. These peripheral device elements may include biometric sensors, additional sensors, or display elements integrated with the head-wearable device (116). For example, the peripheral device elements may include any I / O components including output components, motion components, position components, or any other such elements described herein.

[0101] For example, biometric components include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or EEG-based identification), and performing similar tasks. Biometric components may include a brain-machine interface (BMI) system that allows communication between the brain and an external device or machine. This can be achieved by recording brain activity data, translating this data into a format that a computer can understand, and then using the resulting signals to control the device or machine.

[0102] Exemplary types of BMI technologies include the following:

[0103] ● Electroencephalography (EEG)-based BMIs that record electrical activity in the brain using electrodes placed on the scalp.

[0104] ● Invasive BMIs using electrodes surgically implanted in the brain.

[0105] ● Optogenetic BMIs that use light to control the activity of specific nerve cells in the brain.

[0106] Any biometric data collected by biometric components is captured and stored solely with user authorization and deleted upon user request. Additionally, such biometric data may be used for very limited purposes, such as identification verification. To ensure the limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is restricted, at best, to authorized personnel. Any use of biometric data may be strictly limited to identification purposes, and biometric data is not shared or sold to any third party without the user's explicit consent. Furthermore, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of such sensitive information.

[0107] Motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Position components include position sensor components that generate position coordinates (e.g., Global Positioning System (GPS) receiver components), Wi-Fi or Bluetooth™ transceivers that generate positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc. These positioning system coordinates may also be received from the mobile device (114) via low-power wireless circuits (534) or high-speed wireless circuits (532) via low-power wireless connections (512) and high-speed wireless connections (514).

[0108] The following terms or phrases refer to the components and embodiments mentioned in this specification and discussed at least in FIG. 6 below.

[0109] ● Input Image:

[0110] o An image containing the user's face. It serves as a source of identity and optionally generates some conditional information from it.

[0111] o As machine learning models processing human image(s), identity refers to a set of distinct features and attributes that uniquely identify that person. These models can be trained to extract robust representations of facial features from a set of images, so that the identity representation can be preserved over subsequent operations that modify or transform these features or attributes of the identity.

[0112] ● Text Condition:

[0113] o Specific text that defines the desired style and content of the generated image. In that case, it is used in a text-to-image model (e.g., a diffusion model).

[0114] o Example: "Photo of a man wearing a football uniform, detail, sports"

[0115] ● Pose Template:

[0116] o An image having geometry to be preserved (e.g., applied) on the generated image.

[0117] For example, a pose template could be a man standing with his arms at his sides.

[0118] ● Diffusion Model:

[0119] o A neural network that generates an image from noise. It may also take text conditions as input to control the generation style and may also take the output of additional models (e.g., the identity insertion component or geometry control component described above) to preserve geometry and facial features on the generated image.

[0120] ● Intermediate products:

[0121] o Output of a diffusion model conditioned by text, an identity insertion component, and a geometry control component. The output has geometry similar to the pose template, style and content similar to the text condition, and identity features similar to the input image identity.

[0122] ● Result Image:

[0123] As the output of the entire pipeline, style and content are defined by text conditions, geometry by pose templates, and identity by identity insertion components and face networks.

[0124] FIG. 6 illustrates an exemplary processing flow of a generative pipeline using a single selfie ("one shot") input according to embodiments of the present technology.

[0125] In the example of Fig. 6, the processing flow can be described as follows.

[0126] As illustrated, the user provides a selfie (e.g., a self-portrait corresponding to the input image (602)). In preparation for a subsequent step (discussed further below) related to ensuring identity preservation, the identity insertion component (604) processes this selfie (e.g., the input image) through a neural network (or other machine learning model(s)) and converts the selfie into a latent representation of the identity.

[0127] The text condition (608) can be understood as a text description of the desired style and content in the resulting image. The text condition (608) is converted using a diffusion model (606) into a potential representation compatible with the representation obtained from the identity insertion component (604) discussed above.

[0128] A pose template (607) (e.g., an image having a pose to be represented in a resulting image) is processed by a geometry control component (610) and converted into a set of latent features for a diffusion model (606).

[0129] The outputs of the identity insertion component (604), geometry control component (610), and text condition (608) are transmitted (i.e., forwarded) to the diffusion model (606), where the diffusion model (606) generates an intermediate generation (611) containing an image having styles, facial features, and geometry defined (as potential representations) in the previously discussed previous operations.

[0130] To ensure identity preservation, the intermediate product (611) is passed to a Face Enhancement Network (612), and the Face Enhancement Network (612) also receives, as input, an original selfie (e.g., input image (602)) and further restores the identity from the (intermediate) image (e.g., performs a set of modification actions).

[0131] In one embodiment, the output of this process is provided as input to a Face Restoration Network (614) that restores details of the facial expression and removes artifacts from the previous operations discussed above. A result image (616) corresponding to the final output image is provided as output of an exemplary processing flow, which can be utilized by one or more client devices (e.g., a mobile device (114), a user system (102), a computer client device (118), etc.) or appropriate application(s) running thereon.

[0132] The following discussion explains additional details regarding the various components discussed above, at least in Fig. 6.

[0133] In the implementation, the diffusion model (606) utilizes techniques based on high-resolution image synthesis using potential diffusion models and modifies the text condition (608). During the denoising process performed by the diffusion model (606), the noise is converted into an image based on the text description provided by the text condition (606).

[0134] In one embodiment, the identity insertion module (604) converts the input image into a latent representation compatible with the diffusion model (606) to preserve the face attributes of the input image in the final product (e.g., before or while providing the final output image discussed above).

[0135] In one embodiment, the geometry control module (610) receives geometry information from a pose control image (e.g., a pose template (607)) and converts the geometry into intermediate latent representations used by a diffusion model (606) so that the resulting image has geometry substantially similar to the pose control image (e.g., a pose template (607)). In the implementation, techniques based on adding conditional control to text-to-image diffusion models may be utilized.

[0136] In one embodiment, the face enhancement network (612) uses the user's selfie as a reference to enhance the identity on the image.

[0137] In one embodiment, the face restoration network (614) performs the task of improving details on the resulting image and reducing the number of artifacts.

[0138] FIG. 7 illustrates exemplary interfaces according to embodiments of the present technology. The exemplary interfaces may be provided to be displayed on a client device (e.g., mobile device (114), head-wearable device (116), computer client device (118), etc.) through interface(s) such as a user system (102), an interactive client (104), and an application (106) that perform the processing flow discussed in FIG. 6 at least.

[0139] As described, the interface (702) includes an initial screen that may be provided to be displayed when an application is initially loaded (e.g., when execution starts) on a given client device.

[0140] In interface (704), a set of graphic elements displays a countdown after starting a generative processing flow (e.g., as discussed in FIG. 6 above). In interface (706), a screen is shown having a set of effects (e.g., colorful fuzzy loading) indicating that generative processing is currently in progress.

[0141] FIG. 8 illustrates exemplary interfaces according to embodiments of the present technology. The exemplary interfaces may be provided to be displayed on a client device (e.g., mobile device (114), head-wearable device (116), computer client device (118), etc.) through interface(s) such as a user system (102), an interactive client (104), and an application (106) that perform the processing flow discussed in FIG. 6 at least.

[0142] The exemplary interfaces of Fig. 8 are a continuation of the discussion of the interfaces from Fig. 7 discussed above.

[0143] As described, the interface (802) displays an animation for stylization to appear (provided, for example, to be displayed on a given client device). In the interface (804), an output image corresponding to the result (final) image of the aforementioned generative process (e.g., as discussed in FIG. 6) is provided to be displayed on a client device.

[0144] FIG. 9 illustrates examples of output media content provided by a one-shot personalized generative pipeline according to one embodiment. The exemplary output media content may be provided for display on a client device (e.g., mobile device (114), head-wearable device (116), computer client device (118), etc.) through interface(s) such as a user system (102), an interactive client (104), and an application (106), which perform at least the processing flow discussed in FIG. 6.

[0145] As illustrated, the input image (910) undergoes processing in the generative processing flow discussed in at least FIG. 6. Various output images (output image (912), output image (914), output image (916), output image (918)) are generated using different provided text conditions (text condition (920), text condition (922), text condition (924), text condition (926)).

[0146] FIG. 10 illustrates examples of output media content provided by a one-shot personalized generative pipeline according to one embodiment. The exemplary output media content may be provided for display on a client device (e.g., mobile device (114), head-wearable device (116), computer client device (118), etc.) through interface(s) such as a user system (102), an interactive client (104), and an application (106), which perform the processing flow discussed in FIG. 6 at least.

[0147] As illustrated, the input image (1004) undergoes processing in the generative processing flow discussed in at least FIG. 6. Various output images (output image (1006), output image (1008), output image (1010), output image (1012)) are generated using different provided text conditions (text condition (1020), text condition (1022), text condition (1024), text condition (1026)).

[0148] FIG. 11 is a flowchart illustrating a method according to specific exemplary embodiments. The method may be embodied in computer-readable instructions for execution by one or more computer processors so that the operations of the method can be performed partially or wholly by a computer client device (118). However, it will be understood that at least some of the operations of the method may be placed on various other hardware configurations (e.g., a mobile device (114), a head-wearable device (116)), and that the method is not intended to be limited to the computer client device (118) or any of the components or systems mentioned above.

[0149] In operation 1102, the computer client device (118) receives an input image, and this input image includes a selfie.

[0150] In operation 1104, the computer client device (118) uses a neural network to convert an input image into a latent representation of an identity.

[0151] In operation 1106, the computer client device (118) uses a diffusion model to convert text conditions into a second potential representation compatible with a potential representation of identity.

[0152] In operation 1108, the computer client device (118) converts the pose template into a set of latent features for a diffusion model.

[0153] In operation 1110, the computer client device (118) generates an intermediate image based on a potential representation of the identity, a second potential representation, and a set of potential features.

[0154] In operation 1112, the computer client device (118) uses a face enhancement network to modify an intermediate image based on an input image.

[0155] In operation 1114, the computer client device (118) uses a face restoration network to generate a final output image based on a modified intermediate image.

[0156] In operation 1116, the computer client device (118) is provided to display the final output image on the display of the client device.

[0157] Machine Architecture

[0158] FIG. 12 is a schematic representation of a machine (1200) on which instructions (1202) (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine (1200) to perform any one or more of the methodologies discussed herein. For example, instructions (1202) can cause the machine (1200) to perform any one or more of the methods described herein. Instructions (1202) convert a general non-programmed machine (1200) into a specific machine (1200) programmed to perform the described and illustrated functions in the described manner. The machine (1200) may operate as a standalone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine (1200) may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine (1200) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (1202) that specify actions to be taken by the machine (1200) sequentially or otherwise.Furthermore, although a single machine (1200) is exemplified, the term “machine” should also be considered to include a collection of machines that execute instructions (1202) individually or jointly to perform any one or more of the methodologies discussed herein. The machine (1200) may include, for example, any one of a plurality of server devices forming part of a user system (102) or an interactive server system (110). In some examples, the machine (1200) may also include both client and server systems, where specific operations of a specific method or algorithm are performed on the server side and specific operations of a specific method or algorithm are performed on the client side.

[0159] The machine (1200) may include processors (1204), memory (1206), and I / O (input / output) components (1208) that can be configured to communicate with each other via a bus (1210). In one example, the processors (1204) (e.g., a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), other processors, or any suitable combination thereof) may include, for example, a processor (1212) and a processor (1214) that execute instructions (1202). The term “processor” is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as “cores”) capable of executing instructions simultaneously. FIG. 12 illustrates multiple processors (1204), but the machine (1200) may include a single processor having a single core, a single processor having multiple cores (e.g., a multi-core processor), multiple processors having a single core, multiple processors having multiple cores, or any combination thereof.

[0160] Memory (1206) includes main memory (1216), static memory (1218), and a storage unit (1220), both of which are accessible to processors (1204) via a bus (1210). The main memory (1206), static memory (1218), and storage unit (1220) store instructions (1202) that implement any one or more of the methodologies or functions described herein. The instructions (1202) may also exist, wholly or partially, during execution by the machine (1200), in the main memory (1218), in the static memory (1216), in a machine-readable medium (1222) within the storage unit (1220), in at least one of the processors (1204) (e.g., in the processor's cache memory), or any suitable combination thereof.

[0161] The I / O components (1208) may include a wide variety of components for receiving inputs, providing outputs, producing outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (1208) included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones may include touch input devices or other such input mechanisms, while headless server machines are unlikely to include such touch input devices. It will be acknowledged that the I / O components (1208) may include many other components not shown in FIG. 12. In various examples, the I / O components (1208) may include user output components (1224) and user input components (1226). User output components (1224) may include visual components (e.g., displays such as a PDP (plasma display panel), LED (light-emitting diode) display, LCD (liquid crystal display), projector, or CRT (cathode ray tube)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. User input components (1226) may include alphanumeric input components (e.g., keyboard, touch screen configured to receive alphanumeric input, photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), haptic input components (e.g., physical button, touch screen providing location and force of touch or touch gesture, or other haptic input components), audio input components (e.g., microphone), etc.

[0162] In additional examples, I / O components (1208) may include biometric components (1228), motion components (1230), environment components (1232), or position components (1234), among a wide variety of other components. For example, biometric components (1228) include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or electroencephalogram-based identification), and performing similar tasks. Biometric components may include a brain-machine interface (BMI) system that allows communication between the brain and an external device or machine. This can be achieved by recording brain activity data, translating this data into a format that a computer can understand, and then using the resulting signals to control a device or machine.

[0163] Exemplary types of BMI technologies include the following:

[0164] ● Electroencephalography (EEG)-based BMIs that record electrical activity in the brain using electrodes placed on the scalp.

[0165] ● Invasive BMIs using electrodes surgically implanted in the brain.

[0166] ● Optogenetic BMIs that use light to control the activity of specific nerve cells in the brain.

[0167] Any biometric data collected by biometric components is captured and stored solely with user authorization and deleted upon user request. Additionally, such biometric data may be used for very limited purposes, such as identification verification. To ensure the limited and authorized use of biometric information and other personally identifiable information (PII), access to this data is restricted, at best, to authorized personnel. Any use of biometric data may be strictly limited to identification purposes, and this data is not shared or sold to any third party without the user's explicit consent. Furthermore, appropriate technical and organizational measures are implemented to ensure the security and confidentiality of such sensitive information.

[0168] Motion components (1230) include acceleration sensor components (e.g., accelerometer), gravity sensor components, and rotation sensor components (e.g., gyroscope).

[0169] Environmental components (1232) may include, for example, one or more cameras (having still image / photo and video capabilities), light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or measuring pollutants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment.

[0170] With respect to cameras, the user system (102) may have a camera system including, for example, front cameras located at the front of the user system (102) and rear cameras located at the rear of the user system (102). The front cameras may be used, for example, to capture still images and videos (e.g., "selfies") of the user of the user system (102), which may then be augmented with the augmentation data (e.g., filters) described above. The rear cameras may be used, for example, to capture still images and videos in a more traditional camera mode, and these images are similarly augmented with augmentation data. In addition to the front and rear cameras, the user system (102) may also include a 360° camera for capturing 360° photos and videos.

[0171] Additionally, the camera system of the user system (102) may include dual rear cameras (e.g., a depth-sensing camera as well as a main camera) on the front and rear sides of the user system (102), or even triple, quadruple, or quintuple rear camera configurations. These multiple camera systems may include, for example, a wide camera, an ultra-wide camera, a telephoto camera, a macro camera, and a depth sensor.

[0172] Position components (1234) include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and similar ones.

[0173] Communication can be implemented using a wide variety of technologies. The I / O components (1208) further include communication components (1236) operable to connect the machine (1200) to a network (1238) or devices (1240) through their respective combinations or connections. For example, the communication components (1236) may include a network interface component, or other suitable device for interfacing with the network (1238). In additional examples, the communication components (1236) may include wired communication components, wireless communication components, cellular communication components, near-field communication (NFC) components, Bluetooth ® Components (e.g., Bluetooth) ® Low Energy), Wi-Fi ® It may include other communication components that provide communication through components and other aspects. The devices (1240) may be other machines or any various peripheral devices (e.g., peripheral devices connected via USB).

[0174] Furthermore, the communication components (1236) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (1236) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph™, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). In addition, various information, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location via detection of NFC beacon signals that can indicate a specific location, can be derived through the communication components (1236).

[0175] Various memories (e.g., main memory (1216), static memory (1218), and memory of the processors (1204)) and storage unit (1220) may store one or more sets of instructions and data structures (e.g., software) that implement any one or more of the methodologies or functions described herein. These instructions (e.g., instructions (1202)) enable various operations to implement the disclosed examples when executed by the processors (1204).

[0176] Commands (1202) may be transmitted or received over a network (1238) using a transmission medium through a network interface device (e.g., a network interface component included in the communication components (1236)) and using any one of several well-known transmission protocols (e.g., HTTP (hypertext transfer protocol)). Similarly, commands (1202) may be transmitted or received using a transmission medium through a connection to the devices (1240) (e.g., peer-to-peer connection).

[0177] Software Architecture

[0178] FIG. 13 is a block diagram (1300) illustrating a software architecture (1302) that may be installed in any one or more of the devices described herein. The software architecture (1302) is supported by hardware such as a machine (1304) including processors (1306), memory (1308), and I / O components (1310). In this example, the software architecture (1302) may be conceptualized as a stack of layers, each layer providing specific functionality. The software architecture (1302) includes layers such as an operating system (1312), libraries (1314), frameworks (1316), and applications (1318). Operationally, applications (1318) invoke API calls (1320) through the software stack and receive messages (1322) in response to the API calls (1320).

[0179] The operating system (1312) manages hardware resources and provides common services. The operating system (1312) includes, for example, a kernel (1324), services (1326), and drivers (1328). The kernel (1324) acts as an abstraction layer between the hardware and other software layers. For example, the kernel (1324) provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionalities. Services (1326) may provide other common services for other software layers. Drivers (1328) are responsible for controlling or interfacing with the underlying hardware. For example, drivers (1328) may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, etc.

[0180] Libraries (1314) provide common low-level infrastructure used by applications (1318). Libraries (1314) may include system libraries (1330) (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the libraries (1314) may include API libraries (1332) such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4 (Moving Picture Experts Group-4), Advanced Video Coding (H.264 or AVC), MP3 (Moving Picture Experts Group Layer-3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codecs, Joint Photographic Experts Group (JPEG or JPG), or PNG (Portable Network Graphics), graphics libraries (e.g., OpenGL frameworks used to render graphic content on a display in two dimensions (2D) and three dimensions (3D)), database libraries (e.g., SQLite providing various relational database functions), web libraries (e.g., WebKit providing web browsing functionality). The libraries (1314) may also include a wide variety of other libraries (1334) to provide many different APIs to applications (1318).

[0181] Frameworks (1316) provide common high-level infrastructure used by applications (1318). For example, frameworks (1316) provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. Frameworks (1316) may provide a wide range of other APIs that can be used by applications (1318), some of which may be specific to a particular operating system or platform.

[0182] In one example, applications (1318) may include a wide range of other applications such as a home application (1336), a contact application (1338), a browser application (1340), a book reader application (1342), a location application (1344), a media application (1346), a messaging application (1348), a game application (1350), and a third-party application (1352). Applications (1318) are programs that execute functions defined in programs. Various programming languages ​​may be used to create one or more of applications (1318) that are structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (1352) (e.g., an application developed by an entity other than a vendor of a specific platform using an ANDROID™ or IOS™ software development kit (SDK)) may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third-party application (1352) may invoke API calls (1320) provided by the operating system (1312) to facilitate the functionalities described herein.

[0183] Glossary

[0184] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, for example, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0185] "Client device" refers to any machine that interfaces with a communication network to obtain resources from, for example, one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0186] "Communication network" refers to one or more parts of a network, which may be, for example, an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or other types of cellular or wireless coupling.In this example, the combination can implement any of the various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4th generation wireless (4G) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standard-setting organizations, other long-range protocols, or other data transmission technologies.

[0187] "Component" refers to a device, physical entity, or logic having boundaries defined by, for example, function or subroutine calls, branch points, APIs, or other techniques that provide the division or modularization of specific processing or control functions. Components may be combined with other components through their interfaces to execute machine processes. A component may be a packaged functional hardware unit designed to be used with other components and part of a program that performs a specific function among the related functions. Components may constitute either software components (e.g., code implemented on a machine-readable medium) or hardware components. "Hardware component" is a tangible unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., processors or groups of processors) may be configured by software (e.g., applications or parts of applications) as hardware components that operate to perform specific operations as described herein. Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits temporarily configured by software to perform specific operations.For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely customized to perform the configured functions and are no longer general-purpose processors. It will be recognized that the decision to implement a hardware component mechanically, in a dedicated, permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass tangible entities as long as they are physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular way or to perform the specific actions described herein. When considering examples where hardware components are temporarily configured (e.g., programmed), each hardware component does not need to be configured or instantiated at any single time instance. For example, in the case where a general-purpose processor is configured by software such that a hardware component becomes a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., including different hardware components). Thus, the software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance. Hardware components can provide information to other hardware components and receive information from them. Therefore, the described hardware components may be considered to be coupled in a communicable manner.In cases where multiple hardware components exist simultaneously, communication may be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information within memory structures accessible to multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device coupled to it for communication. Subsequently, additional hardware components may access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and manipulate resources (e.g., collections of information). Various operations of the exemplary methods described herein may be performed at least partially by one or more processors configured temporarily (e.g., by software) or permanently to perform the relevant operations. Whether configured temporarily or permanently, such processors may comprise processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be implemented at least partially by a processor, and specific processors or processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components.Furthermore, one or more processors may also operate to support the execution of related operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines containing processors), and these operations may be accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The execution of specific operations may not only reside within a single machine but may also be distributed among processors deployed across multiple machines. In some examples, the processors or components implemented by the processors may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other examples, the processors or components implemented by the processors may be distributed across multiple geographic locations.

[0188] "Computer-readable storage medium" refers to, for example, both machine storage media and transmission media. Accordingly, the terms include both storage devices / medias and carriers / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0189] An "ephemeral message" refers to a message accessible for, for example, a time-limited duration. An ephemeral message can be text, images, videos, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is transitory.

[0190] "Machine storage medium" refers to, for example, a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Accordingly, the term should be considered to include, but not be limited to, solid-state memories including memory inside or outside processors, and optical and magnetic media. Specific examples of machine storage medium, computer storage medium, and device storage medium include, for example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are included under the term "signal medium."

[0191] "Non-transient computer-readable storage medium" refers to a type of medium capable of storing, encoding, or carrying instructions for execution by a machine, for example.

[0192] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, for example, and includes digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or altered in matters such as encoding information within the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0193] "User device" refers to a device that is, for example, accessed, controlled, or owned by a user, on which the user performs actions or interactions, including interactions with other users or computer systems.

Claims

Claim 1 A method comprising: receiving an input image—the input image includes a selfie—; using a neural network to convert the input image into a latent representation of an identity; using a diffusion model to convert text conditions into a second latent representation compatible with the latent representation of the identity; converting a pose template into a set of latent features for the diffusion model; generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features; using a face enhancement network to modify the intermediate image based on the input image; using a face restoration network to generate a final output image based on the modified intermediate image; and providing for displaying the final output image on a display of a client device. Claim 2 A method according to claim 1, wherein the selfie includes a self-photograph containing a facial expression. Claim 3 A method according to claim 1, wherein the text condition includes text information defining the style and content of the intermediate image. Claim 4 A method according to claim 1, wherein the pose template comprises an image including geometry to be applied to generate the intermediate image. Claim 5 A method according to claim 1, wherein the diffusion model comprises a neural network that generates an image from noise based on the text conditions. Claim 6 In claim 1, the method wherein the intermediate image is generated by the diffusion model. Claim 7 A method according to claim 1, wherein the diffusion model converts noise into a specific image based on a text description from the text condition during a denoising process. Claim 8 A method according to claim 1, wherein the second potential representation compatible with the diffusion model preserves the set face attributes of the input image. Claim 9 A method according to claim 1, wherein geometry information from the pose template is utilized to convert the geometry into a set of latent representations used by the diffusion model, so that the final output image has geometry substantially similar to the pose template. Claim 10 A method according to claim 1, wherein a face enhancement network improves the identity on the intermediate image using the input image as a reference, and a face restoration network improves a set of details on the intermediate image and reduces the number of artifacts after being processed by the face enhancement network. Claim 11 A system comprising: a processor; and a memory comprising instructions that, when executed by the processor, cause the processor to perform operations, wherein the operations include: receiving an input image—the input image includes a selfie—; converting the input image into a latent representation of an identity using a neural network; converting text conditions into a second latent representation compatible with the latent representation of the identity using a diffusion model; converting a pose template into a set of latent features for the diffusion model; generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features; modifying the intermediate image based on the input image using a face enhancement network; generating a final output image based on the modified intermediate image using a face restoration network; and providing for displaying the final output image on a display of a client device. Claim 12 In claim 11, the system comprises a selfie including a self-photograph including a facial expression. Claim 13 In Clause 11, the system, wherein the text condition includes text information defining the style and content of the intermediate image. Claim 14 In claim 11, the system comprises a pose template including an image including geometry to be applied to generate the intermediate image. Claim 15 In claim 11, the above-mentioned diffusion model is a system comprising a neural network that generates an image from noise based on the above-mentioned text conditions. Claim 16 In paragraph 11, the system, wherein the intermediate image is generated by the diffusion model. Claim 17 In claim 11, the above diffusion model is a system that converts noise into a specific image based on a text description from the text condition during the noise removal process. Claim 18 In paragraph 11, the second latent representation compatible with the above diffusion model preserves the set face attributes of the input image, in a system. Claim 19 A system according to claim 11, which utilizes geometry information from the pose template to convert the geometry into a set of latent representations used by the diffusion model, thereby causing the final output image to have geometry substantially similar to the pose template. Claim 20 A non-transient computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations, wherein the operations include: receiving an input image—the input image includes a selfie—; converting the input image into a latent representation of an identity using a neural network; converting text conditions into a second latent representation compatible with the latent representation of the identity using a diffusion model; converting a pose template into a set of latent features for the diffusion model; generating an intermediate image based on the latent representation of the identity, the second latent representation, and the set of latent features; modifying the intermediate image based on the input image using a face enhancement network; generating a final output image based on the modified intermediate image using a face restoration network; and providing for displaying the final output image on a display of a client device.

Images