Generating virtual hairstyles using potential spatial projectors

Abstract

The technique generates a first image of a face using a GAN model. The technique applies 3D virtual hair on the first image to generate a second image having 3D virtual hair. The technique projects a second image with 3D virtual hair into the GAN potential space to generate a third image with realistic virtual hair. The present technique performs a blending of realistic virtual hair with a first image of a face to generate a new image having new realistic hair corresponding to 3D virtual hair. The techniques train a neural network that receives a second image with 3D virtual hair and provides an output image with realistic virtual hair. The techniques generate a particular output image with realistic hair based on a particular input image with 3D virtual hair using a trained neural network.

Description

[0001] Priority Claim

[0002] This patent application claims the benefit of priority of U.S. Application Serial No. 18 / 149,007, filed Dec. 30, 2022, which is hereby incorporated by reference in its entirety. Background of the Invention

[0003] With the increasing use of digital images, the affordability of portable computing devices, the availability of increased capacity of digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become a part of the daily lives of more and more people. Brief Description of the Drawings

[0004] In the drawings (which are not necessarily to scale), like reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, one or more of the most significant digits in the reference numeral refers to the figure number in which the element was first introduced. Some non-limiting examples are shown in the figures of the drawings, in which:

[0005] Figure 1 is a graphical representation of a networked environment in which the present disclosure may be deployed, according to some examples.

[0006] Figure 2 is a graphical representation of a messaging system having both client-side functionality and server-side functionality, according to some examples.

[0007] Figure 3 is a graphical representation of a data structure maintained in a database, according to some examples.

[0008] Figure 4 is a graphical representation of a message, according to some examples.

[0009] Figure 5 illustrates an example interface for an application for developing an augmented reality content generator for generating virtual hair (e.g., for providing an AR experience for virtual hair on a person's image or video).

[0010] Figure 6 illustrates an example processing flow of a computational architecture for projecting an input image into a latent space using a styleGAN (style generative adversarial network) generator, according to an embodiment of the present technology.

[0011] Figure 7 illustrates another example processing flow of a computational architecture for projecting an input image into a latent space using a styleGAN generator, according to an embodiment of the present technology.

[0012] Figure 8 Shows an example processing flow for generating virtual hair according to some embodiments of the present technology.

[0013] Figure 9 Is a flowchart showing a method according to certain example embodiments.

[0014] Figure 10 Is a graphical representation of a machine in the form of a computer system, within which a set of instructions can be executed to cause the machine to perform any one or more of the methods discussed herein.

[0015] Figure 11 Is a block diagram showing a software architecture within which examples can be implemented. Detailed Description

[0016] Users from various locations with various interests can capture digital images of various subjects and make the captured images available to others via a network such as the Internet. To enhance the user experience of digital images and provide various features, it can be challenging and computationally intensive for a computing device to perform image processing operations on various objects and / or features captured under various changing conditions (e.g., changes in image scale, noise, illumination, motion, or geometric distortion).

[0017] Augmented reality technology aims to bridge the gap between virtual and real-world environments by providing an enhanced real-world environment augmented with electronic information. Thus, the electronic information is like a part of the real-world environment perceived by the user. In an example, augmented reality technology also provides a user interface to interact with the electronic information superimposed on the enhanced real-world environment.

[0018] As mentioned above, with the increasing use of digital images, the affordability of portable computing devices, the increasing capacity of digital storage media, and the increasing bandwidth and accessibility of network connections, digital images have become a part of the daily lives of more and more people. Users from various locations with various interests can capture digital images of various subjects and make the captured images available to others via a network such as the Internet. To enhance the user experience of digital images and provide various features, it can be challenging and computationally intensive for a computing device to perform image processing operations on various objects and / or features captured under various changing conditions (e.g., changes in image scale, noise, illumination, motion, or geometric distortion).

[0019] Embodiments of the present technology implement virtual hair generation, which may include generating virtual hair to be applied to the head of a person in a video or image. Virtual hair generation can be used to manipulate content in many applications, such as entertainment shows, computer games, video conversations, virtual reality, augmented reality, and the like.

[0020] Messaging systems are frequently used and are increasingly used by users of mobile computing devices in various environments to provide different types of functionality in a convenient manner. As described herein, the present messaging system includes practical applications that provide improvements in capturing image data and rendering AR content (e.g., images, videos, etc.) based on the captured image data by providing technological improvements at least in capturing image data by power- and resource-constrained electronic devices. Such improvements in capturing image data are enabled by the technology provided by the present technology, which reduces latency and improves the efficiency of processing the captured image data, thereby also reducing the power consumption of the capture device.

[0021] As further discussed herein, the present infrastructure supports the creation and sharing of interactive media, referred to herein as messages, including 3D content or AR effects, among various components of the entire messaging system. In the example embodiments described herein, messages can enter the system from a live camera or via a storage device (e.g., where messages including 3D content and / or AR effects are stored in a memory or database). The system supports motion sensor input as well as the loading of external effects and asset data.

[0022] As mentioned herein, the phrases "augmented reality experience", "augmented reality content item", "augmented reality content generator" include or refer to various image processing operations corresponding to image modification, filtering, AR content generators, media overlays, transformations, etc., as further described herein, and may also include the playback of audio or music content during the presentation of AR content or media content.

[0023] Networked computing environment

[0024] Figure 1FIG. is a block diagram showing an example interaction system 100 for facilitating interactions on a network (e.g., exchanging text messages, making text, audio, and video calls, or playing games). Interaction system 100 includes a plurality of client systems 102, and each client system among the plurality of client systems 102 hosts a plurality of applications including an interaction client 104 and other applications 106. Each interaction client 104 is communicatively coupled via one or more communication networks including network 108 (e.g., the Internet) to other instances of interaction client 104 (e.g., hosted on corresponding other user systems 102), an interaction server system 110, and a third-party server 112. Interaction client 104 can also communicate with locally hosted applications 106 using an application programming interface (API).

[0025] Each user system 102 can include a plurality of user devices, such as mobile devices 114, head-mounted devices 116, and computer client devices 118, which are communicatively connected to exchange data and messages.

[0026] Interaction client 104 interacts via network 108 with other interaction clients 104 and with interaction server system 110. Data exchanged between interaction clients 104 (e.g., interaction 120) and between interaction client 104 and interaction server system 110 includes functionality (e.g., commands for activating functionality) and payload data (e.g., text, audio, video, or other multimedia data).

[0027] Interaction server system 110 provides server-side functionality to interaction client 104 via network 108. Although certain functions of interaction system 100 are described herein as being performed by interaction client 104 or by interaction server system 110, whether certain functions are located within interaction client 104 or within interaction server system 110 can be a design choice. For example, it may be technically preferable to initially deploy a particular technology and functionality within interaction server system 110, but later migrate the technology and functionality to interaction client 104 where the user system 102 has sufficient processing power.

[0028] Interaction server system 110 supports various services and operations provided to interaction client 104. Such operations include sending data to interaction client 104, receiving data from interaction client 104, and processing data generated by interaction client 104. The data can include message content, client device information, geographical location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. Data exchange within interaction system 100 is activated and controlled via functionality available through the user interface (UI) of interaction client 104.

[0029] Now turning specifically to the interaction server system 110, an application programming interface (API) server 122 is coupled to the interaction server 124 and provides a programming interface thereto, making the functions of the interaction server 124 accessible to the interaction client 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126, thereby facilitating access to a database 128 that stores data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. To this end, the web server 130 processes incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0030] The application programming interface (API) server 122 receives and sends interaction data (e.g., commands and message payloads) between the interaction server 124 and the client system 102 (as well as, for example, the interaction client 104 and other applications 106) and third-party servers 112. Specifically, the application programming interface (API) server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate the functions of the interaction server 124. The application programming interface (API) server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transmitting media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting a collection of media data (e.g., a story); retrieving a list of friends of a user of the user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) for an entity graph (e.g., a social graph); locating friends in the social graph; and opening application events (e.g., related to the interaction client 104).

[0031] The interaction server 124 hosts multiple systems and subsystems, which are described below with reference to Figure 2 as follows.

[0032] The linked applications

[0033] Returning to the interactive client 104, the features and functions of external resources (e.g., linked application 106 or applet) are made available to the user via the interface of the interactive client 104. In this context, "external" refers to the fact that the application 106 or applet is outside the interactive client 104. External resources are typically provided by a third party, but may also be provided by the creator or provider of the interactive client 104. The interactive client 104 receives a user selection of an option for initiating or accessing a feature of such an external resource. The external resource may be an application 106 installed on the user system 102 (e.g., "local app"), or a scaled-down version of an application hosted on or remote to the user system 102 (e.g., on a third-party server 112) (e.g., "applet"). The scaled-down version of the application includes a subset of the features and functions of the application (e.g., the full-scale local version of the application) and is implemented using a markup language document. In some examples, the scaled-down version of the application (e.g., "applet") is a web-based markup language version of the application and is embedded within the interactive client 104. In addition to using a markup language document (e.g.,.*ml file), the applet may include a scripting language (e.g.,.*js file or.json file) and a style sheet (e.g.,.*ss file).

[0034] In response to receiving a user selection of an option for initiating or accessing a feature of an external resource, the interactive client 104 determines whether the selected external resource is a web-based external resource or a locally installed application 106. In some cases, an application 106 locally installed on the user system 102 may be launched independently of and separately from the interactive client 104, e.g., by selecting an icon corresponding to the application 106 on the home screen of the user system 102. A scaled-down version of such an application may be launched or accessed via the interactive client 104, and in some examples, no part or only a limited part of the scaled-down application may be accessed outside the interactive client 104. The scaled-down application may be launched by receiving, e.g., a markup language document associated with the scaled-down application from the third-party server 112 via the interactive client 104 and processing such a document.

[0035] In response to determining that the external resource is a locally installed application 106, the interactive client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interactive client 104 communicates with the third-party server 112 (e.g.) to obtain a markup language document corresponding to the selected external resource. The interactive client 104 then processes the obtained markup language document to render the web-based external resource within the user interface of the interactive client 104.

[0036] The interaction client 104 can notify a user of the user system 102 or other users associated with such a user (e.g., "friends") of an activity occurring in one or more external resources. For example, the interaction client 104 can provide notifications related to external resources currently or recently used by one or more members of a user group to participants in a conversation (e.g., a chat conversation) in the interaction client 104. One or more users can be invited to join the external resource of the activity or to start an external resource that was recently used but is currently inactive (in the friend group). The external resource can provide the ability to the participants in the conversation who each use the corresponding interaction client 104 to share items, conditions, statuses, or locations within the external resource in the chat session. The shared item can be an interactive chat card, which the members of the chat can use to interact, for example, to start the corresponding external resource, view specific information within the external resource, or bring the members of the chat to a specific location or status within the external resource. Within a given external resource, a response message can be sent to a user on the interaction client 104. The external resource can selectively include different media items in the response based on the current context of the external resource.

[0037] The interaction client 104 can present a list of available external resources (e.g., applications 106 or applets) to the user to start or access a given external resource. The list can be presented in the form of a context - related menu. For example, the icons representing different applications (or applets) of the application 106 (or applets) can vary based on how the menu is launched (e.g., from a conversation interface or from a non - conversation interface).

[0038] System Architecture

[0039] Figure 2 is a block diagram showing other details regarding the interaction system 100 according to some examples. Specifically, the interaction system 100 is shown to include an interaction client 104 and an interaction server 124. The interaction system 100 includes multiple subsystems, which are supported by the interaction client 104 on the client side and by the interaction server 124 on the server side. The exemplary subsystems are discussed below.

[0040] The image processing system 202 provides various functions that enable a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0041] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls the hardware camera hardware of the user system 102 (e.g., directly or via operating system controls) to modify and enhance the real - time images captured and displayed via the interaction client 104.

[0042] Enhancement system 206 provides functions related to the generation and publication of enhancements (e.g., media overlays) for images captured in real time by the camera of user system 102 or retrieved from the memory of user system 102. For example, enhancement system 206 operably selects, presents, and displays media overlays (e.g., image filters or image lenses) for interactive client 104 to enhance real-time images received via camera system 204 or stored images retrieved from the memory of user system 102. These enhancements are selected by enhancement system 206 based on some inputs and data, such as:

[0043] · The geographical location of user system 102; and

[0044] · The social network information of the user of user system 102.

[0045] Enhancements can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to media content items (e.g., photos or videos) at user system 102 for transmission in a message, or applied to video content such as a video content stream or feed sent from interactive client 104. Thus, image processing system 202 can interact with various subsystems of communication system 208 and support various subsystems of communication system 208, such as messaging system 210 and video communication system 212.

[0046] Media overlays can include text or image data that can be superimposed on photos taken by user system 102 or video streams produced by user system 102. In some examples, the media overlay can be a location overlay (e.g., Venice Beach), the name of a live event, or a business name overlay (e.g., Beach Café). In other examples, image processing system 202 uses the geographical location of user system 102 to identify a media overlay that includes the name of a business located at the geographical location of user system 102. Media overlays can include other markers associated with the business. Media overlays can be stored in database 128 and accessed via database server 126.

[0047] Image processing system 202 provides a user-based publishing platform that enables a user to select a geographical location on a map and upload content associated with the selected geographical location. The user can also specify the circumstances under which a particular media overlay should be provided to other users. Image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographical location.

[0048] The augmented creation system 214 supports the augmented reality developer platform and includes applications for content creators (e.g., artists and developers) to create and publish enhanced (e.g., augmented reality experiences) interactive clients 104. The augmented creation system 214 provides a library of built-in features and tools for content creators, which includes, for example, custom shaders, tracking technologies, and templates.

[0049] In some examples, the augmented creation system 214 provides a merchant-based publishing platform that enables merchants to select specific augmentations associated with a geographic location via a bidding process. For example, the augmented creation system 214 associates the media overlay of the highest-bidding merchant with the corresponding geographic location for a predefined amount of time.

[0050] The communication system 208 is responsible for enabling and handling various forms of communication and interaction within the interactive system 100 and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for implementing temporary or time-limited access by the interactive client 104 to content. The messaging system 210 includes multiple timers (e.g., in the short-lived timer system 218) that selectively enable access (e.g., for presentation and display) to messages and associated content via the interactive client 104 based on the duration and display parameters associated with the message or set of messages (e.g., a story). Additional details regarding the operation of the short-lived timer system 218 are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between multiple interactive clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between multiple interactive clients 104.

[0051] The user management system 220 is operationally responsible for managing user data and profiles and includes a social networking system 222 that maintains information about the relationships between users of the interactive system 100.

[0052] The collection management system 224 is operationally responsible for managing collections or sets of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories". Such collections can be made available for a specified period of time (e.g., the duration of the event to which the content pertains). For example, content related to a concert can be made available as a "story" for the duration of that concert. The collection management system 224 can also be responsible for publishing an icon that provides a notification of a particular collection to the user interface of the interactive client 104. The collection management system 224 includes a curation function that enables a collection manager to manage and curate a particular collection of content. For example, a curation interface enables an event organizer to curate a collection of content related to a particular event (e.g., delete inappropriate content or redundant messages). Additionally, the collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation can be paid to users for including user-generated content in a collection. In such cases, the collection management system 224 operates to automatically pay such users for the use of their content.

[0053] The map system 226 provides various geolocation functions and supports the presentation of map-based media content and messages by the interactive client 104. For example, the map system 226 enables the display on a map of user icons or avatars (e.g., stored in the profile data 302) to indicate the current or past locations of a user's "friends" within the context of the map, as well as media content (e.g., collections of messages including photos and videos) generated by those friends. For example, messages posted by a user from a particular geographic location to the interactive system 100 can be displayed to that user's "friends" within the context of that particular location on the map in the map interface of the interactive client 104. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where that location and status information is similarly displayed to the selected users within the context of the map interface of the interactive client 104.

[0054] The game system 228 provides various game functions within the context of the interactive client 104. The interactive client 104 provides a game interface that presents a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite such other users to participate in playing a particular game by sending an invitation from the interactive client 104 to the other users. The interactive client 104 also supports voice, video, and text messaging (e.g., chat) within the context of playing a game, provides a leaderboard for the game, and also supports providing in-game rewards (e.g., game currency and items).

[0055] The external resource system 230 provides an interface for the interactive client 104 to communicate with a remote server (e.g., the third-party server 112) to launch or access an external resource (i.e., an application or applet). Each third-party server 112 hosts an application or a scaled-down version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on, for example, a markup language (e.g., HTML5). The interactive client 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The application hosted by the third-party server 112 is programmed in JavaScript using a software development kit (SDK) provided by the interactive server 124. The SDK includes application programming interfaces (APIs) having functions that can be called or activated by the web-based application. The interactive server 124 hosts a JavaScript library that provides a given external resource access to the specific user data of the interactive client 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0056] To integrate the functions of the SDK into a web-based resource, the SDK is downloaded from the interactive server 124 by the third-party server 112 or otherwise received by the third-party server 112. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functions of the SDK to integrate features of the interactive client 104 into the web-based resource.

[0057] The SDK stored on the interaction server system 110 effectively provides a bridge between external resources (e.g., application 106 or applet) and the interaction client 104. This gives users a seamless experience of communicating with other users on the interaction client 104 while still preserving the appearance of the interaction client 104. To bridge the communication between the external resources and the interaction client 104, the SDK facilitates the communication between the third-party server 112 and the interaction client 104. The WebView JavaScript Bridge running on the user system 102 establishes two one-way communication channels between the external resources and the interaction client 104. Messages are sent asynchronously between the external resources and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with that callback identifier.

[0058] By using the SDK, not all information from the interaction client 104 is shared with the third-party server 112. The SDK restricts which information is shared based on the needs of the external resources. Each third-party server 112 provides the interaction server 124 with an HTML5 file corresponding to the web-based external resource. The interaction server 124 can add a visual representation (e.g., box design or other graphics) of the web-based external resource in the interaction client 104. Once the user selects the visual representation or indicates via the GUI of the interaction client 104 to access the features of the web-based external resource, the interaction client 104 obtains the HTML5 file and instantiates the resources for accessing the features of the web-based external resource.

[0059] The interactive client 104 presents a graphical user interface for an external resource (e.g., a landing page or a splash screen). During, before, or after presenting the landing page or splash screen, the interactive client 104 determines whether the launched external resource has previously been authorized to access the user data of the interactive client 104. In response to determining that the launched external resource has previously been authorized to access the user data of the interactive client 104, the interactive client 104 presents another graphical user interface of the external resource that includes the functions and features of the external resource. In response to determining that the launched external resource has not previously been authorized to access the user data of the interactive client 104, after a display threshold period (e.g., 3 seconds) of the landing page or splash screen of the external resource, the interactive client 104 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other part of the screen) for authorizing the external resource to access the user data. The menu identifies the types of user data that the external resource will be authorized to use. In response to receiving a user selection of the accept option, the interactive client 104 adds the external resource to the list of authorized external resources and allows the external resource to access the user data from the interactive client 104. The external resource is authorized by the interactive client 104 to access the user data under the OAuth 2 framework.

[0060] The interactive client 104 controls the types of user data shared with an external resource based on the type of the authorized external resource. For example, access to a first type of user data (e.g., a two-dimensional avatar of a user with or without different avatar characteristics) is provided to an external resource that includes a full-scale application (e.g., application 106). As another example, access to a second type of user data (e.g., payment information, a two-dimensional avatar of the user, a three-dimensional avatar of the user, and avatars with various avatar characteristics) is provided to an external resource that includes a scaled-down version of the application (e.g., a web-based version of the application). Avatar characteristics include different ways of customizing the appearance of the avatar (e.g., different poses, facial features, clothing, etc.).

[0061] The advertising system 232 operates to enable third parties to purchase advertisements to be presented to end users via the interactive client 104 and also handles the delivery and presentation of these advertisements.

[0062] Data Architecture

[0063] Figure 3 is a schematic diagram showing a data structure 300 that can be stored in the database 304 of the interactive server system 110 according to certain examples. Although the content of the database 304 is shown as including multiple tables, it should be understood that the data can be stored in other types of data structures (e.g., object-oriented databases).

[0064] The database 304 includes message data stored in a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Details are described below with reference to Figure 3 additional details regarding information that may be included in a message and included within the message data stored in the message table 306.

[0065] The entity table 308 stores entity data and is (e.g., referentially) linked to an entity graph 310 and profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, etc. Any entity for which the interactive server system 110 stores data about it can be an identified entity, regardless of the entity type. Each entity is set with a unique identifier and an entity type identifier (not shown).

[0066] The entity graph 310 stores information related to relationships and associations between entities. By way of example only, such relationships can be social, professional (e.g., working in a common company or organization), interest-based, or activity-based. Some relationships between entities can be one-way, such as a personal user's subscription to digital content of a business or publishing user (e.g., a newspaper or other digital media channel or brand). Other relationships can be two-way, such as the "friend" relationship between various users of the interactive system 100.

[0067] Certain permissions and relationships can be attached to each relationship and can also be attached to each direction of the relationship. For example, a two-way relationship (e.g., the friend relationship between personal users) can include authorization for the publication of digital content items between the personal users, but certain restrictions or filters (e.g., based on content characteristics, location data, or time-of-day data) can be imposed on the publication of such digital content items. Similarly, a subscription relationship between a personal user and a business user can impose different degrees of restrictions on the publication of digital content from the business user to the personal user and can significantly limit or prevent the publication of digital content from the personal user to the business user. As an example of an entity, a particular user can record certain restrictions (e.g., through privacy settings) in the record for that entity within the entity table 308. Such privacy settings can apply to all types of relationships within the context of the interactive system 100 or can be selectively applied to certain types of relationships.

[0068] Profile data 302 stores multiple types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case where the entity is an individual, the profile data 302 includes, for example, a user name, a telephone number, an address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user. Then, a particular user can selectively include one or more of these avatar representations within the content of a message transmitted via the interaction system 100 and on a map interface displayed by the interaction client 104 to other users. The collection of avatar representations can include "status avatars" that present a graphical representation of a status or activity that the user can choose to transmit at a particular time.

[0069] In the case where the entity is a group, in addition to a group name, members, and various settings (e.g., notifications) for the relevant group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group.

[0070] The database 304 also stores enhancement data, such as overlays or filters, in an enhancement table 312. The enhancement data is associated with videos (the data of which is stored in a video table 314) and images (the data of which is stored in an image table 316) and is applied to the videos and images.

[0071] In some examples, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. Filters can be of various types, including a filter selected by a user from a set of filters presented to the sending user by the interaction client 104 when the sending user is composing a message. Other types of filters include geographic location filters (also known as geo-filters), which can be presented to the sending user based on a geographic location. For example, based on geographic location information determined by a global positioning system (GPS) unit of the user system 102, the interaction client 104 can present geographic location filters specific to a nearby or special location within the user interface.

[0072] Another type of filter is a data filter, which can be selectively presented to the sending user by the interaction client 104 based on other input or information collected by the user system 102 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the user system 102, or the current time.

[0073] Other enhancement data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to application "lenses" or augmented reality experiences). Augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0074] The story table 318 stores data about a collection of messages and associated image, video, or audio data, where the messages and associated image, video, or audio data are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for whom a record is maintained in the entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the interaction client 104 can include user-selectable icons to enable the sending user to add specific content to his or her personal story.

[0075] The collection can also constitute a "Live Story", which is a collection of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" can constitute a curated stream of content submitted by users from different locations and events. Options to contribute content to a particular Live Story can be presented, for example, via the user interface of the interaction client 104 to users whose client devices have location services enabled and are at a common location event at a particular time. The interaction client 104 can identify Live Stories to a user based on the user's location. The end result is a "Live Story" told from a group perspective.

[0076] Another type of content collection is called a "Location Story", which enables users whose user system 102 is located within a particular geographical location (e.g., on a college or university campus) to contribute to a particular collection. In some examples, contributions to a Location Story can employ secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0077] As mentioned above, the video table 314 stores video data, which in some examples is associated with messages for which records are maintained in the message table 306. Similarly, the image table 316 stores image data associated with messages whose message data is stored in the entity table 308. The entity table 308 can associate various enhancements from the enhancement table 312 with the various images and videos stored in the image table 316 and the video table 314.

[0078] Data Communication Architecture

[0079] Figure 4FIG. is a schematic diagram showing the structure of a message 400 according to some examples, the message 400 being generated by an interactive client 104 for transmission via an interactive server 124 to another interactive client 104. The content of a particular message 400 is used to populate a message table 306 stored in a database 304 accessible by the interactive server 124. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data of the user system 102 or the interactive server 124. The message 400 is shown as including the following example components:

[0080] · Message identifier 402: A unique identifier that identifies the message 400.

[0081] · Message text payload 404: Text to be generated by the user via the user interface of the user system 102 and included in the message 400.

[0082] · Message image payload 406: Image data captured by a camera component of the user system 102 or retrieved from a memory component in the user system 102 and included in the message 400. The image data for the sent or received message 400 can be stored in an image table 316.

[0083] · Message video payload 408: Video data captured by a camera component or retrieved from a memory component of the user system 102 and included in the message 400. The video data for the sent or received message 400 can be stored in an image table 316.

[0084] · Message audio payload 410: Audio data captured by a microphone or retrieved from a memory component of the user system 102 and included in the message 400.

[0085] · Message enhancement data 412: Enhancement data (e.g., filters, stickers, or other annotations or enhancements) representing an enhancement to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400. The enhancement data for the sent or received message 400 can be stored in an enhancement table 312.

[0086] · Message duration parameter 414: A parameter value indicating, in seconds, the amount of time for which the content of the message (e.g., the message image payload 406, the message video payload 408, the message audio payload 410) is to be presented to the user or made accessible to the user via the interactive client 104.

[0087] · Message geographical location parameter 416: Geographical location data (e.g., latitude coordinates and longitude coordinates) associated with the content payload of the message. Multiple message geographical location parameter 416 values can be included in the payload, and each of these parameter values is associated with a content item included in the content (e.g., a specific image within the message image payload 406 or a specific video within the message video payload 408).

[0088] · Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories" identified in the story table 318) associated with a specific content item in the message image payload 406 of the message 400. For example, the identifier value can be used to associate each of multiple images within the message image payload 406 with multiple content collections.

[0089] · Message tag 420: Each message 400 can be tagged with multiple tags, and each of the multiple tags indicates the theme of the content included in the message payload. For example, in the case where a specific image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tag 420 indicating the relevant animal. The tag value can be manually generated based on user input or can be automatically generated using, for example, image recognition.

[0090] · Message sender identifier 422: An identifier (e.g., a message transceiver system identifier, an email address, or a device identifier) that indicates the user of the user system 102 on which the message 400 is generated and from which the message 400 is sent.

[0091] · Message recipient identifier 424: An identifier (e.g., a message transceiver system identifier, an email address, or a device identifier) that indicates the user of the user system 102 to which the message 400 is addressed.

[0092] The content (e.g., values) of the components of the message 400 can be pointers to the locations in the table where the content data values are stored. For example, the image value in the message image payload 406 can be a pointer (or address) to a location within the image table 316. Similarly, the value within the message video payload 408 can point to data stored within the image table 316, the value stored within the message enhancement data 412 can point to data stored in the enhancement table 312, the value stored within the message story identifier 418 can point to data stored in the story table 318, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored in the entity table 308.

[0093] Figure 5An example interface 502 for an application that develops an augmented reality content generator for generating virtual hair (e.g., for providing an AR experience for virtual hair on a person's image or video) is shown.

[0094] In an implementation, the application can be a program that enables editing of the generated given augmented reality content of a given augmented reality content generator (including modifying various attributes and interaction characteristics of the AR content).

[0095] As shown in interface 502, various graphical elements are included to modify and configure various aspects of the AR content generator for hair generation. Information related to the AR content generator is shown in interface 502, including various parameters for configuring the AR content generator for hair generation. In this example, an optional graphical item (e.g., a slider element) has been selected to configure the smoothness parameter of the AR content generator. Also as shown, the preview area shows virtual hair 504 based on the selected parameters.

[0096] The following discussion below in Figure 6 and Figure 7 relates to an example architecture for using different encoders to project a given input image into a latent space for generating virtual content.

[0097] Figure 6 An example processing flow of a computational architecture for projecting an input image 606 into a latent space using a styleGAN (Style Generative Adversarial Network) generator according to an implementation of the present technology is shown. In an implementation, the styleGAN generator can be a specific version of styleGAN that includes the StyleGAN2 model. StyleGAN2 refers to a generative adversarial network based on StyleGAN, and StyleGAN2 has improvements including a normalization technique called weight demodulation, and improvements in training by focusing on low-resolution images and then transferring to higher resolutions during training without changing the network topology.

[0098] As shown, an encoder (e.g., a pixel2style2pixel (pSp) encoder) receives the input image 606, performs encoding of the input image 606, and provides the encoded image to the styleGAN generator 604. Next, the styleGAN generator 604 provides an output image 608. In an implementation, the styleGAN generator 604 is based on the StyleGAN2 model.

[0099] In Figure 6 the pSp architecture, the encoder backbone is extended with a feature pyramid that generates three levels of feature maps, using an intermediate network such as Figure 6The map2style algorithm shown in

[15] extracts styles from the three levels of feature maps. The styles aligned with the hierarchical representations are then fed into the styleGAN generator 604 according to their scale to generate the output image 608, thereby completing the conversion from input pixels to output pixels through the intermediate style feature representation. Therefore, it should be understood that the pSp architecture is an end-to-end image-to-image conversion framework.

[0100] Figure 7 Another example process flow 702 of a computational architecture for projecting an input image 704 into a latent space using a styleGAN generator according to an embodiment of the present technology is shown. In an embodiment, the styleGAN generator is based on the StyleGAN2 model.

[0101] As shown, an encoder (e.g., an encoder for editing (e4e) encoder) receives an input image 704, performs encoding on the input image 704, and provides the encoded image to a pre-trained styleGAN generator. Next, the styleGAN generator provides an output image 706.

[0102] In the implementation, the encoder receives an input image and outputs a single style code w and a set of offsets Δ1...ΔN-1, where N represents the number of styleGAN style modulation layers. The final latent representation can be determined by replicating the w vector N times and adding each Δi to its corresponding entry. During training, L d-reg Regularization encourages small variance between different entries of the final representation, thus keeping it close to W*. In addition, L adv Guide each latent code towards the mapping network range of styleGAN, so as to get closer to W k As a result of applying the two regularization terms, the encoder’s final learned representation is close to W.

[0103] In an embodiment, Figure 6 and Figure 7 The above encoders discussed in are interchangeable and can be used in place of another encoder.

[0104] Figure 8 An example process flow for generating virtual hair according to some embodiments of the present technology is shown. In an embodiment, Figure 8 The process flow in can be performed by the computer client device 118. Alternatively (or in combination), Figure 8 The process flow in can be performed by the interactive server system 110. For the purpose of discussion, Figure 8 The process flow in FIG. 1 is described with respect to the computer client device 118 .

[0105] Initially, computer client device 118 uses a pre-trained StyleGAN model to generate a random image of a face at operation 802. In an implementation, although Figure 8 mentions the StyleGAN model, it should be understood that the model can be a specific version of StyleGAN including the StyleGAN2 model. As mentioned previously, StyleGAN2 refers to a generative adversarial network based on StyleGAN, which has improvements including a normalization technique called weight demodulation, and improvements in training by focusing on low-resolution images and then transferring to higher resolutions during training without changing the network topology.

[0106] Next, computer client device 118 performs operation 810 to apply 3D hair to the image generated by the StyleGAN model at operation 802 to generate image 804. In an example, the 3D hair will serve as a condition for the dynamics and shape of realistic-looking hair. In an implementation, such 3D hair can be provided by an augmented reality content generator.

[0107] At operation 812, computer client device 118 utilizes a pre-trained neural network (e.g., the pSp or e4e architecture described previously in Figure 6 and Figure 7 ), which projects image 804 into the StyleGAN2 latent space (e.g., generates a latent code for the StyleGAN2 latent space). The image 806 generated according to such a latent code will have a realistic-looking (virtual) hair that is similar in shape and structure to the 3D hair.

[0108] At operation 814, computer client device 118 blends the realistic hair from image 806 with the image obtained at operation 802 to obtain image 808 with a new realistic hairstyle corresponding to the 3D hair generated in image 804. In an implementation, an image2image neural network is trained, which takes as input an image with the applied 3D hair (e.g., image 804) and outputs the same face but with realistic hair instead of 3D hair.

[0109] At operation 816, the model is used for real-time realistic hair generation. First, 3D hair is applied to the face (e.g., at operation 810), and then the neural network model creates an image with realistic hair based on that image.

[0110] Figure 9is a flowchart showing a method 900 according to some example embodiments. The method 900 may be embodied in computer-readable instructions for execution by one or more computer processors such that the operations of the method 900 may be performed, in part or in whole, by a computer client device 118. However, it should be understood that at least some of the operations of the method 900 may be deployed on various other hardware configurations, and the method 900 is not intended to be limited to the computer client device 118 or any of the components or systems mentioned above.

[0111] According to some examples, at operation 902, the computer client device 118 generates a first image of a face using a trained generative adversarial network (GAN) model. In an example, the trained GAN model includes a StyleGAN2 model.

[0112] At operation 904, the computer client device 118 applies three-dimensional (3D) virtual hair to the first image generated by the trained GAN model to generate a second image with 3D virtual hair. In an example, applying the 3D virtual hair is based on an augmented reality (AR) content generator, and the augmented reality (AR) generator uses a process different from the trained neural network to generate the 3D virtual hair.

[0113] At operation 906, the computer client device 118 projects the second image with 3D virtual hair into the GAN latent space to generate a third image with realistic virtual hair. In an example, the GAN latent space includes a compressed representation of data. In an example, projecting the second image with 3D virtual hair into the GAN latent space is based on a pixel2style2pixel (pSp) encoder or an encoder for editing (e4e) encoder.

[0114] At operation 908, the computer client device 118 performs a blend of the realistic virtual hair with the first generated image of the face to generate a new image with new realistic hair corresponding to the 3D virtual hair. In an example, performing the blend is based on one or more of Gaussian blending, Laplacian blending, edge detection, Poisson blending, or gradient blending.

[0115] At operation 910, the computer client device 118 trains a neural network that receives the second image with 3D virtual hair and provides an output image with realistic virtual hair.

[0116] In an example, the neural network includes a deep neural network. In an example, the deep neural network includes an image2image neural network.

[0117] At operation 912, the computer client device 118 uses a trained neural network to generate a particular output image with realistic hair based on a particular input image with 3D virtual hair. In an example, the particular input image is an image different from a first image of a face. In an example, the particular output image with realistic hair is an image different from a new image with new realistic hair.

[0118] Machine architecture

[0119] Figure 10 is a graphical representation of a machine 1000 within which instructions 1002 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 1000 to perform any one or more of the methods discussed herein. For example, the instructions 1002 can cause the machine 1000 to perform any one or more of the methods described herein. The instructions 1002 transform the general, unprogrammed machine 1000 into a particular machine 1000 programmed to perform the described and shown functions in the described manner. The machine 1000 can operate as a stand-alone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in a server-client network environment as a server machine or a client machine, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1000 can include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing the instructions 1002 specifying actions to be taken by the machine 1000. Further, although only a single machine 1000 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute the instructions 1002 to perform any one or more of the methods discussed herein. For example, the machine 1000 can include a user system 102 or any one of a plurality of server devices forming part of an interactive server system 110. In some examples, the machine 1000 can also include both a client system and a server system, where certain operations of a particular method or algorithm are performed on the server side and certain operations of the particular method or algorithm are performed on the client side.

[0120] Machine 1000 may include a processor 1004, a memory 1006, and an input / output (I / O) component 1008 that may be configured to communicate with each other via a bus 1010. In an example, the processor 1004 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1012 and a processor 1014 that execute instructions 1002. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 10 multiple processors 1004 are shown, machine 1000 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0121] Memory 1006 includes a main memory 1016, a static memory 1018, and a storage unit 1020, all of which may be accessed by processor 1004 via bus 1010. The main memory 1006, the static memory 1018, and the storage unit 1020 store instructions 1002 that implement any one or more of the methods or functions described herein. The instructions 1002 may also reside, completely or partially, within a machine-readable medium 1022 within the main memory 1016, within the static memory 1018, within the storage unit 1020, within at least one of the processors 1004 (e.g., within a cache memory of the processor), or any suitable combination thereof during execution by the machine 1000.

[0122] The I / O component 1008 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurement results, etc. The specific I / O components 1008 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will unlikely include such a touch input device. It should be appreciated that the I / O component 1008 may include Figure 10Many other components not shown. In various examples, the I / O component 1008 may include a user output component 1024 and a user input component 1026. The user output component 1024 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The user input component 1026 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, a touch screen that provides the location and force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), etc.

[0123] In other examples, the I / O component 1008 may include a biometric component 1028, a motion component 1030, an environmental component 1032, or a location component 1034, as well as a wide range of other components. For example, the biometric component 1028 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion component 1030 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope).

[0124] The environmental component 1032 includes, for example, one or more cameras (with still image / photo and video capabilities), a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of a hazardous gas for safety or for measuring pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment.

[0125] Regarding the camera, the user system 102 may have a camera system that includes, for example, a front camera on the front surface of the user system 102 and a rear camera on the rear surface of the user system 102. The front camera can be used, for example, to capture still images and videos of the user of the user system 102 (e.g., "selfies"), which can then be enhanced with the above-described enhancement data (e.g., filters). For example, the rear camera can be used to capture still images and videos in a more conventional camera mode, and these images are similarly enhanced with the enhancement data. In addition to the front and rear cameras, the user system 102 may also include a 360° camera for capturing 360° photos and videos.

[0126] In addition, the camera system of the user system 102 may include dual rear cameras (e.g., a main camera and a depth-sensing camera), or even triple, quadruple, or quintuple rear camera configurations on the front and rear sides of the user system 102. For example, these multi-camera systems may include wide-angle cameras, ultra-wide-angle cameras, telephoto cameras, macro cameras, and depth sensors.

[0127] The location component 1034 includes a location sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), and so on.

[0128] Various techniques can be used to implement communication. The I / O component 1008 also includes a communication component 1036 that is operable to couple the machine 1000 to the network 1038 or the device 1040 via a corresponding coupling or connection. For example, the communication component 1036 may include a network interface component that interfaces with the network 1038 or another suitable device. In other examples, the communication component 1036 may include a wired communication component, a wireless communication component, a cellular communication component, a near-field communication (NFC) component, components (e.g., low power consumption), components, and other communication components for providing communication via other forms. The device 1040 can be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0129] In addition, the communication component 1036 can detect identifiers or include components operable to detect identifiers. For example, the communication component 1036 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying audio signals of tags). Additionally, various information can be obtained via the communication component 1036, such as positioning via Internet Protocol (IP) geolocation, positioning via positioning via signal triangulation, positioning via detecting NFC beacon signals that can indicate a specific location, etc.

[0130] Various memories (e.g., main memory 1016, static memory 1018, and the memory of the processor 1004) and the storage unit 1020 can store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instruction 1002), when executed by the processor 1004, cause various operations to implement the disclosed examples.

[0131] The instructions 1002 can be transmitted or received via a network interface device (e.g., the network interface component included in the communication component 1036) using a transmission medium and using any one of several well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)) over the network 1038. Similarly, the instructions 1002 can be transmitted or received using a transmission medium via a coupling (e.g., a peer-to-peer coupling) with the device 1040.

[0132] Software Architecture

[0133] Figure 11 is a block diagram 1100 showing a software architecture 1102 that can be installed on any one or more of the devices described herein. The software architecture 1102 is supported by hardware, such as a machine 1104 including a processor 1106, a memory 1108, and I / O components 1110. In this example, the software architecture 1102 can be conceptualized as a stack of layers, where each layer provides a specific function. The software architecture 1102 includes the following layers, such as an operating system 1112, libraries 1114, frameworks 1116, and applications 1118. In operation, the application 1118 activates API calls 1120 through the software stack and receives messages 1122 in response to the API calls 1120.

[0134] The operating system 1112 manages hardware resources and provides common services. The operating system 1112 includes, for example: a kernel 1124, services 1126, and drivers 1128. The kernel 1124 serves as an abstraction layer between the hardware and other software layers. For example, the kernel 1124 provides functions such as memory management, processor management (e.g., scheduling), component management, networking, and security settings. The services 1126 can provide other common services for other software layers. The drivers 1128 are responsible for controlling or interfacing with the underlying hardware. For example, the drivers 1128 can include a display driver, a camera driver, or a low-power driver, a flash driver, a serial communication driver (e.g., a USB driver), drivers, an audio driver, a power management driver, etc.

[0135] The library 1114 provides common low-level infrastructure used by the applications 1118. The library 1114 can include system libraries 1130 (e.g., the C standard library), which provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1114 can include API libraries 1132, such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., the OpenGL framework for 2D and 3D presentation in graphical content on a display), database libraries (e.g., SQLite that provides various relational database functions), web libraries (e.g., WebKit that provides web browsing functions), etc. The library 1114 can also include a variety of other libraries 1134 to provide many other APIs to the applications 1118.

[0136] The framework 1116 provides common high-level infrastructure used by the applications 1118. For example, the framework 1116 provides various Graphical User Interface (GUI) functions, high-level resource management, and high-level location services. The framework 1116 can provide a wide range of other APIs that can be used by the applications 1118, some of which may be specific to a particular operating system or platform.

[0137] In an example, the applications 1118 can include a home application 1136, a contacts application 1138, a browser application 1140, a book reader application 1142, a location application 1144, a media application 1146, a messaging application 1148, a gaming application 1150, and various other applications such as third-party applications 1152. The applications 1118 are programs that execute functions defined in a program. One or more of the applications 1118 can be created using various programming languages, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C language or assembly language), which are structured in various ways. In a particular example, a third-party application 1152 (e.g., an application developed using an ANDROID TM or IOS TM software development kit (SDK) by an entity other than the vendor of a particular platform) can be mobile software that runs on a mobile operating system such as IOS TM or Phone or other mobile operating systems. In this example, the third-party application 1152 can activate API calls 1120 provided by the operating system 1112 to facilitate the functions described herein.

[0138] Glossary

[0139] "Carrier signal" means any non-tangible medium that can store, encode, or carry instructions executable by a machine and includes digital or analog communication signals or other non-tangible media that facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0140] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smartphone, a tablet computer, a superbook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a gaming console, a set-top box, or any other communication device that a user can use to access a network.

[0141] "Communication network" refers to one or more parts of a network, such as a network, which can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, other types of networks, or a combination of two or more such networks. For example, a network or a part of a network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling may implement any data transmission technology among various types of data transmission technologies, such as single-carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, the fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high-speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other data transmission technologies defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0142] "Component" refers to, for example, a logical or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization for a particular processing or control function. Components can be combined with other components via their interfaces to perform machine processing. A component can be an encapsulated functional hardware unit designed to be used with other components and a part of a program for a specific function that generally performs related functions. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various examples, one or more computer systems (e.g., stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or group of processors) can be configured by software (e.g., an application or part of an application) to operate to perform certain operations as described herein as a hardware component. A hardware component can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic permanently configured to perform certain operations. A hardware component can be a dedicated processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) that is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be appreciated that a decision can be made, for cost and time considerations, as to whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in circuitry that is temporarily configured (e.g., configured by software). Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, i.e., an entity that is physically constructed, permanently configured (e.g., hard-wired), or temporarily configured (e.g., programmed) to operate in some manner or perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component at any given time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a dedicated processor, the general-purpose processor can be configured at different times to be respective different dedicated processors (e.g., including different hardware components). The software accordingly configures a particular one or more processors to, for example, constitute a particular hardware component at one moment and a different hardware component at a different moment. Hardware components can provide information to and receive information from other hardware components. Thus, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving the information from the memory structure. For example, one hardware component can perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Then, another hardware component can access the memory device at a subsequent time to retrieve the stored output and process it. Hardware components can also initiate communication with input devices or output devices and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors temporarily configured (e.g., via software) or permanently configured to perform the associated operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, a "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least in part processor-implemented, where a particular one or more processors are examples of hardware. For example, at least some of the various operations of the method can be performed by one or more processors or processor-implemented components. Additionally, one or more processors can also operate to support the execution of the associated operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), where the operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of certain operations can be distributed among the processors, not residing only within a single machine but deployed across multiple machines. In some examples, the processors or processor-implemented components can be located in a single geographical location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components can be distributed across multiple geographical locations.

[0143] "Computer-readable storage medium" refers to both, for example, machine storage media and transmission media. Thus, these terms include both storage devices / media and carrier / wave-modulated data signals. The terms "machine-readable medium", "computer-readable medium", and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0144] "Ephemeral message" means a message that is accessible, for example, within a time-limited duration. An ephemeral message can be text, image, video, etc. The access time of an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0145] "Machine storage medium" means, for example, a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be regarded as including, but not limited to, solid-state memory as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", "computer storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are subsumed in the term "signal medium".

[0146] "Non-transitory computer-readable storage medium" means, for example, a tangible medium capable of storing, encoding, or carrying instructions executable by a machine.

[0147] "Signal medium" means, for example, any intangible medium capable of storing, encoding, or carrying instructions executable by a machine and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be regarded as including any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose one or more characteristics are set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

[0148] "User equipment" means, for example, a device accessed, controlled, or owned by a user, and with which the user interacts to perform an action or interact with other users or computer systems.

Claims

1. A method, comprising: generating, by one or more hardware processors, a first image of a face using a trained generative adversarial network (GAN) model; applying, by the one or more hardware processors, three-dimensional (3D) virtual hair to the first image generated by the trained GAN model to generate a second image with 3D virtual hair; projecting, by the one or more hardware processors, the second image with 3D virtual hair into a GAN latent space to generate a third image with realistic virtual hair; performing, by the one or more hardware processors, a blending of the realistic virtual hair with the first image of the generated face to generate a new image with new realistic hair corresponding to the 3D virtual hair; training, by the one or more hardware processors, a neural network that receives the second image with 3D virtual hair and provides an output image with realistic virtual hair; and generating, by the one or more hardware processors, a specific output image with realistic hair based on a specific input image with 3D virtual hair using the trained neural network.

2. The method according to claim 1, wherein The trained GAN model includes a StyleGAN2 model.

3. The method according to claim 2, wherein Applying the 3D virtual hair is based on an augmented reality (AR) content generator that uses a process different from the trained neural network to generate the 3D virtual hair.

4. The method according to claim 3, wherein, The GAN latent space includes a compressed representation of data.

5. The method according to claim 1, wherein Performing the blending is based on one or more of Gaussian blending, Laplacian blending, edge detection, Poisson blending, or gradient blending.

6. The method according to claim 1, wherein, The neural network includes a deep neural network.

7. The method according to claim 6, wherein, The deep neural network includes an image2image neural network.

8. The method according to claim 1, wherein, Projecting the second image with 3D virtual hair into the GAN latent space is based on a pixel2style2pixel (pSp) encoder or an encoder for editing (e4e) encoder.

9. The method according to claim 1, wherein The specific input image is an image different from the first image of the face.

10. The method according to claim 1, wherein The specific output image with realistic hair is an image different from the new image with the new realistic hair.

11. A system, comprising: a processor; and a memory that includes instructions that, when executed by the processor, cause the processor to perform operations, the operations including: generating a first image of a face using a trained generative adversarial network (GAN) model; applying three-dimensional (3D) virtual hair to the first image generated by the trained GAN model to generate a second image with 3D virtual hair; projecting the second image with 3D virtual hair into a GAN latent space to generate a third image with realistic virtual hair; performing a blending of the realistic virtual hair with the first image of the generated face to generate a new image with new realistic hair corresponding to the 3D virtual hair; training a neural network that receives the second image with 3D virtual hair and provides an output image with realistic virtual hair; and Generate a specific output image with realistic hair based on a specific input image with 3D virtual hair using the trained neural network.

12. The system according to claim 11, wherein, The trained GAN model includes a StyleGAN2 model.

13. The system according to claim 12, wherein, Apply 3D virtual hair to an augmented reality (AR) content generator, which uses a process different from the trained neural network to generate 3D virtual hair.

14. The system according to claim 13, wherein, The GAN latent space includes a compressed representation of the data.

15. The system according to claim 11, wherein, Perform the mixing based on one or more of Gaussian mixture, Laplace mixture, edge detection, Poisson mixture, or gradient mixture.

16. The system according to claim 11, wherein, The neural network includes a deep neural network.

17. The system according to claim 16, wherein The deep neural network includes an image2image neural network.

18. The system according to claim 11, wherein, Project the second image with 3D virtual hair into the GAN latent space based on a pixel2style2pixel (pSp) encoder or an encoder for editing (e4e) encoder.

19. The system according to claim 11, wherein, The specific input image is an image different from the first image of the face.

20. A non-transitory computer-readable medium, the non-transitory computer-readable medium includes instructions that, when executed by a computing device, cause the computing device to perform operations, the operations including: Generate a first image of a face by one or more hardware processors using a trained generative adversarial network (GAN) model; Apply three-dimensional (3D) virtual hair to the first image generated by the trained GAN model by the one or more hardware processors to generate a second image with 3D virtual hair; Project the second image with 3D virtual hair into the GAN latent space by the one or more hardware processors to generate a third image with realistic virtual hair; Perform mixing of the realistic virtual hair with the first image of the generated face by the one or more hardware processors to generate a new image with new realistic hair corresponding to the 3D virtual hair; Train a neural network by the one or more hardware processors, the neural network receiving the second image with 3D virtual hair and providing an output image with realistic virtual hair; And Generate a specific output image with realistic hair based on a specific input image with 3D virtual hair using the trained neural network by the one or more hardware processors.

Images