Facial synthesis in augmented reality content for advertisements

Abstract

The present technology receives frames of source media content, and said frames of source media content include representations of the head and face of a source actor. The present technology generates sets of source pose parameters. The present technology receives at least one target image, and at least one target image includes representations of the target head and target face of a target entity. The present technology generates output media content based at least partially on the sets of source pose parameters, and each frame of the output media content includes an image of the target face in at least one frame of the output media content. The present technology provides an online advertisement based at least partially on the output media content for display on a computing device.

Description

Technology Field

[0001] Claim of priority

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 200,878 filed March 31, 2021, the full text of which is incorporated herein by reference for all purposes. Background Technology

[0003] With the increasing use of digital images, the affordability of portable computing devices, the availability of digital storage media with increased capacity, and increased network connection bandwidth and accessibility, digital images have become a part of daily life for an increasing number of people. Brief explanation of the drawing

[0004] To facilitate the identification of discussions regarding any specific element or function, the top digit or number of the reference number refers to the drawing number where the element is first introduced. FIG. 1 is a schematic representation of a network environment in which the present disclosure may be placed, according to some exemplary embodiments. FIG. 2 is a schematic representation of a messaging client application according to some exemplary embodiments. FIG. 3 is a schematic representation of a data structure as maintained in a database, according to some exemplary embodiments. FIG. 4 is a schematic representation of a message according to some exemplary embodiments. FIG. 5 is a schematic diagram illustrating the structure of message annotations as described in FIG. 4, which includes additional information corresponding to a given message according to some embodiments. FIG. 6 is a block diagram illustrating various modules of a messaging client application according to specific exemplary embodiments. FIG. 7 illustrates an interface (e.g., a graphical user interface) for selecting the type of advertisement for an online advertising campaign in the messaging system according to some embodiments. FIG. 8 illustrates an interface for configuring one or more characteristics of an online advertisement based on AR content including face animation synthesis in the messaging system according to some embodiments. FIG. 9 illustrates an interface including an example of an online advertisement comprising media content accompanied by face animation synthesis in the messaging system according to some embodiments. FIG. 10 illustrates an interface including an example of a content creation application (e.g., a media content editor) in the messaging system according to some embodiments. FIG. 11 illustrates interfaces including examples of online advertisements including media content accompanied by face animation synthesis in the messaging system according to some embodiments. FIG. 12 is a flowchart illustrating a method according to a specific exemplary embodiment. FIG. 13 is a block diagram illustrating a software architecture in which the present disclosure can be implemented, according to some exemplary embodiments. FIG. 14 is a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to enable the machine to perform one or more of the methodologies discussed, according to some exemplary embodiments. Specific details for implementing the invention

[0005] Users with diverse interests from various locations capture digital images of various subjects and make these captured images available to others through networks such as the Internet. To enhance users' experiences with digital images and provide various features, it can be difficult and computationally intensive to enable computing devices to perform image processing operations on various objects and / or features captured under a wide range of varying conditions (e.g., changes in image scales, noise, lighting, motion, or geometric distortion).

[0006] Augmented reality technology aims to bridge the gap between virtual and real-world environments by providing an enhanced real-world environment augmented by electronic information. As a result, the electronic information appears to be part of the real-world environment as perceived by the user. In one example, augmented reality technology further provides a user interface for interacting with electronic information overlaid on the enhanced real-world environment.

[0007] As mentioned above, with the increasing use of digital images, the availability of portable computing devices, the availability of digital storage media with increased capacity, and increased network connection bandwidth and accessibility, digital images have become a part of daily life for an increasing number of people. Users of various interests from various locations capture digital images of various subjects and make these captured images available to others through networks such as the Internet. To enhance users' experiences with digital images and provide diverse features, it can be difficult and computationally intensive to enable computing devices to perform image processing operations on various objects and / or features captured under a wide range of varying conditions (e.g., changes in image scales, noise, lighting, motion, or geometric distortion).

[0008] Messaging systems are frequently and increasingly utilized by users of mobile computing devices in various settings to provide various types of functionality in a convenient manner. As described herein, the messaging system includes actual applications that provide improvements in rendering augmented reality content generators (e.g., providing augmented reality experiences) on media content (e.g., images, videos, etc.) where specific augmented reality content generators can be activated, through an improved system that enables the provision of augmented reality content that is more advantageously customized for specific requirements associated with online advertising campaigns of each entity (e.g., merchants, companies, individuals, etc.).

[0009] Embodiments of the present technology enable face animation synthesis that may include transmitting the facial expressions of a source individual in a source video to a target individual in a target video or target image. Face animation synthesis can be used for the manipulation and animation of faces in many applications, such as entertainment shows, computer games, video conversations, virtual reality, augmented reality, etc.

[0010] Some current techniques for face animation compositing use morphable face models to re-render a target face with different facial expressions. While the generation of faces with morphable face models can be fast, the generated faces may not be realistic. Some other current techniques for face animation compositing are time-consuming and may not be suitable for performing real-time face animation compositing on standard mobile devices.

[0011] Messaging systems are frequently and increasingly utilized by users of mobile computing devices in various settings to provide various types of functionality in a convenient manner. As described herein, the messaging system includes actual applications that provide improvements in capturing image data and rendering AR content (e.g., images, videos, etc.) based on captured image data by providing technical improvements in capturing image data using electronic devices with at least power and resource constraints. These improvements in capturing image data are made possible by the techniques provided by the present technology, which reduce latency and increase efficiency in processing captured image data, thereby also reducing power consumption in capturing devices.

[0012] As further discussed herein, the infrastructure supports the creation and sharing of interactive media, referred herein as messages containing 3D content or AR effects, across various components of the messaging system. In the exemplary embodiments described herein, messages may enter the system from a live camera or through a repository (e.g., where messages containing 3D content and / or AR effects are stored in memory or a database). The system supports motion sensor input and the loading of external effects and asset data.

[0013] As mentioned in this specification, the phrases “augmented reality experience,” “augmented reality content item,” and “augmented reality content generator” include or refer to various image processing operations corresponding to image modification, filters, AR content generators, media overlays, transformations, etc., and as further described in this specification, may additionally include the playback of audio or music content during the presentation of AR content or media content.

[0014] FIG. 1 is a block diagram illustrating an exemplary messaging system (100) for exchanging data (e.g., messages and associated content) over a network. The messaging system (100) includes multiple instances of client devices (102) (e.g., computing devices), each of which hosts multiple applications including a messaging client application (104). Each messaging client application (104) is coupled to communicate with other instances of the messaging client application (104) and the messaging server system (108) over a network (106) (e.g., the Internet).

[0015] A messaging client application (104) can communicate with other messaging client applications (104) and a messaging server system (108) and exchange data through a network (106). The data exchanged between messaging client applications (104) and between a messaging client application (104) and a messaging server system (108) includes functions (e.g., commands to activate functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

[0016] The messaging server system (108) provides server-side functionality to a specific messaging client application (104) via the network (106). Although specific functions of the messaging system (100) are described in this specification as being performed by the messaging client application (104) or by the messaging server system (108), the location of specific functionality within the messaging client application (104) or the messaging server system (108) is a design choice. For example, it may be technically desirable to initially place specific technology and functionality within the messaging server system (108), but later transfer this technology and functionality to the messaging client application (104) when the client device (102) has sufficient processing capacity.

[0017] The messaging server system (108) supports various services and operations provided to the messaging client application (104). Such operations include transmitting data to the messaging client application (104), receiving data from it, and processing data generated by it. This data may include, for example, message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system (100) is initiated and controlled through functions available via the user interfaces (UIs) of the messaging client application (104).

[0018] Now, referring specifically to the messaging server system (108), an application program interface (API) server (110) is coupled to an application server (112) to provide a programmatic interface. The application server (112) is communicably coupled to a database server (118), which facilitates access to a database (120) where data associated with messages processed by the application server (112) is stored.

[0019] An API (Application Program Interface) server (110) receives and transmits message data (e.g., commands and message payloads) between a client device (102) and an application server (112). Specifically, the application program interface (API) server (110) provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a messaging client application (104) to activate the functionality of the application server (112). The application program interface (API) server (110) exposes various functions supported by the application server (112), including account registration, login functionality, transmission of messages via the application server (112) from a specific messaging client application (104) to another messaging client application (104), transmission of media files (e.g., images or videos) from a messaging client application (104) to a messaging server application (114), and possible access by another messaging client application (104), such as setting up a collection of media data (e.g., stories), searching for a list of friends of a user of a client device (102), searching for such collections, searching for messages and content, adding and removing friends to a social graph, the location of friends within a social graph, and opening application events (e.g., related to the messaging client application (104)).

[0020] The application server (112) hosts a number of applications and subsystems, including a messaging server application (114), an image processing system (116), and a social network system (122). The messaging server application (114) implements a number of message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) contained in messages received from multiple instances of the messaging client application (104). As described in more detail, text and media content from multiple sources may be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client application (104) by the messaging server application (114). Other processor and memory-intensive data processing may also be performed on the server side by the messaging server application (114), taking into account the hardware requirements for such processing.

[0021] The application server (112) also includes an image processing system (116) dedicated to performing various image processing operations on an image or video received within the payload of a message in a messaging server application (114).

[0022] The social network system (122) supports various social networking function services and enables the messaging server application (114) to use these functions and services. To this end, the social network system (122) maintains and accesses an entity graph (304) (as illustrated in FIG. 3) within a database (120). Examples of functions and services supported by the social network system (122) include the identification of other users of the messaging system (100) with whom a specific user has a relationship or 'follows', and also the identification of other entities and the interests of the specific user.

[0023] The application server (112) is communicably coupled to the database server (118), which facilitates access to the database (120) where data associated with messages processed by the messaging server application (114) is stored.

[0024] FIG. 2 is a block diagram illustrating additional details regarding a messaging system (100) according to exemplary embodiments. Specifically, the messaging system (100) is illustrated as including a messaging client application (104) and an application server (112), which ultimately implements a number of subsystems, namely a short-term timer system (202), a collection management system (204), and an annotation system (206).

[0025] The short-term timer system (202) is responsible for enforcing temporary access to content allowed by the messaging client application (104) and the messaging server application (114). To this end, the short-term timer system (202) includes a plurality of timers that selectively display messages and associated content through the messaging client application (104) and enable access thereto, based on durations and display parameters associated with a message or a collection of messages (e.g., a story). Further details regarding the operation of the short-term timer system (202) are provided below.

[0026] The collection management system (204) is responsible for managing collections of media (e.g., collections of text, image, video, and audio data). In some examples, collections of content (e.g., messages containing images, videos, text, and audio) may be organized into 'event galleries' or 'event stories'. These collections may be available for a specified period of time, such as the duration of the event to which the content is related. For example, content about a music concert may be available as a 'story' for the duration of that music concert. The collection management system (204) may also be responsible for posting an icon in the user interface of the messaging client application (104) that provides a notification of the existence of a specific collection.

[0027] The collection management system (204) further includes a curation interface (208) that allows a collection manager to manage and curate specific collections of content. For example, the curation interface (208) enables an event organizer to curate a collection of content related to a specific event (e.g., remove inappropriate content or duplicate messages). Additionally, the collection management system (204) automatically curates content collections using machine vision (or image recognition technology) and content rules. In certain embodiments, a reward may be paid to users for including user-generated content in a collection. In such cases, the curation interface (208) operates to automatically pay such users for using their content.

[0028] The annotation system (206) provides various functions that enable the user to annotate media content associated with a message, or otherwise modify or edit it. For example, the annotation system (206) provides functions related to the creation and publication of media overlays for messages processed by the messaging system (100). The annotation system (206) operatively supplies a media overlay or complement (e.g., an image filter) to the messaging client application (104) based on the geolocation of the client device (102). In another example, the annotation system (206) operatively supplies a media overlay to the messaging client application (104) based on other information, such as the social network information of the user of the client device (102). The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include photos, text, logos, animations, and sound effects. Examples of visual effects include color overlaying. Audio and visual content or visual effects may be applied to a media content item (e.g., a photo) on the client device (102). For example, a media overlay may include text that can be overlaid on a photo taken by the client device (102). In another example, the media overlay may include a location identification overlay (e.g., Venice Beach), the name of a live event, or a merchant name overlay (e.g., Beach Coffee House). In another example, the annotation system (206) uses the geolocation of the client device (102) to identify a media overlay containing the name of a merchant at the geolocation of the client device (102). The media overlay may include other indicia associated with the merchant.Media overlays are stored in a database (120) and can be accessed through a database server (118).

[0029] In one exemplary embodiment, the annotation system (206) provides a user-based publication platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. Users can also specify situations in which a particular media overlay should be provided to other users. The annotation system (206) creates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0030] In another exemplary embodiment, the annotation system (206) provides a merchant-based publishing platform that enables merchants to select a specific media overlay associated with a geolocation through a bidding process. For example, the annotation system (206) associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time.

[0031] FIG. 3 is a schematic diagram illustrating data structures (300) that may be stored in a database (120) of a messaging server system (108) according to certain exemplary embodiments. Although the contents of the database (120) are depicted as including a number of tables, it will be understood that data may be stored in data structures of other types (e.g., as an object-oriented database).

[0032] The database (120) contains message data stored in the message table (314). The entity table (302) stores entity data including the entity graph (304). Entities for which records are maintained in the entity table (302) may include individuals, legal entities, organizations, objects, places, events, etc. Regardless of type, any entity for which the messaging server system (108) stores data may be a recognized entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0033] The entity graph (304) further stores information regarding relationships and associations between entities. Such relationships may be based on interests or activities, for example, social, professional (for example, work in a general corporation or organization).

[0034] The database (120) also stores annotation data in an annotation table (312) in the form of exemplary filters. Filters for which data is stored in the annotation table (312) are associated with and applied to videos (for which data is stored in the video table (310)) and / or images (for which data is stored in the image table (308)). In one example, the filters are overlays that are overlaid on images or videos and displayed to the receiving user during a presentation. The filters may be of various types, including user-selected filters from a gallery of filters presented to the sending user by the messaging client application (104) when the sending user is composing a message. Other types of filters include geolocation filters (also known as geo-filters) that may be presented to the sending user based on geolocation. For example, geolocation filters specific to neighborhoods or specific locations may be presented within the user interface by the messaging client application (104) based on geolocation information determined by the GPS unit of the client device (102). Another type of filter is a data filter that may be optionally presented to the sending user by the messaging client application (104) based on information or other inputs collected by the client device (102) during the message generation process. Examples of data filters include the current temperature at a specific location, the current speed at which the sending user is moving, the battery life of the client device (102), or the current time.

[0035] Other annotation data that can be stored in the image table (308) is augmented reality content generators (e.g., corresponding to AR content generators, augmented reality experiences, or applying augmented reality content items). Augmented reality content generators may be real-time special effects and sounds that can be added to an image or video.

[0036] As described above, augmented reality content generators, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to videos or images. This includes real-time modifications, where an image is captured using device sensors and then displayed on the device screen along with the modifications. This also includes modifications to stored content, such as video clips within a gallery, that can be modified. For example, on a device accessing multiple augmented reality content generators, a user can use a single video clip with multiple augmented reality content generators to see how different augmented reality content generators modify the stored clip. For example, multiple augmented reality content generators applying different pseudo-random motion models can be applied to the same content by selecting different augmented reality content generators for the content. Similarly, real-time video capture with the exemplified modifications may be used to show how video images currently being captured by the device's sensors will modify the captured data. This data may simply be displayed on the screen and not stored in memory, or content captured by device sensors may be written to and stored in memory with or without modification (or both). In some systems, a preview feature may show how different augmented reality content generators will appear simultaneously within different windows on the display. This may, for example, allow multiple windows with different pseudo-random animations to be displayed simultaneously on the display.

[0037] Accordingly, data and various systems using augmented reality content generators or other such transformation systems to modify content using this data may involve the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking of these objects as they leave, enter, and move around in the video frames, and modification or transformation of these objects when they are tracked. In various embodiments, different methods may be used to achieve these transformations. For example, some embodiments may involve generating a 3D mesh model of an object or objects, and using a transformation of the model within the video and an animated texture to achieve the transformation. In other embodiments, by using the tracking of points on the object, an image or texture (which may be 2D or 3D) may be placed at the tracking location. In yet other embodiments, neural network analysis of video frames may be used to place images, models, or textures in the content (e.g., images or frames of the video). Therefore, augmented reality content generators refer to both the images, models, and textures used to generate content transformations, as well as the additional modeling and analysis information required to achieve these transformations through object detection, tracking, and placement.

[0038] Real-time video processing can be performed with any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or generate video streams using the device's sensors. Additionally, any objects, such as human faces and parts of the human body, animals, or inanimate objects like chairs, cars, or other objects, can be processed using computer animation models.

[0039] In some embodiments, when a specific modification is selected along with the content to be transformed, the elements to be transformed are identified by a computing device and subsequently detected and tracked if they exist in the frames of the video. The elements of the object are modified according to a request for modification, thereby transforming the frames of the video stream. The transformation of the frames of the video stream may be performed by different methods for different types of transformations. For example, for transformations of frames that primarily refer to changing the shapes of the elements of the object, characteristic points for each element of the object are calculated (e.g., using the Active Shape Model (ASM) or other known methods). Then, a mesh based on the characteristic points is generated for each of at least one element of the object. This mesh is used in the next stage of tracking the elements of the object within the video stream. In the tracking process, the mesh mentioned for each element is aligned with the location of each element. Subsequently, additional points are generated on the mesh. A first set of first points is generated for each element based on a request for modification, and a second set of points is generated for each element based on the first set of points and the request for modification. Then, frames of a video stream can be transformed by modifying the elements of an object based on the mesh and the first and second sets of points. In this method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0040] In one or more embodiments, transformations that modify parts of an object using the object's elements may be performed by calculating characteristic points for each element of the object and generating a mesh based on the calculated characteristic points. Points are generated on the mesh, and then various regions based on the points are generated. Subsequently, the object's elements are tracked by aligning the regions for each element with the positions for at least one element, and the characteristics of the regions may be modified based on a request for modification, thereby transforming the frames of the video stream. Depending on a specific request for modification, the characteristics of the mentioned regions may be transformed in different ways. Such modifications may involve changing the color of the regions; removing at least a portion of the regions from the frames of the video stream; including one or more new objects in the regions based on the modification request; and modifying or distorting the regions or the object's elements. In various embodiments, any combination of these modifications or other similar modifications may be used. For specific models to be animated, some feature points can be selected as control points to be used to determine the entire state-space of options for model animation.

[0041] In some embodiments of computer animation models that transform image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). Then, an Active Shape Model (ASM) algorithm is applied to the face region of the image to detect face feature reference points.

[0042] In other embodiments, other methods and algorithms suitable for face detection may be used. For example, in some embodiments, features are located using landmarks representing distinguishable points present in most of the images under consideration. For face landmarks, for example, the location of the left pupil may be used. When the initial landmark is not identifiable (e.g., when a person is wearing an eye patch), auxiliary landmarks may be used. These landmark identification procedures may be used for any of these objects. In some embodiments, a set of landmarks forms a shape. Shapes may be represented as vectors using the coordinates of points within the shape. One shape is aligned with another using a similarity transformation (allowing for translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0043] In some embodiments, a search for landmarks from the average shape aligned with the position and size of the face determined by the whole face detector is initiated. Subsequently, this search proposes a provisional shape by adjusting the positions of the shape points by template matching of the image texture around each point, and then repeats the steps of matching the provisional shape to the global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model pools the results of weak template matchers to form a stronger whole classifier. The whole search is repeated at each level of the image pyramid from coarse resolution to fine resolution.

[0044] Embodiments of a transformation system can capture an image or video stream on a client device (e.g., client device (102)) and perform complex image manipulation locally on the client device (102) while maintaining a suitable user experience, computation time, and power consumption. Complex image manipulation may include size and shape changes, emotion transfers (e.g., changing a face from a frown to a smile), state transfers (e.g., aging an object, reducing its apparent age, changing its gender), style transfers, the application of graphic elements, and any other suitable image or video manipulation implemented by a convolutional neural network configured to be executed efficiently on the client device (102).

[0045] In some exemplary embodiments, a computer animation model for transforming image data may be used by a system capable of capturing a user's image or video stream (e.g., a selfie) using a client device (102) having a neural network that operates as part of a messaging client application (104) operating on a client device (102). A transformation system operating within the messaging client application (104) determines the presence of a face in the image or video stream and provides modification icons associated with the computer animation model to transform the image data, or the computer animation model may exist as being associated with the interface described herein. The modification icons include changes that may be the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., creating a smiling face on the user). In some embodiments, the modified image or video stream may be presented in a graphical user interface displayed on a mobile client device as soon as the image or video stream is captured and a specified modification is selected. The transformation system can implement a complex convolutional neural network on a portion of an image or video stream to generate and apply selected modifications. That is, the user can capture an image or video stream, and once the modification icon is selected, they can be presented with the modified result in real-time or near real-time. Additionally, the modification can be persistent as long as the video stream is being captured and the selected modification icon remains toggled.Machine-taught neural networks can be used to enable these modifications.

[0046] In some embodiments, a graphical user interface presenting modifications performed by a transformation system may provide the user with additional interaction options. These options may be based on an interface used to initiate the capture and selection of content from a specific computer animation model (e.g., an initiation from a content creator user interface). In various embodiments, modifications may be persistent after an initial selection of a modification icon. The user may toggle the modification on or off by tapping or otherwise selecting the face being modified by the transformation system, and may save it for later viewing or browsing to other areas of the imaging application. If multiple faces are modified by the transformation system, the user may toggle the modification on or off globally by tapping or selecting a single face that is modified and displayed within the graphical user interface. In some embodiments, individual faces may be modified individually among a group of multiple faces, or these modifications may be toggled individually by tapping or selecting an individual face or a series of individual faces displayed within the graphical user interface.

[0047] In some exemplary embodiments, a graphics processing pipeline architecture is provided that enables different augmented reality experiences (e.g., AR content generators) to be applied to corresponding different layers. This graphics processing pipeline provides a scalable rendering engine for providing a number of augmented reality experiences contained in composite media (e.g., images or videos) or composite AR content for rendering by a messaging client application (104) (or messaging system (100)).

[0048] As mentioned above, the video table (310), in one embodiment, stores video data associated with messages for which records are maintained in the message table (314). Similarly, the image table (308) stores image data associated with messages for which message data is stored in the entity table (302). The entity table (302) can associate various comments from the comment table (312) with various images and videos stored in the image table (308) and the video table (310).

[0049] The story table (306) stores data regarding collections of messages and associated image, video, or audio data that are compiled into collections (e.g., stories or galleries). The creation of a specific collection may be initiated by a specific user (e.g., each user for whom a record is maintained in the entity table (302)). The user may create a 'personal story' in the form of a collection of content created and transmitted / broadcasted by that user. To this end, the user interface of the messaging client application (104) may include a user-selectable icon to enable the transmitting user to add specific content to their personal story.

[0050] The collection may also constitute a 'Live Story,' which is a collection of content from multiple users generated manually, automatically, or using a combination of manual and automatic techniques. For example, a 'Live Story' may constitute a curated stream of user-submitted content from various locations and events. For users whose client devices are location-serviceable and who are at a common location event at a specific time, an option to contribute content to a specific Live Story may be presented, for example, through the user interface of a messaging client application (104). A Live Story may be identified to the user by the messaging client application (104) based on their location. The final result is a 'Live Story' as described in community terms.

[0051] An additional type of content collection is known as a 'location story,' which enables a client device (102) to allow a user located within a specific geographical location (e.g., a college or university campus) to contribute to a specific collection. In some embodiments, contribution to a location story may require a second degree of authentication to verify that the end user belongs to a specific organization or other entity (e.g., a student on a university campus).

[0052] FIG. 4 is a schematic diagram illustrating the structure of a message (400) according to some embodiments, generated by the messaging client application (104) or the messaging client application (104) for communication with an additional messaging client application (104) or messaging server application (114). The content of a particular message (400) is used to populate a message table (314) stored in a database (120) accessible by the messaging server application (114). Similarly, the content of the message (400) is stored in memory as 'in transit' or 'in flight' data of the client device (102) or application server (112). The message (400) is illustrated as comprising the following components:

[0053] Message identifier (402): A unique identifier that identifies the message (400).

[0054] Message text payload (404): Text to be generated by the user through the user interface of the client device (102) and included in the message (400).

[0055] Message image payload (406): Image data captured by the camera component of the client device (102) or retrieved from the memory component of the client device (102) and included in the message (400).

[0056] Message video payload (408): Video data captured by the camera component or retrieved from the memory component of the client device (102) and included in the message (400).

[0057] Message audio payload (410): Audio data captured by a microphone or retrieved from a memory component of a client device (102) and included in a message (400).

[0058] Message annotation (412): Annotation data (e.g., filters, stickers, or other enhancements) representing annotations to be applied to the message image payload (406), message video payload (408), or message audio payload (410) of the message (400).

[0059] Message duration parameter (414): A parameter value indicating the amount of time in seconds during which the content of a message (e.g., message image payload (406), message video payload (408), message audio payload (410)) is presented to or made accessible to a user through a messaging client application (104).

[0060] Message geolocation parameter (416): geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter (416) values ​​may be included in the payload, and each of these parameter values ​​is associated with content items included in the content (e.g., a specific image in the message image payload (406), or a specific video in the message video payload (408)).

[0061] Message Story Identifier (418): Identifier values ​​that identify one or more content collections (e.g., 'story') to which a specific content item within the message image payload (406) of the message (400) is associated. For example, multiple images within the message image payload (406) may each be associated with multiple content collections using identifier values.

[0062] Message Tag (420): Each message (400) may be tagged with multiple tags, each of which indicates the subject of the content included in the message payload. For example, if a specific image included in the message image payload (406) depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tag (420). The tag values ​​may be generated manually based on user input or automatically using, for example, image recognition.

[0063] Message sender identifier (422): An identifier representing the user of the client device (102) to which the message (400) was created and the message (400) was sent (e.g., a messaging system identifier, an email address, or a device identifier).

[0064] Message recipient identifier (424): An identifier representing the user of the client device (102) to which the message (400) is addressed (e.g., a messaging system identifier, an email address, or a device identifier).

[0065] The contents (e.g., values) of the various components of the message (400) may be pointers to locations within tables where content data values ​​are stored. For example, an image value within the message image payload (406) may be a pointer (or address) to a location within the image table (308). Similarly, values ​​within the message video payload (408) may point to data stored within the video table (310), values ​​within the message annotation (412) may point to data stored within the annotation table (312), values ​​within the message story identifier (418) may point to data stored within the story table (306), and values ​​within the message sender identifier (422) and message receiver identifier (424) may point to user records stored within the entity table (302).

[0066] As described above, media overlays, such as AR content generators, overlays, image transformations, AR images, and similar terms, refer to modifications that can be made to videos or images. This includes real-time modifications, where an image is captured using device sensors and then displayed on the device screen along with the modifications. This also includes modifications to stored content, such as video clips within a gallery that can be modified. For example, on a device with access to multiple media overlays (e.g., AR content generators), a user may use a single video clip containing multiple AR content generators to see how different AR content generators modify the stored clip. For example, multiple AR content generators applying different pseudo-random motion models may be applied to the same content by selecting different AR content generators for the content. Similarly, real-time video capture with the exemplified modifications may be used to show how video images currently being captured by the device's sensors will modify the captured data. This data may simply be displayed on the screen and not stored in memory, or content captured by device sensors may be written to and stored in memory with or without modification (or both). In some systems, a preview feature may show how different AR content generators will appear simultaneously within different windows on the display. This may, for example, allow multiple windows with different pseudo-random animations to be displayed simultaneously on the display.

[0067] Accordingly, data and various systems using AR content generators or other such transformation systems to modify content using this data may involve the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), the tracking of these objects as they leave, enter, and move around within video frames, and the modification or transformation of these objects when they are tracked. In various embodiments, different methods may be used to achieve these transformations. For example, some embodiments may involve generating a 3D mesh model of an object or objects, and using a transformation of the model within the video and an animated texture to achieve the transformation. In other embodiments, by using the tracking of points on the object, an image or texture (which may be 2D or 3D) may be placed at the tracking location. In yet other embodiments, neural network analysis of video frames may be used to place images, models, or textures in the content (e.g., images or frames of the video). Therefore, lens data refers to the images, models, and textures used to generate transformations from content, as well as both the additional modeling and analysis information required to achieve these transformations through object detection, tracking, and placement.

[0068] Real-time video processing can be performed with any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the device's memory, or generate video streams using the device's sensors. Additionally, any objects, such as human faces and parts of the human body, animals, or inanimate objects like chairs, cars, or other objects, can be processed using computer animation models.

[0069] In some embodiments, when a specific modification is selected along with the content to be transformed, the elements to be transformed are identified by a computing device and subsequently detected and tracked if they exist in the frames of the video. The elements of the object are modified according to a request for modification, thereby transforming the frames of the video stream. The transformation of the frames of the video stream may be performed by different methods for different types of transformations. For example, for transformations of frames that primarily refer to changing the shapes of the elements of the object, characteristic points for each element of the object are calculated (e.g., using the Active Shape Model (ASM) or other known methods). Then, a mesh based on the characteristic points is generated for each of at least one element of the object. This mesh is used in the next stage of tracking the elements of the object within the video stream. In the tracking process, the mesh mentioned for each element is aligned with the location of each element. Subsequently, additional points are generated on the mesh. A first set of first points is generated for each element based on a request for modification, and a second set of points is generated for each element based on the first set of points and the request for modification. Then, frames of a video stream can be transformed by modifying the elements of an object based on the mesh and the first and second sets of points. In such a method, the background of the modified object can likewise be altered or distorted by tracking and modifying the background.

[0070] In one or more embodiments, transformations that modify parts of an object using the object's elements may be performed by calculating characteristic points for each element of the object and generating a mesh based on the calculated characteristic points. Points are generated on the mesh, and then various regions based on the points are generated. Subsequently, the object's elements are tracked by aligning the regions for each element with the positions for at least one element, and the characteristics of the regions may be modified based on a request for modification, thereby transforming the frames of the video stream. Depending on a specific request for modification, the characteristics of the mentioned regions may be transformed in different ways. Such modification may involve changing the color of the regions; removing at least a portion of the regions from the frames of the video stream; including one or more new objects in the regions based on the modification request; and modifying or distorting the regions or the object's elements. In various embodiments, any combination of these modifications or other similar modifications may be used. For specific models to be animated, some feature points can be selected as control points to be used to determine the entire state-space of options for model animation.

[0071] In some embodiments of computer animation models that transform image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). Then, an Active Shape Model (ASM) algorithm is applied to the face region of the image to detect face feature reference points.

[0072] In other embodiments, other methods and algorithms suitable for face detection may be used. For example, in some embodiments, features are located using landmarks representing distinguishable points present in most of the images under consideration. For face landmarks, for example, the location of the left pupil may be used. When the initial landmark is not identifiable (e.g., when a person is wearing an eye patch), auxiliary landmarks may be used. These landmark identification procedures may be used for any of these objects. In some embodiments, a set of landmarks forms a shape. Shapes may be represented as vectors using the coordinates of points within the shape. One shape is aligned with another using a similarity transformation (allowing for translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0073] In some embodiments, a search for landmarks from the average shape aligned with the position and size of the face determined by the whole face detector is initiated. Subsequently, this search proposes a provisional shape by adjusting the positions of the shape points by template matching of the image texture around each point, and then repeats the steps of matching the provisional shape to the global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model pools the results of weak template matchers to form a stronger whole classifier. The whole search is repeated at each level of the image pyramid from coarse resolution to fine resolution.

[0074] Embodiments of a transformation system can capture an image or video stream on a client device and perform complex image manipulations locally on a client device, such as a client device (102), while maintaining a suitable user experience, computation time, and power consumption. Complex image manipulations may include size and shape changes, emotion transfer (e.g., changing a face from a frown to a smile), state transfer (e.g., aging of a subject, reduction of apparent age, change of gender), style transfer, graphic element application, and any other suitable image or video manipulations implemented by a convolutional neural network configured to run efficiently on the client device.

[0075] In some exemplary embodiments, a computer animation model for transforming image data may be used by a system capable of capturing a user's image or video stream (e.g., a selfie) using a client device (102) having a neural network that operates as part of a messaging client application (104) operating on a client device (102). A transformation system operating within the messaging client application (104) determines the presence of a face in the image or video stream and provides modification icons associated with the computer animation model to transform the image data, or the computer animation model may exist as being associated with the interface described herein. The modification icons include changes that may be the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., creating a smiling face on the user). In some embodiments, the modified image or video stream may be presented in a graphical user interface displayed on a mobile client device as soon as the image or video stream is captured and a specified modification is selected. The transformation system can implement a complex convolutional neural network on a portion of an image or video stream to generate and apply selected modifications. That is, the user can capture an image or video stream, and once the modification icon is selected, they can be presented with the modified result in real-time or near real-time. Additionally, the modification can be persistent as long as the video stream is being captured and the selected modification icon remains toggled.Machine-taught neural networks can be used to enable these modifications.

[0076] In some embodiments, a graphical user interface presenting modifications performed by a transformation system may provide the user with additional interaction options. These options may be based on an interface used to initiate content capture and the selection of a specific computer animation model (e.g., initiation from a content creator user interface). In various embodiments, modifications may be persistent after the initial selection of a modification icon. The user may toggle the modification on or off by tapping or otherwise selecting the face being modified by the transformation system. The user may also save it for later viewing or browsing to other areas of the imaging application. If multiple faces are modified by the transformation system, the user may toggle the modification on or off globally by tapping or selecting a single face that is modified and displayed within the graphical user interface. In some embodiments, individual faces may be modified individually among a group of multiple faces, or these modifications may be toggled individually by tapping or selecting an individual face or a series of individual faces displayed within the graphical user interface.

[0077] In some exemplary embodiments, a graphics processing pipeline architecture is provided that enables different media overlays to be applied to corresponding different layers. This graphics processing pipeline provides a scalable rendering engine for providing a plurality of augmented reality content generators included in composite media (e.g., images or videos) or composite AR content for rendering by a messaging client application (104) (or messaging system (100)).

[0078] As discussed in this specification, the target infrastructure supports the creation and sharing of interactive messages having interactive effects across various components of the messaging system (100). In example, to provide such interactive effects, a given interactive message may include image data along with 2D data or 3D data. The infrastructure as described in this specification enables other forms of 3D and interactive media (e.g., 2D media content) to be provided across the target system, which enables such interactive media to be shared along with photo and video messages across the messaging system (100). In the exemplary embodiments described in this specification, messages may enter the system from a live camera or from a storage (e.g., messages having 2D or 3D content or augmented reality (AR) effects (e.g., 3D effects, or other interactive effects) are stored in memory or a database). In the example of an interaction message with 3D data, the target system supports motion sensor input and manages the transmission and storage of 3D data, as well as the loading of external effects and asset data.

[0079] As mentioned above, the interaction message includes 2D effects, or 3D effects and an image combined with depth data. In an exemplary embodiment, in addition to traditional image textures, the message is rendered using a target system to visualize the spatial details / geometry of what the camera sees. When a viewer interacts with this message by moving the client device, the movement triggers a corresponding change in the perspective from which the image and geometry are rendered to the viewer.

[0080] In an embodiment, the system provides an AR effect (which may include a 3D effect using 3D data, or an interactive 2D effect not using 3D data) that works in conjunction with other components of the system to provide particles, shaders, 2D assets, and 3D geometry that may reside in different 3D planes within a message. In one example, an AR effect as described herein is rendered in real-time for the user.

[0081] As mentioned in this specification, gyro-based interaction refers to a type of interaction in which the rotation of a given client device is used as an input to change the mode of effect (e.g., rotating a phone along the x-axis to change the color of light in a scene).

[0082] As mentioned in this specification, an augmented reality content generator refers to real-time special effects and / or sounds that can be added to a message, and modifies images and / or 3D data with other 3D content such as AR effects and / or 3D animated graphic elements, 3D objects (e.g., non-animated), etc.

[0083] The following discussion relates to exemplary data stored in connection with such messages according to some embodiments.

[0084] FIG. 5 is a schematic diagram illustrating the structure of message annotations (412) as described in FIG. 4, which include additional information corresponding to a given message according to some embodiments, generated by a messaging client application (104) or by the messaging client application (104).

[0085] In one embodiment, as illustrated in FIG. 3, the content of a specific message (400) containing additional data illustrated in FIG. 5 is used to populate a message table (314) stored in a database (120) for a given message, and the given message is then accessible by a messaging client application (104). As illustrated in FIG. 5, a message annotation (412) includes the following components corresponding to various data:

[0086] o Augmented Reality (AR) Content Identifier (552): Identifier of the AR content generator used in the message

[0087] o Message identifier (554): Identifier of the message

[0088] o Asset identifiers (556): A set of identifiers for assets within a message. For example, each asset identifier may be included for an asset determined by a specific AR content generator. In an embodiment, such assets are generated by an AR content generator on a sender-side client device, uploaded to a messaging server application (114), and used on a receiver-side client device to regenerate the message. Examples of typical assets include:

[0089] - Original clear RGB image(s) captured by the camera

[0090] - Post-processed image(s) with AR content generator effects applied to the original image

[0091] o Augmented Reality (AR) Content Metadata (558): Additional metadata associated with an AR content generator corresponding to an AR identifier (552), e.g.:

[0092] o AR Content Generator Category: Corresponding to the type or classification of a specific AR content generator

[0093] o AR Content Generator Carousel Index

[0094] o Carousel Group: This can be populated and used when AR content generators are inserted into the carousel interface after eligible capture. In the implementation, a new value "AR_DEFAULT_GROUP" (e.g., the default group assigned to the AR content generator) can be added to the list of valid group names.

[0095] o Capture metadata (560) corresponding to the following additional metadata:

[0096] o Camera image metadata

[0097] - Camera unique data

[0098] · Focal length

[0099] · Origin

[0100] - Other camera information (e.g., camera location)

[0101] o Sensor Information

[0102] - Gyroscope sensor data

[0103] - Location sensor data

[0104] - Accelerometer sensor data

[0105] - Other sensor data

[0106] - Location sensor data

[0107] FIG. 6 is a block diagram illustrating various modules of a messaging client application (104) according to specific exemplary embodiments. The messaging client application (104) is illustrated as including an AR content system (600). As further illustrated, the AR content system (600) includes a camera module (602), a capture module (604), an image data processing module (606), a rendering module (608), and a content recording module (610). The various modules of the AR content system (600) are configured to communicate with each other (e.g., via a bus, shared memory, or a switch). Since one or more of these modules may be implemented using one or more computer processors (620) (e.g., by configuring these one or more computer processors to perform the functions described for the module), one or more of the computer processors (620) may be included (e.g., a set of processors provided by the client device (102)).

[0108] One or more of the described modules may be implemented using hardware alone (e.g., one or more of the computer processors (620) of the machine (e.g., machine (1400)) or a combination of hardware and software. For example, any described module of a messaging client application (104) may physically include an arrangement of one or more computer processors (620) configured to perform the operations described herein for that module (e.g., one or more of the computer processors of the machine (e.g., machine (1400)) or a subset thereof). As another example, any module of an AR content system (600) may include software, hardware, or both that constitute an arrangement of one or more computer processors (620) (e.g., among one or more of the computer processors of the machine (e.g., machine (1400)) to perform the operations described herein for that module. Accordingly, different modules of the AR content system (600) may include and be configured with different arrays of these computer processors (620) or a single array of these computer processors (620) at different points in time. Furthermore, any two or more modules of the messaging client application (104) may be combined into a single module, and the functions described herein for the single module may be subdivided among multiple modules. Additionally, according to various exemplary embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0109] The camera module (602) performs camera-related operations including functionality for operations involving one or more cameras of the client device (102). In example, the camera module (602) may access camera functionality across different processes running on the client device (102) to determine surfaces for face or surface tracking, and in response to various requests for camera data or image data (e.g., frames) from such processes (e.g., involving image data of a specific resolution or format), provide metadata to such processes consuming the requested camera data or image data. As mentioned herein, "process" or "computing process" may refer to an instance of a computer program being executed by one or more threads of a given processor(s).

[0110] As mentioned herein, surface tracking refers to operations for tracking one or more representations of surfaces corresponding to planes (e.g., a given horizontal plane, floor, table) in an input frame. In one example, surface tracking is achieved using hit testing and / or ray casting techniques. Hit testing, in example, determines whether a selected point (e.g., a pixel or a set of pixels) within an input frame intersects a surface or plane of a representation of a physical object within the input frame. Ray casting, in example, uses a Cartesian-based coordinate system (e.g., x and y coordinates) and projects a ray (e.g., a vector) into a camera view of the world as captured in the input frame to detect planes where the ray intersects.

[0111] As further illustrated, the camera module (602) receives an input frame (or alternatively, a copy of the input frame in the embodiment). The camera module (602) may include various tracking capabilities based on the type of object to be tracked. In the example, the camera module (602) includes tracking capabilities for surface tracking, face tracking, object tracking, etc. In the implementation, the camera module (602) may execute only one of a plurality of tracking processes at a time to facilitate the management of computing resources on the client device (102) or the client device (102). Additionally, the camera module (602) may perform one or more object recognition or detection operations on the input frame.

[0112] As mentioned herein, tracking refers to operations for determining spatial characteristics (e.g., position and / or orientation) of a given object (or part thereof) during a post-processing stage. In an implementation, during tracking, the position and orientation of the object are measured in a continuous manner. Different objects, such as a user's head, eyes, or limbs, surfaces, or other objects, may be tracked. Tracking involves dynamic sensing and measurement to enable virtual objects and / or effects to be rendered for physical objects in a three-dimensional space corresponding to a scene (e.g., an input frame). Accordingly, the camera module (602) determines metrics corresponding to at least the relative position and orientation of one or more physical objects in the input frame and includes these metrics in the tracking data provided to the rendering module (608). In an example, the camera module (602) updates these metrics from frame to subsequent frame (e.g., tracking over time).

[0113] In an implementation, the camera module (602) provides tracking data (e.g., metadata) corresponding to the aforementioned metrics (e.g., position and orientation) as output. In some cases, the camera module (602) includes logic for shape recognition, edge detection, or any other suitable object detection mechanism. The object of interest may also be determined by the camera module (602) to be an example of a matching shape, edge, or landmark within the range of an object type of a predetermined object type, or a set of predetermined object types.

[0114] In an implementation, the camera module (602) may use techniques to combine information from the device's motion sensors (e.g., accelerometer and gyroscope sensors, etc.) with an analysis of the scene provided in the input frame. For example, the camera module (602) detects features in the input frame and, as a result, tracks the differences at each location of such features across multiple input frames using information derived at least partially from the data from the device's motion sensors.

[0115] As mentioned herein, face tracking refers to operations for tracking representations of facial features, such as parts of a user's face, in an input frame. In some embodiments, the camera module (602) includes face tracking logic for identifying all or part of a face in one or more images and tracking landmarks of the face across a set of images in a video stream. As mentioned herein, object tracking refers to tracking representations of physical objects in an input frame.

[0116] In an embodiment, the camera module (602) uses machine learning techniques to detect whether a physical object corresponding to a representation on a display screen is included in the captured image data (e.g., from the current field of view of the client device (102)).

[0117] In the example, the camera module (602) utilizes a machine learning model so that this neural network is used to detect a representation of the display screen in the image data. The neural network model may refer to a feedforward deep neural network implemented to approximate a function f. In this regard, the models are referred to as feedforward because information flows from the input x through a function evaluated, through one or more intermediate operations used to define f, and finally to the output y. Feedforward deep neural networks are referred to as networks because the networks can be represented by connecting different operations together. A model of a feedforward deep neural network can be represented as a graph representing how operations are connected together from the input layer, through one or more hidden layers, and finally to the output layer. Each node in this graph represents an operation to be performed in the example. However, it is understood that other types of neural networks are considered by the implementations described herein. For example, recurrent neural networks such as long short-term memory (LSTM) neural networks can be provided for annotation, or convolutional neural networks (CNN) can be used.

[0118] In the example, regarding the computer vision techniques of the present invention, the camera module (602) detects a representation of a display screen (or other applicable objects) in image data using a convolutional neural network model. Such a convolutional neural network (CNN) may be trained using training data containing thousands or millions of images of display screens, provided that input data (e.g., image or video data) is provided to the trained CNN and the CNN can perform tasks to detect the presence of display screen(s) in the input data. Convolutional operations involve discovering local patterns in input data, such as image data. Thus, such patterns learned by the CNN can be recognized in any other part of the image data, which advantageously provides transformation-invariant capabilities. For example, an image of a display screen viewed from the side can still generate an accurate classification of the display screen as if the display screen were viewed from the front. Similarly, in cases of occlusion, where the object to be detected (e.g., a display screen) is partially blocked from view, the CNN can still detect the object in the image data.

[0119] In an embodiment, the camera module (602) acts as an intermediary between other components of the AR content system (600) and the capture module (604). As mentioned above, the camera module (602) can receive requests for captured image data from the image data processing module (606). The camera module (602) can also receive requests for captured image data from the content recording module (610). The camera module (602) can forward these requests to the capture module (604) for processing.

[0120] The capture module (604) captures images (which may also include depth data) captured by one or more cameras of the client device (102) (e.g., in response to the aforementioned requests from other components). For example, an image is a photograph captured by an optical sensor (e.g., a camera) of the client device (102). The image includes one or more real-world features, such as a user's face or real-world object(s) detected in the image. In some embodiments, the image includes metadata describing the image. Each captured image may be included in a data structure described herein as a "frame" that may include raw image data along with metadata and other information. In an embodiment, the capture module (604) may transmit the captured image data and metadata as (captured) frames to one or more components of the AR content system (600).

[0121] The image data processing module (606) generates tracking data and other metadata for the captured image data, including metadata associated with actions for generating AR effects and AR content applied to the captured image data. The image data processing module (606) performs actions on the received image data. For example, various image processing actions are performed by the image data processing module (606). The image data processing module (606) performs various actions based on algorithms or techniques corresponding to animations and / or provides visual and / or auditory effects to the received image data. In an embodiment, a given augmented reality content generator may utilize the image data processing module (606) to perform actions as part of generating AR content and AR effects, which is then provided to a rendering process to render such AR content and AR effects (e.g., including 2D effects or 3D effects).

[0122] The rendering module (608) performs rendering of AR content for display by the messaging client application (104) based on data provided by at least one of the aforementioned modules. In an example, the rendering module (608) uses a graphics processing pipeline to perform graphics operations for rendering AR content for display. The rendering module (608) implements, in an example, an extensible rendering engine that supports a plurality of image processing operations corresponding to each augmented reality content generator. In an example, the rendering module (608) can receive composite AR content for rendering on a display provided by the client device (102).

[0123] In some implementations, the rendering module (608) provides a graphics system that renders two-dimensional (2D) objects or objects from a three-dimensional (3D) world (real or virtual) onto a 2D display screen. Such a graphics system (e.g., included on a client device (102)) includes, in some implementations, a graphics processing unit (GPU) for performing image processing operations and rendering graphic elements for display.

[0124] In an implementation, the GPU includes a logical graphics processing pipeline capable of receiving a representation of a 2D or 3D scene and providing output of a bitmap representing a 2D image for display. Existing application programming interfaces (APIs) have implemented graphics pipeline models. Examples of such APIs include the Open Graphics Library (OPENGL) API and the METAL API. The graphics processing pipeline includes multiple stages for converting a group of vertices, textures, buffers, and state information into an image frame on the screen. In an implementation, one of the stages of the graphics processing pipeline is a shader, which can be used as part of a specific augmented reality content generator applied to an input frame (e.g., an image or video). The shader can be implemented as code running on a specialized processing unit, also referred to as a shader unit or shader processor, which typically runs multiple computing threads and is programmed to generate appropriate levels of color and / or special effects for the fragments being rendered. For example, vertex shaders handle vertex attributes (position, texture coordinates, color, etc.), and pixel shaders handle pixel attributes (texture values, color, z-depth, and alpha values). In some cases, pixel shaders are referred to as fragment shaders.

[0125] It should be understood that other types of shader processes may be provided. In the example, a specific sampling rate is used within the graphics processing pipeline to render the entire frame and / or pixel shading is performed at a specific per-pixel rate. In this way, a given electronic device (e.g., client device (102)) operates the graphics processing pipeline to convert information corresponding to objects into a bitmap that can be displayed by the electronic device.

[0126] The content recording module (610) transmits request(s) to the camera module (602) to initiate recording of image data by one or more cameras provided by the client device (102). In an embodiment, the camera module (602) acts as an intermediary between other components within the AR content recording system. For example, the camera module may receive a request to initiate recording from the content recording module (610) and forward the request to the capture module (604) for processing. When the capture module (604) receives a request from the camera module (602), it performs operations to initiate the capture of image data by the camera(s) provided by the client device (102). The captured image data, including timestamp information for each frame of the captured image data, may then be transmitted to the content recording module (610) for processing. In an example, the content recording module (610) may perform operations to process the captured image data for rendering by the rendering module (608).

[0127] In an embodiment, the components of the AR content system (600) can communicate using an IPC (inter-process communication) protocol. In an embodiment, the components of the AR content system (600) can communicate through an API provided by the AR content system (600).

[0128] In an embodiment, the camera module (602) receives a signal or command (or request) to stop recording image data (e.g., transmitted from the content recording module (610)). In response, the camera module (602) transmits a request to the capture module (604) to stop capturing image data. The capture module (604), in response to the request to stop recording, complies with the request and stops additional operations to capture image data using one or more cameras of the client device (102). After receiving the signal or command to stop recording, the camera module (602) may also asynchronously transmit a signal to the image data processing module (606) indicating that the recording of image data (e.g., the capture of image data by the capture module (604)) has been stopped (or requested to be stopped). After receiving the signal, the image data processing module (606) performs operations to complete or complete image processing operations, including performing operations to generate metadata related to AR content and AR effects. After that, this metadata can be transmitted to a capture module (604), and the capture module (604) then generates composite AR content including the metadata. The composite AR content can be received by a rendering module (608) and rendered to be displayed on a display device provided by a client device (102).

[0129] As mentioned in this specification, the present technology enables operations related to facial animation synthesis (e.g., image processing operations) as described below. Some embodiments of the present disclosure may allow taking source media content (e.g., images, videos, etc.) of a first person ("source actor") and setting target photos (or videos) of a second entity, such as a second person (hereinafter referred to as "target actor" or "target entity"), as input, and synthesizing the animation of the target actor with facial mimics and head movements of the source actor. The present technology enables the target actor to be animated and thereby mimic the movements and facial expressions of the source actor. In an embodiment, the present technology may be used in an entertainment or advertising context where a user takes a selfie (e.g., media content including images or videos), and the present technology may select a scenario for animating the person and applying visual effects. The scenarios have different settings and source actor movements, which are transmitted to the media content corresponding to the user selfie. The resulting media content (e.g., AR content including face animation composite) may feature the user in different situations and locations. The user may share the AR content including face animation composite with other users (e.g., friends). Additionally, the AR content including face animation composite may be used as stickers (e.g., media overlays including AR content) in messaging applications (e.g., messaging client application (104)) or social networking services (e.g., social network system (122)), or as content for online advertisements to be displayed in various situations as further described herein.

[0130] In some embodiments, the system may manipulate or modify a target face based on the facial expressions of a source face by performing face synthesis operations that enable real-time mimicry of the head positions of the source actor and the facial expressions of the source actor. Additionally, in some embodiments, technical improvements to the operation of the computing device include significantly reducing the computation time for generating AR content in which the face of the target entity mimics the head positions of the source actor and the facial expressions of the source actor, and allowing such generation of AR content to be performed on a mobile device that may have limited computing resources.

[0131] In some embodiments, the client device (102) may be configured to display target media content (e.g., an image or video). The target media content may include at least one frame containing the face of the target entity to which face synthesis is to be applied based on the face of the source actor (e.g., a user). In some embodiments, the target media content may include a single image (e.g., a still or static image instead of a video). In some embodiments, the target media content may be written and stored in advance in the memory of the client device (102) or in a cloud-based computing resource to which the client device (102) is communically coupled, such as a messaging server system (108).

[0132] In the example, the camera module (602) can capture source video through, for example, the camera of the client device (102). The source video may include at least the user's face (e.g., "source face") and may be stored in the memory of the client device (102).

[0133] According to some embodiments, a client device (102) (e.g., an image data processing module (606)) or a messaging server system (108) may be configured to analyze stored images of a given user (e.g., a single image or multiple frames of a source video) to extract the user's face parameters. The client device (102) or the messaging server system (108) may be further configured to modify a target video by replacing the target face in the target video with the user's face using face synthesis techniques based on the user's face parameters.

[0134] Similarly, the client device (102) or the messaging server system (108) may be configured to analyze the user's stored images to extract face parameters of another individual (e.g., the user's friend). The client device (102) may be further configured to modify the target video by using face synthesis technologies based on the individual's face parameters to replace the target face within the target video with the individual's face.

[0135] As mentioned in this specification, these face synthesis technologies may include, for example, determining facial expressions and head poses of a source actor, determining facial landmarks of the source actor and replacing the identity parameters of the source actor with the identity parameters of the target actor, using machine learning models including neural networks, and generating a realistic and plausible-looking frame sequence (e.g., video) of the target actor’s head that expresses and moves emotions (e.g., facial expressions or facial movements) extracted from the source actor.

[0136] FIG. 7 illustrates an interface (700) (e.g., a graphical user interface) for selecting a type of advertisement for an online advertising campaign in the messaging system according to some embodiments. In an embodiment, the interface (700) is provided by a messaging client application (104) and / or a messaging server system (108) and can be accessed by the client device (102) to be presented to the user on the display screen of the client device (102).

[0137] As illustrated, the interface (700) includes various parameters (e.g., options) corresponding to each selectable graphic item for selecting the type of online advertisement. In the example, a user can use the interface (700) to select one of the aforementioned graphic items. As illustrated, the user selected a graphic item to select an online advertisement based on AR content including a face animation composite (e.g., "Cameos ad").

[0138] FIG. 8 illustrates an interface (800) for configuring one or more features of an online advertisement based on AR content including face animation synthesis in the messaging system according to some embodiments. In an embodiment, the interface (800) is provided by a messaging client application (104) and / or a messaging server system (108) and can be accessed by the client device (102) to be presented to a user on the display screen of the client device (102).

[0139] As described, the interface (800) includes various characteristics (e.g., options) corresponding to each graphic item for configuring an online advertisement. In the example, a user may use the interface (800) to provide input for each characteristic, such as a name, brand name, or headline for the online advertisement. The interface (800) further includes a graphic area that displays a preview of the online advertisement based on the input characteristics, and also includes selectable graphic items for creating an online advertisement, including uploading media content and compositing face animations.

[0140] FIG. 9 illustrates an interface (900) including an example of an online advertisement comprising media content accompanied by face animation synthesis in the messaging system according to some embodiments. In an embodiment, the interface (900) is provided by a messaging client application (104) and / or a messaging server system (108) and can be accessed by a client device (102) to be presented to a user on a display screen of a client device (102).

[0141] As described, the interface (900) includes a representation (910) of the source actor's face applied to a representation (915) of a target entity using face synthesis technologies. The interface (900) further includes an online advertisement (920) that may be based on the characteristics described in relation to FIG. 8. In this example, a video sequence (e.g., less than 10 seconds in length) that animates the representation (910) of the source actor's face based on the face synthesis technologies described herein is presented for display on a client device (102).

[0142] FIG. 10 illustrates an interface (1000) including an example of a content creation application (e.g., a media content editor) in the messaging system according to some embodiments. In an embodiment, the interface (1000) is provided by a client device (102) and / or a messaging server system (108) and can be accessed by the client device (102) to be presented to a user on the display screen of the client device (102).

[0143] In the example of FIG. 10, the interface (1000) includes various selectable graphic items on the left side for modifying corresponding modalities of media content (1010) (e.g., head tracking, lip synchronization video, foreground layers, background video) with face synthesis corresponding to a given online advertisement. As additionally illustrated, the interface (1000) includes a chronological sequence of frames of media content (1010) accompanied by face synthesis (e.g., timeline display) and a graphic area (1020) including playback control elements (e.g., play button).

[0144] FIG. 11 illustrates interfaces (e.g., interface (1100) and interface (1150)) including examples of online advertisements including media content accompanied by face animation synthesis in the messaging system according to some embodiments.

[0145] The interface (1100) shows an example of a static image including a user's selfie (e.g., face image (1110)) and a corresponding online advertisement (1115). The interface (1150) shows another example of a static image of a user including selectable graphic items (1160 and 1165) that enable various functionalities (e.g., installing an application and viewing a corresponding link or website).

[0146] FIG. 12 is a flowchart illustrating a method (1200) according to specific exemplary embodiments. The method (1200) may be implemented as computer-readable instructions for execution by one or more computer processors so that the operations of the method (1200) can be performed partially or wholly by a client device (102) particularly for each component of the AR content system (600) described in FIG. 6; thus, the method (1200) is described below as an example by reference. However, it should be understood that at least some of the operations of the method (1200) may be placed on various other hardware configurations and that the method (1200) is not intended to be limited to the AR content system (600).

[0147] In operation 1202, the image data processing module (606) receives frames of source media content, and the frames of source media content include representations of the head and face of the source actor.

[0148] In operation 1204, the image data processing module (606) generates sets of source pose parameters based at least partially on frames of source media content, and the sets of source pose parameters include positions of head expressions of the source actor in frames of source media content and facial expressions of the source actor.

[0149] In operation 1206, the image data processing module (606) receives at least one target image, and at least one target image includes representations of the target head and target face of the target entity.

[0150] In operation 1208, the image data processing module (606) generates output media content based at least partially on sets of source pose parameters, and each frame of the output media content includes an image of a target face in at least one frame of the output media content, and the image of the target face is modified based on at least one of sets of source pose parameters to mimic at least one of the head positions of the source actor and at least one of the facial expressions of the source actor.

[0151] In operation 1210, the rendering module (608) provides an online advertisement based at least partially on the output media content to be displayed on a computing device.

[0152] In an embodiment, the rendering module (608) further triggers the display or playback of an online advertisement based on the duration of the online advertisement in a messaging client application on a client device.

[0153] In one embodiment, the online advertisement includes a video, and the video cannot be skipped or paused during the playback of the video.

[0154] In one embodiment, the online advertisement includes an image, and the image cannot be hidden during the duration of the online advertisement.

[0155] In an embodiment, the image data processing module (606) generates sets of source pose parameters, which, using a shared encoder network, generates a first set of face features based on frames of source media content—this generation generates a first set of image data with a lower dimensionality than the frames of source media content—and the lower dimensionality includes a resolution lower than a specific resolution of the frames of source media content.

[0156] In an embodiment, the image data processing module (606) uses a shared encoder network to generate a second set of face features based on at least one target image, and this generation generates a second set of image data with a lower dimensionality than at least one target image.

[0157] In an embodiment, the step of generating output media content based at least partially on sets of source pose parameters includes an image data processing module (606) generating a first set of output images or frames using a first decoder network associated with at least one target image, and the first set of output images or frames includes modifications to the head and facial expressions of a source actor based on a second set of facial features from at least one target image.

[0158] In an embodiment, the image data processing module (606) uses a first decoder network associated with at least one target image to generate a decoded version of at least one target image, and the decoded version of at least one target image is a second representation of a target head and a target face.

[0159] In an embodiment, the image data processing module (606) generates a second set of output images or frames using a second decoder network associated with frames of source media content, and the second set of output images or frames includes decoded versions of the head and face expressions of the source actor based on the first set of face features.

[0160] In an embodiment, the shared encoder network includes a first deep convolutional neural network, the first decoder network includes a second deep convolutional neural network, the second decoder network includes a third deep convolutional neural network, and the first deep convolutional neural network, the second deep convolutional neural network, and the third deep convolutional neural network are different neural networks.

[0161] FIG. 13 is a block diagram illustrating an exemplary software architecture (1306) that can be used with various hardware architectures described herein. FIG. 13 is a non-limiting example of a software architecture, and it will be understood that many other architectures may be implemented to facilitate the functionality described herein. The software architecture (1306) may be executed on hardware such as the machine (1400) of FIG. 14, which includes, among others, processors (1404), memory (1414), and (input / output) I / O components (1418). A representative hardware layer (1352) is illustrated and may represent, for example, the machine (1400) of FIG. 14. The representative hardware layer (1352) includes a processing unit (1354) having associated executable instructions (1304). Executable instructions (1304) represent executable instructions of a software architecture (1306) including implementations of methods, components, etc. described herein. A hardware layer (1352) also includes memory / storage (1356), which are memory and / or storage modules having executable instructions (1304). A hardware layer (1352) may also include other hardware (1358).

[0162] In the exemplary architecture of FIG. 13, the software architecture (1306) can be conceptualized as a stack of layers, each layer providing specific functionality. For example, the software architecture (1306) may include layers such as an operating system (1302), libraries (1320), frameworks / middleware (1318), applications (1316), and a presentation layer (1314). Operationally, applications (1316) and / or other components within the layers may initiate API calls (1308) through the software stack and receive responses such as messages (1312) for API calls (1308). The layers illustrated are essentially representative, and not all software architectures have all layers. For example, some mobile or special-purpose operating systems may not provide frameworks / middleware (1318), while others may provide such layers. Other software architectures may include additional or different layers.

[0163] The operating system (1302) can manage hardware resources and provide common services. The operating system (1302) may include, for example, a kernel (1322), services (1324), and drivers (1326). The kernel (1322) may act as an abstraction layer between the hardware and other software layers. For example, the kernel (1322) may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services (1324) may provide other common services to other software layers. Drivers (1326) are responsible for controlling or interfacing with the underlying hardware. For example, drivers (1326) include display drivers, camera drivers, Bluetooth® drivers, flash memory drivers, serial communication drivers (e.g., USB (Universal Serial Bus) drivers), Wi-Fi® drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

[0164] Libraries (1320) provide a common infrastructure used by applications (1316) and / or other components and / or layers. Libraries (1320) provide functionality that enables other software components to perform tasks in a way that is easier than directly interfacing with the underlying operating system (1302) functionality (e.g., kernel (1322), services (1324) and / or drivers (1326)). Libraries (1320) may include system libraries (1344) (e.g., C standard library) that can provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, the libraries (1320) may include API libraries (1346), such as media libraries (e.g., libraries for supporting the presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., an OpenGL framework that can be used to render 2D and 3D graphic content on a display), database libraries (e.g., SQLite that can provide various relational database functions), web libraries (e.g., WebKit that can provide web browsing functionality). The libraries (1320) may also include a wide variety of other libraries (1348) to provide many different APIs to applications (1316) and other software components / modules.

[0165] Frameworks / middleware (1318) (sometimes referred to as middleware) provide a high-level common infrastructure that can be used by applications (1316) and / or other software components / modules. For example, frameworks / middleware (1318) can provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, etc. Frameworks / middleware (1318) can provide a wide spectrum of other APIs that can be used by applications (1316) and / or other software components / modules, some of which may be specific to a specific operating system (1302) or platform.

[0166] Applications (1316) include built-in applications (1338) and / or third-party applications (1340). Examples of representative built-in applications (1338) may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications (1340) may include applications developed using the ANDROID™ or IOS™ software development kit (SDK) by entities other than the vendor of a specific platform, and may be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Third-party applications (1340) may initiate API calls (1308) provided by a mobile operating system (such as an operating system (1302)) to facilitate the functionality described herein.

[0167] Applications (1316) may use built-in operating system functions (e.g., kernel (1322), services (1324) and / or drivers (1326)), libraries (1320), and frameworks / middleware (1318) to create user interfaces for interacting with users of the system. Alternatively or additionally, in some systems, interactions with the user may occur through a presentation layer such as a presentation layer (1314). In these systems, application / component 'logic' may be separated from the modes of application / component interacting with the user.

[0168] FIG. 14 is a block diagram illustrating components of a machine (1400) according to some exemplary embodiments capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing one or more of the methodologies discussed herein. Specifically, FIG. 14 illustrates a schematic representation of a machine (1400) in an exemplary form of a computer system, in which instructions (1410) (e.g., software, program, application, applet, app, or other executable code) may be executed to cause the machine (1400) to perform one or more of the methodologies discussed herein. Thus, instructions (1410) may be used to implement the modules or components described herein. Instructions (1410) convert a general non-programmed machine (1400) into a specific machine (1400) programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, the machine (1400) may operate as a standalone device or be coupled to other machines (e.g., networked). In a networked deployment, the machine (1400) may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.The machine (1400) may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart device), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing commands (1410) that specify actions to be taken by the machine (1400) sequentially or otherwise. Additionally, although only a single machine (1400) is exemplified, the term 'machine' should also be considered to include a collection of machines that execute commands (1410) individually or jointly to perform any one or more of the methodologies discussed herein.

[0169] The machine (1400) may include processors (1404) comprising processors (1408) to processors (1412), memory / storage (1406), and I / O components (1418), which may be configured to communicate with each other, for example, via a bus (1402). The memory / storage (1406) may include memory (1414), such as main memory or other memory storage, and a storage unit (1416), both of which are accessible to the processors (1404), for example, via the bus (1402). The storage unit (1416) and the memory (1414) store instructions (1410) that implement one or more of the methodologies or functions described herein. Instructions (1410) may also reside, wholly or partially, during the execution by the machine (1400), in memory (1414), in a storage unit (1416), in at least one of the processors (1404) (e.g., in the cache memory of a processor), or in any suitable combination thereof. Thus, memory (1414), storage unit (1416), and memory of the processors (1404) are examples of machine-readable media.

[0170] I / O components (1418) may include a wide variety of components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components (1418) included in a specific machine (1400) will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, whereas a headless server machine may not include such a touch input device. It will be understood that I / O components (1418) may include many other components not shown in FIG. 14. I / O components (1418) are grouped according to functionality merely to simplify the discussion below, and the grouping is by no means limited. In various exemplary embodiments, I / O components (1418) may include output components (1426) and input components (1428). The output components (1426) may include visual components (e.g., displays such as a plasma display panel (PDP), light-emitting diode (LED) display, liquid crystal display (LCD), projector, or cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc.Input components (1428) may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing device), haptic input components (e.g., a physical button, a touch screen providing the location and / or force of touches or touch gestures, or other haptic input components), audio input components (e.g., a microphone), etc.

[0171] In additional exemplary embodiments, I / O components (1418) may include biometric components (1430), motion components (1434), environment components (1436), or position components (1438), among a wide range of other components. For example, biometric components (1430) may include components that detect expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), and identify a person (e.g., voice identification, retinal identification, face identification, fingerprint identification, or electroencephalogram-based identification). Motion components (1434) may include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental components (1436) may include, for example, light sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting concentrations of hazardous gases for safety or measuring contaminants in the atmosphere), or other components capable of providing indications, measurements, or signals corresponding to the surrounding physical environment. Location components (1438) may include location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers for detecting atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), etc.

[0172] Communication can be implemented using a wide variety of technologies. I / O components (1418) may include communication components (1440) operable to connect the machine (1400) to a network (1432) or devices (1420) respectively through coupling (1424) and coupling (1422). For example, communication components (1440) may include a network interface component or other suitable device for interfacing with the network (1432). In additional examples, communication components (1440) may include wired communication components, wireless communication components, cellular communication components, NFC (Near Field Communication) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components that provide communication through other aspects. Devices (1420) may be any of the other machine or a wide variety of peripheral devices (e.g., peripheral devices coupled via USB).

[0173] Furthermore, the communication components (1440) may include components capable of detecting identifiers or operable to detect identifiers. For example, the communication components (1440) may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., optical sensors for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or acoustic detection components (e.g., microphones for identifying tagged audio signals). Additionally, various information such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, and location via NFC beacon signal detection capable of indicating a specific location may be derived through the communication components (1440).

[0174] The following discussion concerns various terms or phrases mentioned throughout the subject disclosure.

[0175] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or altered, such as encoding information within the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0176] "Communication network" refers to one or more parts of a network that may be an ad-hoc network, intranet, extranet, VPN (virtual private network), LAN (local area network), wireless LAN (WLAN), WAN (wide area network), wireless WAN (WWAN), MAN (metropolitan area network), the Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a coupling may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) connection, or other types of cellular or wireless coupling.In this example, the combination can implement any of the various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4th generation wireless (4G) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standard-setting organizations, other long-range protocols, or other data transmission technologies.

[0177] "Processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "instructions," "op codes," "machine code," etc.) and generates corresponding output signals applied to operate the machine. A processor may be, for example, a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0178] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Accordingly, the term should be considered to include, but not be limited to, solid-state memories including memory inside or outside processors, and optical and magnetic media. Specific examples of machine storage media, computer storage media, and / or device storage media include, by example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are included under the term "signal medium."

[0179] "Component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide the division or modularization of specific processing or control functions. Components may be combined with other components through their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed to be used with other components and a part of a program that performs a specific function among the typically related functions. Components may constitute either software components (e.g., code implemented on a machine-readable medium) or hardware components. "Hardware component" is a type of unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various exemplary embodiments, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or a part of an application) as hardware components that operate to perform specific operations as described herein. Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits that are temporarily configured by software to perform specific operations.For example, a hardware component may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely customized to perform the configured functions and are no longer general-purpose processors. It will be understood that the decision to implement a hardware component mechanically, in a dedicated, permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a type entity, that is, an entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a specific manner or perform the specific operations described herein. When considering embodiments in which hardware components are temporarily configured (e.g., programmed), it is not necessary for each hardware component to be configured or instantiated at any single time instance. For example, in the case where a general-purpose processor is configured by software such that a hardware component becomes a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., including different hardware components). Thus, the software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance. Hardware components can provide information to other hardware components and receive information from them.Accordingly, the described hardware components may be considered to be coupled communicably. In cases where multiple hardware components exist simultaneously, communication may be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information within memory structures accessible to multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device coupled communicably to it. Subsequently, additional hardware components may access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and manipulate resources (e.g., collections of information). Various operations of the exemplary methods described herein may be performed at least partially by one or more processors configured temporarily (e.g., by software) or permanently to perform the relevant operations. Whether configured temporarily or permanently, such processors may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be implemented at least partially by a processor, and specific processors or processors are examples of hardware.For example, at least some of the operations of the method may be performed by one or more processors or processor implementation components. Furthermore, one or more processors may also operate to support the performance of the relevant operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines containing processors), and these operations are accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The performance of specific operations may not only exist within a single machine but may also be distributed among processors deployed across multiple machines. In some exemplary embodiments, the processors or components implemented by the processors may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other exemplary embodiments, the processors or components implemented by the processors may be distributed across multiple geographic locations.

[0180] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0181] "Computer-readable medium" refers to both machine storage media and transmission media. Accordingly, the terms include both storage devices / medias and carriers / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0182] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network. In the subject disclosure, a client device is also referred to as an "electronic device."

[0183] "Short-term messages" refer to messages accessible for a time-limited duration. Short-term messages can be text, images, videos, etc. The access time for short-term messages can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, messages are transitory.

[0184] "Signal medium" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media for facilitating the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal in which one or more of its characteristics are set or altered, such as encoding information within the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0185] "Communication network" refers to one or more parts of a network that may be an ad-hoc network, intranet, extranet, VPN (virtual private network), LAN (local area network), wireless LAN (WLAN), WAN (wide area network), wireless WAN (WWAN), MAN (metropolitan area network), the Internet, part of the Internet, part of the PSTN (Public Switched Telephone Network), POTS (plain old telephone service) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks, or a combination of two or more of these networks. For example, a network or part of a network may include a wireless or cellular network, and a coupling may be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile communications) connection, or other types of cellular or wireless coupling.In this example, the combination can implement any of the various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, 4th generation wireless (4G) networks, UMTS (Universal Mobile Telecommunications System), HSPA (High Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other things defined by various standard-setting organizations, other long-range protocols, or other data transmission technologies.

[0186] "Processor" refers to any circuit or virtual circuit (a physical circuit emulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "instructions," "op codes," "machine code," etc.) and generates corresponding output signals applied to operate the machine. A processor may be, for example, a CPU (Central Processing Unit), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an RFIC (Radio-Frequency Integrated Circuit), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0187] "Machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Accordingly, the term should be considered to include, but not be limited to, solid-state memories including memory inside or outside processors, and optical and magnetic media. Specific examples of machine storage media, computer storage media, and / or device storage media include, by example, non-volatile memory including semiconductor memory devices, e.g., EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are included under the term "signal medium."

[0188] "Component" refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide the division or modularization of specific processing or control functions. Components may be combined with other components through their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed to be used with other components and a part of a program that performs a specific function among the typically related functions. Components may constitute either software components (e.g., code implemented on a machine-readable medium) or hardware components. "Hardware component" is a type of unit capable of performing specific operations and may be configured or arranged in a specific physical manner. In various exemplary embodiments, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or a part of an application) as hardware components that operate to perform specific operations as described herein. Hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuits or logic permanently configured to perform specific operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuits that are temporarily configured by software to perform specific operations.For example, a hardware component may include software executed by a general-purpose processor or another programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely customized to perform the configured functions and are no longer general-purpose processors. It will be understood that the decision to implement a hardware component mechanically, in a dedicated, permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass a type entity, that is, an entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a specific manner or perform the specific operations described herein. When considering embodiments in which hardware components are temporarily configured (e.g., programmed), it is not necessary for each hardware component to be configured or instantiated at any single time instance. For example, in the case where a general-purpose processor is configured by software such that a hardware component becomes a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., including different hardware components). Thus, the software configures a specific processor or processors to configure a specific hardware component at one time instance and a different hardware component at a different time instance. Hardware components can provide information to other hardware components and receive information from them.Accordingly, the described hardware components may be considered to be coupled communicably. In cases where multiple hardware components exist simultaneously, communication may be achieved through the transmission of signals between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may be achieved, for example, through the storage and retrieval of information within memory structures accessible to multiple hardware components. For example, one hardware component may perform an operation and store the output of that operation in a memory device coupled communicably to it. Subsequently, additional hardware components may access the memory device to retrieve and process the stored output. Hardware components may also initiate communication with input or output devices and manipulate resources (e.g., collections of information). Various operations of the exemplary methods described herein may be performed at least partially by one or more processors configured temporarily (e.g., by software) or permanently to perform the relevant operations. Whether configured temporarily or permanently, such processors may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be implemented at least partially by a processor, and specific processors or processors are examples of hardware.For example, at least some of the operations of the method may be performed by one or more processors or processor implementation components. Furthermore, one or more processors may also operate to support the performance of the relevant operations in a "cloud computing" environment or as "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines containing processors), and these operations are accessible via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., APIs). The performance of specific operations may not only exist within a single machine but may also be distributed among processors deployed across multiple machines. In some exemplary embodiments, the processors or components implemented by the processors may be located in a single geographic location (e.g., a home environment, an office environment, or within a server farm). In other exemplary embodiments, the processors or components implemented by the processors may be distributed across multiple geographic locations.

[0189] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0190] "Computer-readable medium" refers to both machine storage media and transmission media. Accordingly, the terms include both storage devices / medias and carriers / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0191] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop, PDAs (portable digital assistants), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0192] "Short-term messages" refer to messages accessible for a time-limited duration. Short-term messages can be text, images, videos, etc. The access time for short-term messages can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, messages are transitory.

Claims

Claim 1 A method comprising: generating sets of source pose parameters by one or more hardware processors, wherein the sets of source pose parameters include the positions of facial expressions of a source actor and the head expressions of the source actor in frames of source media content; receiving at least one target image by one or more hardware processors, wherein the at least one target image includes the target face and the head expressions of a target entity; generating output media content by one or more hardware processors, wherein each frame of the output media content includes an image of the target face in at least one frame of the output media content, and the image of the target face is modified based on at least one of the sets of source pose parameters to mimic at least one of the facial expressions of the source actor and at least one of the head positions of the source actor; and inducing the display or playback of the online advertisement based on the duration of the online advertisement in a messaging client application on a client device. Claim 2 A method according to claim 1, wherein the online advertisement includes a video, and the video cannot be skipped or paused during the playback of the video. Claim 3 A method according to claim 1, wherein the online advertisement includes an image, and the image cannot be hidden during the duration of the online advertisement. Claim 4 The method of claim 1, wherein the step of generating the set of source pose parameters comprises the step of generating a first set of face features based on the frames of the source media content using a shared encoder network—the generating step generates a first set of image data with a lower dimensionality than the frames of the source media content—wherein the lower dimensionality includes a resolution lower than a specific resolution of the frames of the source media content. Claim 5 A method according to claim 4, further comprising the step of generating a second set of face features based on at least one target image using the shared encoder network, wherein the generating step generates a second set of image data with a lower dimensionality than the at least one target image. Claim 6 A method according to claim 5, wherein the step of generating the output media content based at least partially on the sets of source pose parameters comprises the step of generating a first set of output images or frames using a first decoder network associated with the at least one target image, wherein the first set of output images or frames includes modifications to the representations of the face and the head of the source actor based on the second set of face features from the at least one target image. Claim 7 A method according to claim 6, further comprising the step of generating a decoded version of the at least one target image using a first decoder network associated with the at least one target image, wherein the decoded version of the at least one target image is a second representation of the target face and the target head. Claim 8 A method according to claim 7, further comprising the step of generating a second set of output images or frames using a second decoder network associated with the frames of the source media content, wherein the second set of output images or frames comprises decoded versions of the representations of the face and head of the source actor based on the first set of face features. Claim 9 In claim 8, the shared encoder network comprises a first deep convolutional neural network, the first decoder network comprises a second deep convolutional neural network, the second decoder network comprises a third deep convolutional neural network, and the first deep convolutional neural network, the second deep convolutional neural network, and the third deep convolutional neural network are different neural networks. Claim 10 As a system, the system comprises: at least one processor; and memory containing instructions, wherein the instructions, when executed by the at least one processor, cause the at least one processor to generate sets of source pose parameters by one or more hardware processors, wherein the sets of source pose parameters include the positions of facial expressions of a source actor and the head expressions of the source actor in frames of source media content; wherein the at least one target image is received by the one or more hardware processors, wherein the at least one target image includes the target face and the head expressions of a target entity; wherein the output media content is generated by the one or more hardware processors at least partially based on the sets of source pose parameters, wherein each frame of the output media content includes an image of the target face in at least one frame of the output media content, and the image of the target face is modified based on at least one of the sets of source pose parameters to mimic at least one of the facial expressions of the source actor and at least one of the head positions of the source actor. A system that enables actions including actions that trigger the display or playback of an online advertisement based on the duration of the online advertisement in a messaging client application on a client device. Claim 11 In paragraph 10, the above online advertisement includes a video, and the system, said video cannot be skipped or paused during the playback of said video. Claim 12 In paragraph 10, the above online advertisement includes an image, and the image cannot be concealed during the duration of the above online advertisement, the system. Claim 13 In claim 10, generating the sets of source pose parameters comprises the operation of generating a first set of face features based on the frames of the source media content using a shared encoder network—the generating operation generates a first set of image data with a lower dimensionality than the frames of the source media content—wherein the lower dimensionality includes a resolution lower than a specific resolution of the frames of the source media content. Claim 14 A system according to claim 13, wherein the operations further comprise an operation to generate a second set of face features based on at least one target image using the shared encoder network—the generating operation generates a second set of image data with a lower dimensionality than the at least one target image. Claim 15 A system according to claim 14, wherein generating the output media content based at least partially on the sets of source pose parameters comprises using a first decoder network associated with the at least one target image to generate a first set of output images or frames—the first set of output images or frames comprising modifications to the representations of the face and the head of the source actor based on the second set of face features from the at least one target image. Claim 16 A system according to claim 15, wherein the operations further comprise an operation to generate a decoded version of the at least one target image using a first decoder network associated with the at least one target image, wherein the decoded version of the at least one target image is a second representation of the target face and the target head. Claim 17 A system according to claim 16, further comprising the operations of generating a second set of output images or frames using a second decoder network associated with frames of the source media content, wherein the second set of output images or frames comprises decoded versions of representations of the face and head of the source actor based on the first set of face features. Claim 18 A non-transient computer-readable storage medium comprising instructions, wherein the instructions, when executed by a computing device, cause the computing device to generate, by one or more hardware processors, sets of source pose parameters—the sets of source pose parameters include the positions of facial expressions of a source actor and the representations of the head of the source actor in frames of source media content—; receive at least one target image by the one or more hardware processors—the at least one target image includes the target face and representations of the target head of a target entity—; generate output media content by the one or more hardware processors at least partially based on the sets of source pose parameters—each frame of the output media content includes an image of the target face in at least one frame of the output media content, and the image of the target face is modified based on at least one of the sets of source pose parameters to mimic at least one of the facial expressions of the source actor and at least one of the positions of the head of the source actor—; A non-transient computer-readable storage medium that enables actions including actions that trigger the display or playback of an online advertisement based on the duration of the online advertisement in a messaging client application on a client device. Claim 19 In paragraph 18, the online advertisement comprises a video, and the video is a non-transient computer-readable storage medium that cannot be skipped or paused during the playback of the video. Claim 20 In paragraph 18, the online advertisement comprises an image, and the image is a non-transient computer-readable storage medium that cannot be hidden during the duration of the online advertisement.

Images