Text-guided guest string role generation

Abstract

A method of generating an image for use in a conversation occurring in a messaging application is disclosed. A dialog input text is received from a user of a portable device including a display. Model input text is generated from the dialog input text, the model input text being processed using a text-to-image model to generate an image based on the model input text. Coordinates of the face in the image are determined, and the face of the user or another person is added to the image at the location. The final image is displayed on the portable device, and user input is received to communicate the image to a remote recipient.

Description

[0001] Priority Claim

[0002] This application claims the benefit of priority to U.S. patent application serial number 17 / 950,945, filed on September 22, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to the field of messaging, including but not limited to text messaging. Background Art

[0004] Electronic text messages can be enhanced with relevant images such as emoticons, ideograms, and static or looping gifs to communicate more visually and emotionally. In some cases, an image of the user's face can be included in the image to provide personalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In the drawings, which are not necessarily drawn to scale, similar reference numerals may describe similar components in different views. To easily identify the discussion of any particular element or action, the most significant digit or digits in the reference numeral refer to the figure number in which the element is first introduced. Some non-limiting examples are shown in the figures of the accompanying drawings, in which:

[0006] Figure 1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network.

[0007] Figure 2 is a block diagram showing additional details regarding a messaging system according to some examples.

[0008] Figure 3 A guest character generation system 300 is shown according to some examples.

[0009] Figure 4A and Figure 4B are two examples of existing guest characters that can be used to generate ML models for use in Figure 3 used in the image generation system.

[0010] Figure 5 is a schematic diagram illustrating the generation of a machine learning model for generating cameo character stickers according to some examples.

[0011] Figure 6 is a flow chart 600 illustrating a method of generating guest character stickers according to some examples.

[0012] Figure 7is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein.

[0013] Figure 8 is a block diagram illustrating a software architecture in which examples may be implemented. DETAILED DESCRIPTION

[0014] Snap Inc.'s popular Snapchat app allows photos, videos, and text messages to be enhanced with so-called "stickers," which are graphical elements that can be applied to a photo or video like a label, or included in or over a messaging conversation. Stickers can be fun, including cartoon-like items (e.g., birthday cakes, champagne bottles, animals, etc.), representations of people (which may or may not be an avatar of the user or one of their friends), and decorative text.

[0015] In some examples, an image of the face of a user of a messaging system or portable device is included in a sticker to provide a so-called cameo. A cameo provides a personalized sticker experience for users to interact with their friends. A cameo can take the form of a sticker in which the protagonist performs some actions. The image of the face can be animated to provide expression changes, which can be predetermined, recorded, or live. In some examples, an image of the user's face is animated based on the emotion of the text entered by the user into the messaging application.

[0016] A machine learning framework is disclosed herein that can generate guest character images based on text queries provided by users of a messaging application. The framework includes two aspects. First, a text analysis system that identifies important phrases from the user's text query, and an image generation system that uses imported phrases obtained from the text analysis system for guest character image generation.

[0017] The image generation system has two components. First, one or more background images and associated metadata including facial position data are generated. Second, by using the facial position information stored in the metadata, the image generation system combines the image of the user's face and the background image generated from the first stage into a complete image. As a final step, the user's face can be animated based on the emotion of the user input phrase.

[0018] Also disclosed are methods and systems for generating a machine learning model for use in an image generation system.

[0019] In some examples, a computer-implemented method of generating an image including an existing representation of a person's face for use in a conversation occurring in a messaging application is provided. The method includes: receiving conversation input text from a user of a portable device including a display; generating model input text from the conversation input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; applying the existing representation of the person's face to the image based on the coordinates of the face in the image to generate an updated image; displaying the updated image on the display of the portable device; receiving user input to transmit the updated image in a message; and transmitting the image to a remote recipient in response to receiving the user input.

[0020] The generation of the model input text may include generating additional text using a creative captioning function, and extracting key phrases from the additional text. The method may also include processing the conversation input text using a safety filter to determine the appropriateness of the conversation input text for image generation.

[0021] The text-to-image model may have been generated from a large-scale image dataset and refined by an existing collection of images used in conversations occurring in a messaging application. The image-to-text module may be used to expand text associated with an image in the existing collection of images. The method may also include animating the face of a person in the updated image.

[0022] In some examples, a non-transitory computer-readable storage medium is provided that includes instructions that, when executed by a computer, cause the computer to perform operations for generating an image including an existing representation of a person's face for use in a conversation occurring in a messaging application, the operations including the method steps and limitations set forth above, including but not limited to: receiving conversation input text from a user of a portable device including a display; generating model input text from the conversation input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; based on the coordinates of the face in the image, applying the existing representation of the person's face to the image to generate an updated image; displaying the updated image on the display of the portable device; receiving user input to transmit the updated image in a message; and transmitting the image to a remote recipient in response to receiving the user input.

[0023] In some examples, a computing device is provided that includes a processor and a memory that stores instructions that, when executed by the processor, configure the device to perform operations for generating an image that includes an existing representation of a person's face for use in a conversation occurring in a messaging application, the operations including the method steps and limitations set forth above, including but not limited to: receiving conversation input text from a user of a portable device that includes a display; generating model input text from the conversation input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; applying the existing representation of the person's face to the image based on the coordinates of the face in the image to generate an updated image; displaying the updated image on the display of the portable device; receiving user input to transmit the updated image in a message; and transmitting the image to a remote recipient in response to receiving the user input.

[0024] Other technical features may be readily apparent to those skilled in the art from the accompanying drawings, descriptions and claims.

[0025] Networked computing environment

[0026] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of a user device 102, each of the multiple instances hosting multiple applications, including a messaging client 104 and other applications 106.

[0027] Each messaging client 104 is communicatively coupled to other instances of the messaging client 104 (e.g., hosted on respective other user devices 102) and a messaging server system 108 via a network 110 (e.g., the Internet). The messaging client 104 may also communicate with locally hosted applications 106 using an application program interface (API). The messaging system 100 also includes one or more user devices 102 communicatively coupled to the messaging server system 108 via the network 110.

[0028] As used herein, the terms messaging client 104 and messaging system 100 are considered to include instant messaging platforms, social media platforms, content sharing platforms, and other platforms where users forward images to other users or post images onto the platform.

[0029] The messaging clients 104 are able to communicate and exchange data with other messaging clients 104 and messaging server systems 108 via the network 110. The data exchanged between the messaging clients 104 and between the messaging clients 104 and messaging server systems 108 include functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0030] The messaging server system 108 provides server-side functionality to specific messaging clients 104 via the network 110. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client 104 or by the messaging server system 108, it may be a design choice whether certain functions are located within the messaging client 104 or within the messaging server system 108. For example, it may be technically preferred to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client 104 where the user device 102 has sufficient processing power.

[0031] The messaging server system 108 supports various services and operations provided to the messaging clients 104. Such operations include transmitting data to the messaging clients 104, receiving data from the messaging clients 104, and processing data generated by the messaging clients 104. As examples, the data may include message content, user device information, geographic location information, media enhancements and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is stimulated and controlled by functions available via the user interface (UI) of the messaging client 104.

[0032] Turning now specifically to the messaging server system 108, an application program interface (API) server 114 is coupled to and provides a programming interface to the application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 124 that stores data associated with messages processed by the application server 112. Similarly, a web server 126 is coupled to and provides a web-based interface to the application server 112. To this end, the web server 126 handles incoming network requests via the Hypertext Transfer Protocol (HTTP) and several other related protocols.

[0033] The application program interface (API) server 114 receives and transmits message data (e.g., commands and message payloads) between the user device 102 and the application server 112. Specifically, the application program interface (API) server 114 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client 104 to activate the functions of the application server 112. An application program interface (API) server 114 exposes various functions supported by application server 112, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via application server 112; sending media files (e.g., images or videos) from a messaging client 104 to messaging server 116 and for possible access by another messaging client 104; setting of media data collections (e.g., stories); retrieving a friend list of a user of user device 102; retrieving such collections; retrieving messages and content; adding and removing entities (e.g., friends) from an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to messaging client 104).

[0034] The application server 112 hosts several server applications and subsystems, including, for example, a messaging server 116, an image processing server 120, and a social network server 122. The messaging server 116 implements several message processing technologies and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Given the hardware requirements for other processor- and memory-intensive data processing, such processing can also be performed on the server side by the messaging server 116.

[0035] The application server 112 also includes an image processing server 120 that is dedicated to performing various image processing operations, typically on images or videos within the payload of messages sent from or received at the messaging server 116 .

[0036] The social network server 122 supports various social networking functions and services and makes them available to the messaging server 116. To this end, the social network server 122 maintains and accesses an entity graph within the database 124. Examples of functions and services supported by the social network server 122 include identifying other users of the messaging system 100 that have a relationship with a particular user or that the particular user is "following", and also includes identifying interests and other entities of a particular user.

[0037] The messaging client 104 may notify the user of the user device 102 or other users (e.g., "friends") associated with such a user of an activity occurring in one or more external resources, such as a streamlined version of a game or a third-party application. For example, the messaging client 104 may provide a participant in a conversation (e.g., a chat session) in the messaging client 104 with a notification related to the current or recent use of an external resource by one or more members of a user group. One or more users may be invited to join an active external resource or to start an external resource that was recently used but is currently inactive (in the friend group). The external resource may provide the participants of the conversation who each use the corresponding messaging client 104 with the ability to share items, conditions, states, or locations in the external resource with one or more members of the user group entering the chat session. The shared item may be an interactive chat card that members of the chat may interact with, for example, to start a corresponding external resource, view specific information within an external resource, or bring members of the chat to a specific location or state within an external resource. Within a given external resource, a response message may be sent to the user on the messaging client 104. The external resource may selectively include different media items in the response based on the current context of the external resource.

[0038] The messaging system 100 in the illustrated example includes an image quality assessment server 128 as part of the messaging server system 108, but the image quality assessment server 128 may alternatively be provided as part of a separate server system. The image quality assessment server 128 receives image quality assessment results and related data transmitted from the user device 102, and performs analysis and data mining on the accumulated image quality assessment results to generate image quality insights.

[0039] System Architecture

[0040] Figure 21 is a block diagram showing additional details about the messaging system 100 according to some examples. Specifically, the messaging system 100 is shown to include a messaging client 104 and an application server 112. The messaging system 100 contains several subsystems that are supported on the client side by the messaging client 104 and on the server side by the application server 112. These subsystems include, for example, a transient timer system 202, a collection management system 204, an enhancement system 208, a map system 210, and a game system 212.

[0041] The transient timer system 202 is responsible for implementing temporary or time-limited access to content by the messaging client 104 and the messaging server 116. The transient timer system 202 includes a plurality of timers that selectively enable access (e.g., for presentation and display) of messages and associated content via the messaging client 104 based on duration and display parameters associated with a message or a collection of messages (e.g., a story). Further details regarding the operation of the transient timer system 202 are provided below.

[0042] The collection management system 204 is responsible for managing collections or collections of media (e.g., collections of text, image video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event galleries" or "event stories." Such collections can be made available for a specified time period, such as the duration of an event related to the content. For example, content related to a concert can be available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing an icon that provides notification of the existence of a particular collection to the user interface of the messaging client 104.

[0043] In addition, the collection management system 204 includes a curation interface 206 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 206 enables an event organizer to curate a collection of content related to a particular event (e.g., to delete inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some examples, compensation may be paid to users for including user-generated content in a collection. In such a case, the collection management system 204 operates to automatically pay such users for use of their content.

[0044] The enhancement system 208 provides various functions that enable users to enhance (e.g., annotate or otherwise modify or edit) media content associated with a message. For example, the enhancement system 208 provides functions related to generating and publishing media overlays (media overlays) for messages processed by the messaging system 100. The enhancement system 208 is operable to provide media overlays or enhancements (e.g., image filters) to the messaging client 104 based on the geographic location of the user device 102. In another example, the enhancement system 208 is operable to provide media overlays to the messaging client 104 based on other information such as social network information of the user of the user device 102. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the user device 102. For example, media overlays can include text or images that can be superimposed on the top of photos taken by the user device 102. In another example, the media overlay includes a location identifier (e.g., Venice Beach) overlay, the name of a live event, or a business name (e.g., Beach Cafe) overlay. In another example, the augmentation system 208 uses the geographic location of the user device 102 to identify a media overlay that includes the name of a business at the geographic location of the user device 102. The media overlay may include other logos associated with the business. The media overlay may be stored in the database 124 and accessed through the database server 118.

[0045] In some examples, the enhancement system 208 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify situations in which a particular media overlay should be provided to other users. The enhancement system 208 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0046] The mapping system 210 provides various geo-location functions and supports the presentation of map-based media content and messages by the messaging client 104. For example, the mapping system 210 enables the display of user icons or avatars on a map to indicate the current or past locations of the user's "friends" within the context of the map, as well as media content (e.g., a collection of messages including photos and videos) generated by such friends. For example, a message posted by a user to the messaging system 100 from a particular geographic location can be displayed to a particular user's "friends" on the map interface of the messaging client 104 within the context of that particular location on the map. The user can also share his or her location and status information with other users of the messaging system 100 (e.g., using an appropriate status avatar) via the messaging client 104, where the location and status information is similarly displayed to the selected user within the context of the mapping interface of the messaging client 104.

[0047] The gaming system 212 provides various gaming functions within the context of the messaging client 104. The messaging client 104 provides a gaming interface that provides a list of available games that can be launched by a user within the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by sending an invitation to such other users from the messaging client 104. The messaging client 104 also supports both voice messaging and text messaging within the context of gaming (e.g., chatting), provides leaderboards for gaming, and also supports providing in-game rewards (e.g., gaming coins and items).

[0048] Figure 3 A guest character generation system 300 is shown according to some examples. The guest character generation system 300 includes a text processing module 304 that processes input text 302 to prepare it for use by an image generation system 306. The processed input text 302 is then provided as input to the image generation system 306, which operates on the processed input text 302 to generate a background 308 and associated metadata 310. The background 308, metadata 310, and an image of a user's face 314 are provided as input to a guest character generation module 312, which generates a guest character 316 therefrom.

[0049] In some examples, input text 302 is user input in a conversation that has begun or is occurring in a messaging session in a messaging client 104 running on user device 102. In other examples, input text 302 may be entered into a search bar for guest characters, emoticons, photos, or videos.

[0050] In some examples, the input text 302 is processed and run through a safety filter to determine whether the text includes inappropriate topics, such as violence, hate, or sexually suggestive language. If the input text is appropriate, the text processing module 304 uses a creative description text function to expand the input text 302. The creative description text function is a function based on natural language processing (NLP) that generates creative description text based on short queries. For example, a seed query like "love" can be expanded to "a boy holding a heart-shaped balloon looking at the girl he loves." Therefore, the creative description text function provides a more interesting text input, which is still based on the original input text 302.

[0051] The expanded text is then changed by a text simplification function to make the expanded text consistent with a text distribution known to be well suited for a machine learning (ML) image generation model. The text simplification function accepts the expanded text input and extracts key phrases. The extracted key phrases are then ranked, and the top one or two key phrases by ranking are cleaned. Text cleaning is the process by which the original text is prepared for use by the ML model so that it can be used more effectively by the model. Text cleaning can, for example, include removing capitalization, punctuation, and stop words.

[0052] Finally, the text processing module 304 can add specific text to the output of the text simplification function to refine the output of the ML image generation model. The added text can relate to the context in which the input text is used, such as "person" or "man" or "two women". The context can also be determined by NLP methods and the identity or characteristics of the person or intended recipient of the input text. For example, if two women are chatting in a messaging application and the input text "Let's go get some pizza together" is received at one device, "two women" can be added, but if a man enters "I'm craving some pizza!" in a group conversation, the text "a man" can be added. For example, the added text can also relate to the expected style of the image to be generated, such as "photo-realistic", which will ensure that the generated image contains as little text as possible.

[0053] The processed input text 302 is then passed to an image generation system 306, which generates a background 308 for the guest character, and metadata 310 defining at least the coordinates of the face in the background 308. In some examples, the image generation system 306 is a latent diffusion model (LDM), referred to below as Figure 5 The potential diffusion model is described in more detail.

[0054] The background 308, metadata 310, and the image of the user's face 314 are then provided to a guest character generation module 312, which adds the image of the user's face 314 to the background based on the metadata 310. As a final step, the user's face in a guest character 316 may be animated based on the emotion of the user's input phrase.

[0055] Figure 4A and Figure 4B 402 and 412 are two examples of existing guest characters, which can be used for the generation of ML models for use in the image generation system 306. Each of the guest characters 402 and 412 includes a first person 404 and a second person 406. Figure 4A The characters 404 and 406 include a generic male face 408 used as a male face in the entire sticker set, so that the characteristics of the face do not affect the training of the ML model.

[0056] Figure 4B Figures 404 and 406 in the figure include transparent faces 414. Transparent faces 414 each include an outline or contour line (not shown in the figure). Using transparent faces with outlines in the training and refinement of the ML model enables similar facial outlines to be generated in the output from the ML model, from which facial coordinates for use in adding one or more faces can be determined. As with the use of generic faces, the use of transparent faces 414 also prevents the characteristics of specific faces from affecting the training of the ML model.

[0057] Characters 404 and 406 are holding heart-shaped pizzas 410. Associated with guest characters 402 and 412 are related tags describing the content of the guest characters.

[0058] Figure 5 is a schematic diagram illustrating the generation of a machine learning model for generating guest character stickers according to some examples. The machine learning model is a text-to-image machine learning model that converts input text into an image. In some examples, the text-to-image machine learning model is a latent diffusion model (LDM).

[0059] Since it is expected that the guest characters generated using the techniques disclosed herein are somewhat similar in style and appearance to the guest characters used in the current guest character system, the initial LDM that has been trained using a large-scale image database is refined using an existing guest character set that is updated as follows. Guest character 402 (or guest character 412) is an example of a guest character considered for refinement of the initial LDM, however it will be understood that many different types and styles of guest characters may be provided, including photo-realistic, artistic, abstract, line drawings, etc. Each guest character has an associated annotated label, including relevant descriptive text. For example, guest characters 402, 412 may have annotated labels such as "couple" and "love" and "pizza", etc. However, some of the existing labels associated with the existing guest character set may be noisy because the labels may be irrelevant or only peripherally relevant.

[0060] Thus, in operation 504, labels of the current guest character dataset 502, including existing guest characters, are initially filtered for relevance of the labels to corresponding guest character images using an image-text relevance model. In some examples, this filtering is performed using contrastive language image pre-training (CLIP). CLIP is an image recognition and vision benchmarking system that is trained by using CLIP to predict which caption from randomly selected captions is the correct caption for a given image. CLIP learns to associate a wide variety of objects with their names and the words that describe them, which allows it to recognize objects in images outside of its training set.

[0061] The CLIP model is used to match the current label of each guest character with the guest character image. If the matching score provided by the CLIP model for a particular label is greater than 0.3, the label is retained, otherwise it is discarded. For guest characters 402, 412, the labels filtered using the example of clip are "heart-shaped pizza", "pizza", "pepperoni", "salami", and "hungry".

[0062] To supplement the filtered guest characters and label sets, additional labels or captions are generated using an image-to-text module in operation 506 to provide greater flexibility in the training of the LDM. This may be useful, for example, in situations where a user provides a more literal description of an expected or requested guest character. In some examples, a model called Bootstrapped Language Image Pre-training (BLIP) is used to generate the additional annotation set. The BLIP model describes guest character 402 as "two people holding a heart-shaped pizza."

[0063] The filtered labels from operation 504 and the additional descriptors generated in operation 506 are then combined with the guest character image data to create an updated guest character dataset 508, which is then provided as input for refining the LDM, as will be described in more detail below.

[0064] In operation 512, LDM is initially trained on a large-scale image dataset 510, such as the LAION-400M dataset, which includes 400 million English language image and text pairs generated by a crawling network. Therefore, the initial large-scale dataset covers a variety of topics and image styles. Initial training on this dataset provides generalization capabilities that enable the proposed system to outperform traditional text search and retrieval systems in terms of identifying and suggesting existing guest roles for user selection.

[0065] The LDM is then fine-tuned in two steps using the updated guest character dataset 508. The LDM includes an image generation module, such as VQGAN, which is a vector quantized image generation module. The model only requires images from the updated guest character dataset 508 for fine-tuning. Therefore, the guest character images are used in operation 514 to fine-tune the image generation module of the LDM.

[0066] The LDM also includes a text-to-image module, which in some examples can be a vector diffusion model. In operation 516, the text-to-image vector diffusion model, which has been initially trained on the image, text pairs in the large-scale image dataset 510, is fine-tuned using the image and text pairs in the updated guest character dataset 508. The resulting final LDM 518 is then used in the guest character generation method discussed below.

[0067] Figure 6 600 is a flowchart illustrating a method of generating guest character stickers according to some examples. For illustrative purposes, the operations of flowchart 600 are described herein as occurring serially or linearly. However, multiple operations of flowchart 600 may occur in parallel. In addition, the operations of flowchart 600 do not need to be performed in the order shown and / or one or more boxes in flowchart 600 do not need to be performed and / or may be replaced by other operations. In addition, although the operations in flowchart 600 are described with reference to a process running on user device 102, the associated functionality may alternatively be provided as part of a distributed processing system including one or more servers.

[0068] The flowchart begins at operation 602, where input text is received by the user device 102. In some examples, the input text is user input in a conversation that has begun or is occurring in a message session in a messaging client 104 running on the user device 102. In other examples, the text can be entered into a search input bar for a guest character, emoticon, photo, or video. In response to receiving the input text, in operation 604, the messaging client 104 (or other application 106) on the user device 102 processes the input text using a safety filter to determine whether the text includes inappropriate topics, such as violence, hate, or sexually suggestive language. In operation 606, if the messaging client 104 determines that the text is inappropriate, the flowchart 600 ends at operation 608, or alternatively, returns to operation 602 to receive additional, potentially suitable text.

[0069] In operation 606, if the messaging client 104 determines that the text is suitable, then in operation 610, the messaging client 104 processes the input text using a creative caption function to expand the input text. The creative caption function is a module based on natural language processing (NLP) that generates creative captions based on short queries. For example, a seed query like "love" can be expanded to "a boy holding a heart-shaped balloon looking at the girl he loves."

[0070] Some other examples may be expanding "hi" to "cat says hi to bird", expanding "angry" to "angry sumo wrestler hits ground", and expanding "please" to "bear holding please sign". Thus, the creative caption feature provides more interesting text input - which is still based on the original text input - for use in text-to-image cameo generation as disclosed herein.

[0071] Then, in operation 612, the expanded text from the creative description text function is changed by a text simplification function running on the messaging client 104 to make the expanded text consistent with a text distribution known to be well suited for the LDM image generation model. The text simplification function accepts the expanded text input and extracts key phrases. The extracted key phrases are then ranked by the messaging client 104, and the top one or two key phrases by ranking are cleaned. Text cleaning is the process by which the original text is prepared for use by the ML model so that it can be used more effectively by the model. Text cleaning can, for example, include removing capitalization, punctuation, and stop words.

[0072] Then, in operation 614, specific text is added to the output of the text simplification function to refine the output of the LDM. For example, the added text can relate to the expected style of the image to be generated, such as "photo-realistic", which will also ensure that the generated image contains as little text as possible. As mentioned above, the added text can also relate to the context in which the input text is used, such as "person" or "man" or "two women". The context can also be determined by NLP methods and the identity or characteristics of the person or intended recipient of the input text.

[0073] Then, in operation 616, the final text received, generated, and modified in operations 602 to 614 is provided as input to the final LDM 518 that generates the guest character background.

[0074] Then, in operation 618, the generated guest character background is analyzed using known facial recognition and characterization techniques to determine the coordinates of the face, including the size, position, and orientation of the face therein in some examples, and the location of key facial features in some other examples.

[0075] In operation 620, a face or faces (such as the user's face, the face of another conversation participant, the user's "friend") are added to the guest character background based on the coordinates determined in operation 618. In some examples, the face or faces may be animated using known facial animation techniques in operation 622. In some examples, the face may be animated so that it speaks the input text or text derived from the input text, such as "Congratulations" for the input text of congratulations, or an expression (sad, happy, surprised, etc.) derived from the input text.

[0076] Then, in operation 624, the guest character is displayed on the display of the user device, and in operation 626, the guest character is available for user input. User input may include selecting a guest character to be used (transmitted) in the conversation that is taking place, forwarding, saving, canceling, etc. In operation 628, the messaging client 104 performs the selected action, such as, for example, transmitting the generated guest character to the user device 102 of the remote recipient via the messaging system 100. Then, in operation 630, the method ends, or alternatively, returns to operation 602 to receive additional input text.

[0077] In some examples, these operations from operation 610 to operation 620 can be run more than once, with or without changes in the input text, such as the order of words or the case of additional text that was previously discarded in the text ranking. Therefore, multiple generated guest roles can be provided for the user to view and select. Further, in addition to the generated (one or more) guest roles, the multiple guest roles can include guest roles from an existing set, which is retrieved by the messaging application 846 using text search and retrieval methods to operate on the original user input text or the expanded text generated by the text processing module 304.

[0078] Machine Architecture

[0079] Figure 7 700 within which instructions 710 (e.g., software, programs, applications, applet, app, or other executable code) may be executed for causing the machine 700 to perform any one or more of the methods discussed herein. For example, the instructions 710 may cause the machine 700 to perform any one or more of the methods described herein. The instructions 710 convert a general purpose, unprogrammed machine 700 into a specific machine 700 that is programmed to perform the functions described and shown in the manner described. The machine 700 may operate as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 700 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 700 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 710 specifying actions to be taken by the machine 700. In addition, although only a single machine 700 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 710 to perform any one or more of the methods discussed herein. For example, the machine 700 may include any of the user device 102 or a plurality of server devices forming part of the messaging server system 108. In some examples, the machine 700 may also include both a client system and a server system, wherein certain operations of a particular method or algorithm are performed on the server side, and certain operations of the particular method or algorithm are performed on the client side.

[0080] The machine 700 may include a processor 704, a memory 706, and an input / output (I / O) component 702 that may be configured to communicate with each other via a bus 740. In an example, the processor 704 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 708 and a processor 712 that execute instructions 710. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 7 Multiple processors 704 are shown, but machine 700 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0081] The memory 706 includes a main memory 714, a static memory 716, and a storage unit 718, which are all accessible by the processor 704 via the bus 740. The main memory 706, the static memory 716, and the storage unit 718 store instructions 710 that implement any one or more of the methods or functions described herein. The instructions 710 may also reside, completely or partially, within the main memory 714, within the static memory 716, within the machine-readable medium 720 within the storage unit 718, within at least one of the processors 704 (e.g., within a cache memory of a processor), or within any suitable combination thereof during execution thereof by the machine 700.

[0082] The I / O components 702 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O components 702 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine will be less likely to include such a touch input device. It will be appreciated that the I / O components 702 may include Figure 7720 , and the like. In various examples, the I / O components 702 may include a user output component 726 and a user input component 728. The user output component 726 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an acoustic component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The user input component 728 may include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input component), a point-based input component (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other pointing instrument), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides location and force of a touch or touch gesture), an audio input component (e.g., a microphone), etc.

[0083] In another example, the I / O component 702 may include: a biometric component 730, a motion component 732, an environment component 734, or a position component 736, as well as various other components. For example, the biometric component 730 includes components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 732 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope).

[0084] Environmental components 734 include, for example, one or more cameras (with still image / photo and video capabilities), lighting sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., an infrared sensor that detects nearby objects), gas sensors (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0085] With respect to cameras, user device 102 may have a camera system including, for example, a front-facing camera on a front surface of user device 102 and a rear-facing camera on a rear surface of user device 102. The front-facing camera may, for example, be used to capture still images and videos of a user of user device 102 (e.g., a "selfie"), which may then be enhanced using the enhancement data (e.g., filters) described above. The rear-facing camera may, for example, be used to capture still images and videos in a more traditional camera mode, where the images are similarly enhanced using the enhancement data. In addition to the front-facing camera and the rear-facing camera, user device 102 may also include a 360° camera for capturing 360° photos and videos.

[0086] Additionally, the camera system of user device 102 may include dual rear cameras (e.g., a main camera and a depth sensing camera), or even a triple, quad, or penta rear camera configuration on the front and back sides of user device 102. For example, these multiple camera systems may include a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0087] The location component 736 includes a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure, from which the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0088] A variety of technologies may be used to achieve communication. The I / O component 702 also includes a communication component 738 that is operable to couple the machine 700 to the network 722 or device 724 via a corresponding coupling or connection. For example, the communication component 738 may include a network interface component or another suitable device that interfaces with the network 722. In other examples, the communication component 738 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Parts (e.g. Low energy consumption), Device 724 may be another machine or any of a variety of peripherals (eg, a peripheral coupled via USB).

[0089] In addition, the communication component 738 can detect the identifier or include a component that can be operated to detect the identifier. For example, the communication component 738 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting a one-dimensional barcode such as a universal product code (UPC) barcode, a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a Data Matrix, a Dataglyph, a MaxiCode, PDF417, an Ultra Code, a UCC RSS-2D barcode, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying a tagged audio signal). In addition, various information can be obtained via the communication component 738, such as positioning via an Internet Protocol (IP) geographic location, obtaining a location via a wireless communication interface, obtaining a location information via a wireless communication interface, obtaining a location information via a wireless communication interface, and obtaining a location information via a wireless communication interface. Positioning is obtained by signal triangulation, positioning is obtained via detection of NFC beacon signals that can indicate a specific position, etc.

[0090] Various memories (e.g., main memory 714, static memory 716, and memory of processor 704) and storage unit 718 may store one or more sets of instructions and data structures (e.g., software) implemented or used by any one or more of the methods or functions described herein. These instructions (e.g., instructions 710) when executed by processor 704 cause various operations to implement the disclosed examples.

[0091] The instructions 710 may be transmitted or received over the network 722 via a network interface device (e.g., a network interface component included in the communication component 738) using a transmission medium and using any of several well-known transmission protocols (e.g., the Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 710 may be transmitted or received via a coupling (e.g., a peer-to-peer coupling) with the device 724 using a transmission medium.

[0092] Software Architecture

[0093] Figure 8 800 is a block diagram illustrating a software architecture 804 that can be installed on any one or more of the devices described herein. The software architecture 804 is supported by hardware, such as a machine 802 including a processor 820, a memory 826, and an I / O component 838. In this example, the software architecture 804 can be conceptualized as a stack of layers, each of which provides specific functionality. The software architecture 804 includes layers such as an operating system 812, a library 810, a framework 808, and an application 806. In operation, the application 806 invokes an API call 850 through the software stack and receives a message 852 in response to the API call 850.

[0094] The operating system 812 manages hardware resources and provides public services. The operating system 812 includes, for example, a kernel 814, services 816, and drivers 822. The kernel 814 acts as an abstraction layer between the hardware and other software layers. For example, the kernel 814 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 816 can provide other public services to other software layers. Drivers 822 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 822 may include display drivers, camera drivers, or Low-power drivers, Flash drivers, Serial communication drivers (e.g., USB drivers), Drivers, audio drivers, power management drivers, etc.

[0095] The library 810 provides a common low-level infrastructure used by the application 806. The library 810 may include a system library 818 (e.g., a C standard library) that provides functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 810 may include an API library 824, such as a media library (e.g., a library for supporting the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for presenting in two dimensions (2D) and three dimensions (3D) in graphics content on a display), a database library (e.g., SQLite providing various relational database functions), a web library (e.g., WebKit providing web browsing functions), etc. The library 810 may also include various other libraries 828 to provide many other APIs to the application 806.

[0096] The framework 808 provides a common high-level infrastructure used by the applications 806. For example, the framework 808 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 808 can provide a wide range of other APIs that can be used by the applications 806, some of which can be specific to a particular operating system or platform.

[0097] In an example, applications 806 may include a home application 836, a contacts application 830, a browser application 832, a book reader application 834, a positioning application 842, a media application 844, a messaging application 846, a game application 848, and a variety of other applications such as third-party applications 840. Applications 806 are programs that execute functions defined in the program. Various programming languages ​​can be used to create one or more of the applications 806 constructed in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C language or assembly language). In a specific example, third-party applications 840 (e.g., those created by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or another mobile operating system. In this example, the third-party application 840 can activate the API call 850 provided by the operating system 812 to facilitate the functions described in this article.

[0098] Glossary

[0099] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media that facilitates communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0100] "User Device" refers to any machine that interfaces with a communications network to obtain resources from one or more server systems or other user devices. A user device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communications device that a user may use to access a network.

[0101] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of a network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0102] "Component" refers to a logic, device or physical entity with boundaries defined by function or subroutine calls, branch points, APIs or other technical definitions that provide partitioning or modularization for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be part of a program designed for encapsulated functional hardware units used with other components, and specific functions that generally perform related functions. Components can constitute software components (e.g., codes implemented on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., independent computer systems, client computer systems or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., applications or application parts) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically or in any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It will be recognized that it can be determined for cost and time considerations whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system to implement the hardware component. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an example where a hardware component is temporarily configured (e.g., programmed), each of the hardware components does not need to be configured or instantiated at any time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a special-purpose processor, the general-purpose processor can be configured into different special-purpose processors (e.g., including different hardware components) at different times. Software accordingly configures one or more specific processors to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times. Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components in the hardware components. In examples where multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure to which multiple hardware components have access rights and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to it in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process the stored output. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., a collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described in this article can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operation of the method can be performed by one or more processors or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or as a "software as a service" (SaaS) operation. For example, at least some of the operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations in the operation can be distributed between processors, not only resident in a single machine, but deployed across multiple machines. In some examples, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other examples, a processor or a part implemented by a processor can be distributed across multiple geographical locations.

[0103] "Computer-readable storage media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium", "computer-readable medium" and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0104] "Ephemeral messages" are messages that are accessible for a limited duration of time. Ephemeral messages can be text, images, videos, etc. The access time for ephemeral messages can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is transient.

[0105] "Machine storage media" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data in a non-transitory manner. Thus, the term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and device storage media include: non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage media", "device storage media", and "computer storage media" mean the same thing and may be used interchangeably in this disclosure. The terms "machine storage media", "computer storage media", and "device storage media" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are encompassed by the term "signal media".

[0106] “Non-transitory computer-readable storage medium” refers to a tangible medium capable of storing, encoding, or carrying instructions for execution by a machine.

[0107] "Signal medium" refers to any intangible medium that can store, encode or carry instructions executed by a machine and includes digital or analog communication signals or other intangible media that facilitates the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

Claims

1. A computer-implemented method of generating an image comprising an existing representation of a person's face for use in a conversation occurring in a messaging application, the method comprising: receiving conversation input text from a user of a portable device including a display; generating a model input text from the dialogue input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; applying an existing representation of the person's face to the image based on coordinates of the face in the image to generate an updated image; displaying the updated image on a display of the portable device; receiving user input transmitting the updated image in a message; as well as In response to receiving the user input, the image is transmitted to a remote recipient.

2. The method according to claim 1, wherein: The generation of the model input text includes generating additional text using a creative description text function.

3. The method according to claim 2, wherein: The generation of the model input text also includes: Key phrases are extracted from the additional text.

4. The method according to claim 1, further comprising: The conversation input text is processed using a safety filter to determine the suitability of the conversation input text for image generation.

5. The method according to claim 1, wherein: The text-to-image model is generated from a large-scale image dataset and refined by an existing collection of images used in conversations taking place in messaging applications.

6. The method according to claim 5, wherein: An image-to-text module is used to expand text associated with an image in the existing image collection.

7. The method according to claim 5, further comprising: A face of a person in the updated image is animated.

8. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations for generating an image comprising an existing representation of a person's face for use in a conversation occurring in a messaging application, the operations comprising: receiving conversation input text from a user of a portable device including a display; generating a model input text from the dialogue input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; applying an existing representation of the person's face to the image based on coordinates of the face in the image to generate an updated image; displaying the updated image on a display of the portable device; receiving user input transmitting the updated image in a message; as well as In response to receiving the user input, the image is transmitted to a remote recipient.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The generation of the model input text includes generating additional text using a creative description text function.

10. The non-transitory computer-readable storage medium of claim 9, wherein: The generation of the model input text also includes: Key phrases are extracted from the additional text.

11. The non-transitory computer-readable storage medium of claim 8, wherein: The operations also include: The conversation input text is processed using a safety filter to determine the suitability of the conversation input text for image generation.

12. The non-transitory computer-readable storage medium of claim 8, wherein: The text-to-image model is generated from a large-scale image dataset and refined by an existing collection of images used in conversations taking place in messaging applications.

13. The non-transitory computer readable storage medium of claim 12, wherein: An image-to-text module is used to expand text associated with an image in the existing image collection.

14. The non-transitory computer readable storage medium of claim 13, wherein: Texts associated with images in the existing image collection are filtered for relevance of the texts to images in the existing image collection using an image-text relevance model.

15. A computing device comprising: processor; as well as a memory storing instructions that, when executed by the processor, configure the apparatus to perform operations for generating an image including an existing representation of a person's face for use in a conversation occurring in a messaging application, the operations comprising: receiving conversation input text from a user of a portable device including a display; generating a model input text from the dialogue input text; providing the model input text to a text-to-image model to generate an image based on the model input text; determining coordinates of a face in the image; applying an existing representation of the person's face to the image based on coordinates of the face in the image to generate an updated image; displaying the updated image on a display of the portable device; receiving user input transmitting the updated image in a message; and In response to receiving the user input, the image is transmitted to a remote recipient.

16. The computing device of claim 15, wherein: The generation of the model input text includes generating additional text using a creative description text function.

17. The computing device of claim 16, wherein: The generation of the model input text also includes: Key phrases are extracted from the additional text.

18. The computing device of claim 15, wherein: The text-to-image model is generated from a large-scale image dataset and refined by an existing collection of images used in conversations taking place in messaging applications.

19. The computing device of claim 18, wherein: An image-to-text module is used to expand text associated with an image in the existing image collection.

20. The computing device of claim 18, wherein: Texts associated with images in the existing image collection are filtered for relevance of the texts to images in the existing image collection using an image-text relevance model.

Images