Generating true values for generative AI applications

Abstract

A first neural network is trained to generate truth values using a small set of example images that account for a target truth value output image, which example images may be whole body images of a person of AR style. The first neural network is used for generating a true value output image according to the random input image. An example method of a first neural network includes determining a pose in an input image, changing values of pixels within a region of the input image, inputting the pose, the changed region of the input image, and a textual cue describing the input image into the neural network to generate an output image. The method further includes determining a loss between the output image and the input image and updating a weight of the neural network based on the loss. A second neural network is then trained using the generated truth values. And generating the application using the second neural network.

Description

CLAIM OF PRIORITY

[0001] This application claims the benefit of priority of Greek Patent Application Serial No. 20230100508, filed June 23, 2023, and U.S. Patent Application Serial No. 18 / 226,929, filed July 27, 2023, which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] Examples of the present disclosure generally relate to generating ground truth for generative artificial intelligence (AI) or neural networks and utilizing the ground truth to train generative neural networks. More specifically, but not by way of limitation, examples of the present disclosure relate to training a first neural network to generate ground truth for training a second neural network, where the training can be based on knowledge distillation utilizing a text-to-image diffusion model of a reference image. Examples include training a second neural network with the generated ground truth and generating an application with the trained second neural network. BACKGROUND

[0003] Neural networks are becoming ubiquitous for performing image processing tasks within AI. And users are increasingly desiring more and more functionality from neural networks. But it is often difficult to obtain ground truth to train neural networks to provide the desired functionality. BRIEF DESCRIPTION OF DRAWINGS

[0004] In the drawings, which are not necessarily drawn to scale, like numerals can describe similar components in different views. To easily identify the discussion of any particular element or act, the most significant digit or digits in a figure reference number can correspond to the figure number in which that element is first introduced. Some non-limiting examples are illustrated in the drawings, in which:

[0005] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure can be deployed, in accordance with some examples.

[0006] Figure 2 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality, in accordance with some examples.

[0007] Figure 3 is a diagrammatic representation of a data structure maintained in a database, in accordance with some examples.

[0008] Figure 4 is a diagrammatic representation of a message, in accordance with some examples.

[0009] Figure 5 shows a system having a head wearable device, in accordance with some examples.

[0010] Figure 6 is a diagrammatic representation of a machine in the form of a computer system, within which a set of instructions can be executed to cause the machine to perform any one or more of the methodologies discussed herein, according to some examples.

[0011] Figure 7 is a block diagram illustrating a software architecture, upon which an example can be implemented.

[0012] Figure 8 is a perspective view of a head wearable device in the form of eyeglasses, according to some examples.

[0013] Figure 9 An image generator training component is shown, according to some examples.

[0014] Figure 10 An example of a system to generate ground truth for a generative neural network is shown, according to some examples.

[0015] Figure 11 A training set generator component is shown, according to some examples.

[0016] Figure 12 A system to generate ground truth for a generative neural network is shown, according to some examples.

[0017] Figure 13 A production training component is shown, according to some implementations.

[0018] Figure 14 Training of a production generative neural network is shown, according to some examples.

[0019] Figure 15 An example generative neural network application is shown, according to some examples.

[0020] Figure 16 A method for generating an image generator neural network application is shown, according to some implementations.

[0021] Figure 17 A generative image generator neural network application component is shown, according to some implementations.

[0022] Figure 18 A method to generate ground truth for a generative neural network is shown, according to some implementations. DETAILED DESCRIPTION

[0023] The following description includes illustrative examples of systems, methods, techniques, instruction sequences, and computer program products that embody the contents of this disclosure. In this description, numerous specific details are set forth for illustrative purposes to provide an understanding of various examples of the subject matter of the invention. However, it will be apparent to those skilled in the art that examples of the subject matter of the invention can be practiced without these specific details. Generally, well-known examples of instructions, protocols, structures, and techniques are not necessarily shown in detail.

[0024] Typically, truth values ​​are generated or collected, for example... Figure 11 The generated inputs and outputs shown are difficult for 1114. For example, refer to... Figure 15 Generative neural network application 1508 generates an output image 1512 of a person wearing a pink dress based on an input image 1510 of a person wearing a regular dress. For training the generative neural network, the ground values ​​need to include images of the person wearing a regular dress and images of the same person wearing a pink dress. However, it may be difficult to find examples of the same person specifically wearing two different types of dresses in the same pose as ground values ​​to train the generative neural network application 1508. Figure 14 1402, a production-type generative neural network.

[0025] The technical challenge is how to generate the generated input-output pair 1114 or the ground truth to train the productive generative neural network 1402. In some examples, this technical challenge is addressed by training an image generator neural network component 1008. A sample set of reference image 904 is used to train the image generator neural network component 1008. The trained image generator neural network component 1008 is then used to generate an image from the input image 1205. Figure 12 The generated image 1214. The input image 1205 can be a random image, such as an image of a person wearing ordinary clothing that can be obtained from an image database or via the Internet. The reference images 904 can be a relatively small number of images, such as ten to one hundred images, which is less than the typical ground truth, which can include thousands or even millions of images. As an example, a set of ten reference images 904 of a person wearing a pink dress 1006 can be used to generate the generated input and output pairs 1114 (which include thousands of pairs) to train a productive generative neural network 1402.

[0026] The image generator training component 902 trains the image generator neural network component 1008 using the pose 1002 of the person depicted within the reference image 1004, the reference image 1004 with the inpainting region 1012 (which is the portion of the reference image 1004 that has changed), and the textual cue 1016 describing the reference image 1004 (e.g., "full body pose of person wearing pink dress"). The image generator training component 902 trains the image generator neural network component 1008 to fill in the inpainting region 1012 with a pose that matches the pose 1002 of the reference image 1004.

[0027] After training, the training set generator component 1102 utilizes the image generator neural network component 1008 to generate generated input and output pairs 1114 that the production training component 1302 can use to train the production generative neural network 1402.

[0028] Referring to Figure 12 Another technical challenge is how to preserve some features of the input image 1205, such as the long hair 1206, while generating a generated image 1214 with features such as the "pink dress" 1020. This technical challenge is addressed by the mask generator component 1106 determining the portions of the input image 1205 that are not to be transformed. For example, the hands, arms, and head can be excluded from the inpainting mask 1208 that is input to the image generator neural network component 1008. The inpainting mask 1208 excludes those areas of the input image 1205 that are to be preserved. Additionally, the mask generator component 1106 can determine that the hair 1206 is too long to exclude the area of the hair 1206 in the inpainting mask 1208, so the hair 1206 is extracted from the input image 1205 and then fused with the generated image 1214, e.g., the hair 1209. Furthermore, the pose 1210 of the input image 1205 is preserved in the generated image 1214 by the initial training and by extracting the pose 1210 from the input image 1205 and using it as input to the image generator neural network component 1008.

[0029] The example enables the production generative neural network 1402 to be trained using a relatively small number of reference images 904. Furthermore, the example enables the pose 1002 and features, such as hair, to be preserved in the ground truth or generated input and output pairs 1114 by identifying the features and extracting them from the input image 1205 and then fusing them back into the generated image 1214.

[0030] Networked Computing Environment

[0031] Figure 1is a block diagram illustrating an example interaction system 100 for facilitating interactions (e.g., exchanging text messages, making text, audio, and video calls, or playing games) on a network. The interaction system 100 includes a plurality of client systems, each of which hosts a plurality of applications including an interaction client 104 and other applications 106. Each interaction client 104 is communicatively coupled to other instances of the interaction client 104 (e.g., hosted on respective other user systems), an interaction server system 110, and third-party servers 112 via one or more communication networks including a network 108 (e.g., the Internet). The interaction client 104 can also communicate with locally-hosted applications 106 using an application program interface (API).

[0032] Each user system 102 can include a plurality of user devices, such as a computing device 114, a head-wearable device 116, and a computer client device 118, which are communicatively connected to exchange data and messages.

[0033] The interaction client 104 interacts with other interaction clients 104 and with the interaction server system 110 via the network 108. Data exchanged between interaction clients 104 (e.g., interactions 120) and between interaction clients 104 and the interaction server system 110 includes functions (e.g., commands for activating functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0034] The interaction server system 110 provides server-side functionality to the interaction clients 104 via the network 108. While certain functions of the interaction system 100 are described herein as being performed by the interaction client 104 or by the interaction server system 110, it can be a design choice as to whether a certain function is located within the interaction client 104 or within the interaction server system 110. For example, it can be technically preferable to initially deploy a particular technology or function within the interaction server system 110, but to later migrate that technology or function to the interaction client 104 of a user system 102 that has sufficient processing power.

[0035] The interaction server system 110 supports various services and operations provided to the interaction clients 104. Such operations include sending data to the interaction clients 104, receiving data from the interaction clients 104, and processing data generated by the interaction clients 104. The data can include message content, client device information, geolocation information, media augmentations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the interaction system 100 is activated and controlled through functions available via a user interface (UI) of the interaction client 104.

[0036] Turning now specifically to the interaction server system 110, an application program interface (API) server 122 is coupled to and provides a programmatic interface to the interaction server 124 that enables the functionality of the interaction server 124 to be accessed by the interaction client 104, other applications 106, and third-party servers 112. The interaction server 124 is communicatively coupled to a database server 126 that facilitates access to a database 128 in which is stored data associated with interactions processed by the interaction server 124. Similarly, a web server 130 is coupled to the interaction server 124 and provides a web-based interface to the interaction server 124. In this regard, the web server 130 processes incoming network requests through the Hypertext Transfer Protocol (HTTP) and several related protocols.

[0037] The application program interface (API) server 122 receives and transmits interaction data (e.g., commands and message payloads) between the interaction server 124 and client systems (and, for example, the interaction client 104 and other applications 106) and third-party servers 112. Specifically, the application program interface (API) server 122 provides a set of interfaces (e.g., routines and protocols) that the interaction client 104 and other applications 106 can call or query to activate the functionality of the interaction server 124. The application program interface (API) server 122 exposes various functions supported by the interaction server 124, including account registration; login functionality; sending interaction data from a particular interaction client 104 to another interaction client 104 via the interaction server 124; transferring media files (e.g., images or videos) from the interaction client 104 to the interaction server 124; setting collections of media data (e.g., stories); retrieving a user's friends list for a user system 102; retrieving messages and content; adding and deleting entities (e.g., friends) to an entity graph (e.g., a social graph); locating friends within a social graph; and opening application events (e.g., related to the interaction client 104). The interaction server 124 hosts a number of systems and subsystems, described below with respect to Figure 2

[0038] Linked applications

[0039] ​Returning to the interaction client 104, features and functionality of external resources (e.g., linked applications 106 or applets) are available to the user via the interface of the interaction client 104. In this context, "external" refers to the fact that the applications 106 or applets are external to the interaction client 104. External resources are typically provided by third parties, but can also be provided by the creator or provider of the interaction client 104. The interaction client 104 receives a selection by the user of an option to launch or access features of such external resources. The external resources can be applications 106 installed on the user system 102 (e.g., "native applications") or small-scale versions of applications (e.g., "applets") hosted on the user system 102 or remote from the user system 102 (e.g., on a third-party server 112). The small-scale versions of applications include a subset of the features and functionality of the applications (e.g., full-scale, native versions of the applications) and are implemented using markup language documents. In some examples, the small-scale versions of applications (e.g., "applets") are web-based markup language versions of the applications and are embedded in the interaction client 104. In addition to using markup language documents (e.g., files), the applets can incorporate script files (e.g., files or files) and style sheets (e.g., files).

[0040] In response to receiving a selection by the user of an option to launch or access features of an external resource, the interaction client 104 determines whether the selected external resource is a web-based external resource or a native installed application 106. In some cases, an application 106 that is natively installed on the user system 102 can be launched independently of and separately from the interaction client 104, e.g., by selecting an icon on the home screen of the user system 102 that corresponds to the application 106. Such small-scale versions of applications can be launched or accessed via the interaction client 104, and in some examples, no part of the small-scale application can be accessed outside of the interaction client 104 or limited parts of the small-scale application can be accessed outside of the interaction client 104. The small-scale application can be launched by the interaction client 104 receiving, e.g., a markup language document associated with the small-scale application from a third-party server 112 and processing such document.

[0041] In response to determining that the external resource is a locally installed application 106, the interaction client 104 instructs the user system 102 to launch the external resource by executing locally stored code corresponding to the external resource. In response to determining that the external resource is a web-based resource, the interaction client 104 communicates with the third-party server 112, for example, to obtain a markup language document corresponding to the selected external resource. The interaction client 104 then processes the obtained markup language document to present the web-based external resource within the user interface of the interaction client 104.

[0042] The interaction client 104 can notify users of the user system 102 or other users (e.g., “friends”) associated with such users of activity occurring in one or more external resources. For example, the interaction client 104 can provide participants in a conversation (e.g., a chat session) in the interaction client 104 with notifications related to current or recent use of external resources by one or more members of a group of users. One or more users can be invited to join an active external resource, or launch an external resource that was recently used but is not currently active (of a group of friends). An external resource can provide participants in a conversation (each participant using a respective interaction client 104) with the ability to share items, conditions, states, or locations in the external resource in a chat session with one or more members of a group of users. The shared item can be an interactive chat card that chat members can interact with, for example, to launch the corresponding external resource, view particular information within the external resource, or bring the chat member to a particular location or state within the external resource. Within a given external resource, responsive messages can be sent to users on the interaction client 104. Based on the current context of the external resource, the external resource can selectively include different media items in the response.

[0043] The interaction client 104 can present a list of available external resources (e.g., applications 106 or widgets) to a user to launch or access a given external resource. The list can be presented in a contextually relevant menu. For example, icons representing different applications in the applications 106 (or widgets) can vary based on how the user launched the menu (e.g., from a conversation interface or from a non-conversation interface).

[0044] System Architecture

[0045] Figure 2 is a block diagram illustrating additional details regarding the interaction system 100 according to some examples. In particular, the interaction system 100 is shown to include the interaction client 104 and the interaction server 124. The interaction system 100 includes a number of subsystems that are supported on the client side by the interaction client 104 and on the server side by the interaction server 124. Example subsystems are discussed below.

[0046] The image processing system 202 provides various functionality that enables a user to capture and enhance (e.g., annotate or otherwise modify or edit) media content associated with a message.

[0047] The camera system 204 includes control software (e.g., in a camera application) that interacts with and controls hardware camera hardware of the user system 102 (e.g., directly or via an operating system) to modify and enhance live images captured and displayed via the interactive client 104.

[0048] The augmentation system 206 provides functionality related to the generation and publication of augmentations (e.g., media overlays) for images captured in real-time by the camera of the user system 102 or images retrieved from the memory of the user system 102. For example, the augmentation system 206 is operable to select, present, and display media overlays (e.g., image filters or image lenses) for the interactive client 104 for use in augmenting live images received via the camera system 204 or stored images retrieved from the memory 502 of the user system 102. These augmentations are selected by the augmentation system 206 based on some input and data, for example: a geographic location of the user system 102; and social network information of a user of the user system 102.

[0049] Augmentations can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to a media content item (e.g., a photo or video) at the user system 102 for communication in a message or to video content, such as a video content stream or feed sent from the interactive client 104. Thus, the image processing system 202 can interact with and support various subsystems of the communication system 208, such as the messaging system 210 and the video communication system 212.

[0050] Media overlays can include text or image data that can be overlaid on a photo taken by the user system 102 or a video stream made by the user system 102. In some examples, a media overlay can be a location overlay (e.g., Venice Beach), a name of a live event, or a business name overlay (e.g., Beachside Café). In further examples, the image processing system 202 uses the geolocation of the user system 102 to identify a media overlay that includes the name of a business at the geolocation of the user system 102. The media overlay can include other indicia associated with the business. The media overlays can be stored in the database 128 and accessed through the database server 126.

[0051] The image processing system 202 provides a user-based publishing platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. The user can also specify circumstances under which a particular media overlay should be provided to other users. The image processing system 202 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

[0052] The augmentation creation system 214 supports an augmented reality developer platform and includes an application for content creators (e.g., artists and developers) to create and publish augmentations (e.g., augmented reality experiences) for the interactive client 104. The augmentation creation system 214 provides a library of built-in features and tools for content creators, including, for example, custom shaders, tracking techniques, and templates.

[0053] In some examples, the augmentation creation system 214 provides a business-based publishing platform that enables businesses to select particular augmentations associated with a geolocation via a bidding process. For example, the augmentation creation system 214 associates a media overlay of the highest bidding business with a corresponding geolocation for a predefined amount of time.

[0054] The communication system 208 is responsible for enabling and handling various forms of communication and interaction within the interaction system 100, and includes a messaging system 210, an audio communication system 216, and a video communication system 212. The messaging system 210 is responsible for implementing temporary or time-limited access to content by the interaction clients 104. The messaging system 210 includes a plurality of timers (e.g., in a transient timer system 218) that selectively enable access (e.g., for presentation and display) of messages and associated content via the interaction clients 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 218 are provided below. The audio communication system 216 enables and supports audio communication (e.g., real-time audio chat) between a plurality of interaction clients 104. Similarly, the video communication system 212 enables and supports video communication (e.g., real-time video chat) between a plurality of interaction clients 104.

[0055] The user management system 220 is operationally responsible for managing user data and profiles, and includes a social network platform 222 that maintains information regarding relationships between users of the interaction system 100.

[0056] The collection management system 224 is operationally responsible for managing collections or sets of media (e.g., collections of text, image, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be made available for a specified period of time (e.g., the duration of an event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 224 can also be responsible for publishing icons to the user interfaces of the interaction clients 104 that provide notifications of particular collections. The collection management system 224 includes curation functionality that enables a curator to manage and curate particular collections of content. For example, a curation interface enables an event organizer to curate a collection of content related to a particular event (e.g., to delete inappropriate content or redundant messages). Additionally, the collection management system 224 employs machine vision (or image recognition technology) and content rules to automatically curate collections of content. In certain examples, users can be paid to include user-generated content into a collection. In such cases, the collection management system 224 operates to automatically make payments to such users for use of their content.

[0057] The map system 226 provides various geolocation functionality and supports the presentation of map-based media content and messages by the interactive clients 104. For example, the map system 226 enables the display of user icons or avatars (e.g., stored in the profile data 302) on a map to indicate the current or past locations of a user's "friends" and media content (e.g., collections of messages including photos and videos) generated by those friends within the context of the map. For example, a message posted by a user from a particular geographic location to the interactive system 100 can be displayed to a particular user's "friends" within the context of the map at that particular location on a map interface of the interactive client 104. A user can also share his or her location and status information with other users of the interactive system 100 via the interactive client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users within the context of a map interface of the interactive client 104.

[0058] The game system 228 provides various game functionality within the context of the interactive clients 104. The interactive clients 104 provide a game interface that provides a list of available games that can be launched by a user within the context of the interactive client 104 and played with other users of the interactive system 100. The interactive system 100 also enables a particular user to invite other users to participate in playing a particular game by issuing invitations from the interactive client 104 to the other users. The interactive clients 104 also support voice, video, and text messaging (e.g., chat) within the context of playing games, provide leaderboards for games, and also support the provision of in-game rewards (e.g., game currency and items).

[0059] The external resource system 230 provides an interface for the interactive clients 104 to communicate with remote servers (e.g., third-party servers 112) to launch or access external resources (i.e., applications or applets). Each third-party server 112 hosts an application or a small-scale version of an application (e.g., a game application, a utility application, a payment application, or a ride-sharing application) based on a markup language (e.g., HTML5), for example. The interactive clients 104 can launch a web-based resource (e.g., an application) by accessing an HTML5 file from a third-party server 112 associated with the web-based resource. The applications hosted by the third-party servers 112 are programmed in JavaScript with the use of a software development kit (SDK) provided by the interactive server 124. The SDK includes an application programming interface (API) with functions that can be called or activated by the web-based applications. The interactive server 124 hosts a JavaScript library that provides given external resource access to particular user data of the interactive clients 104. HTML5 is an example of a technology for programming games, but applications and resources programmed based on other technologies can be used.

[0060] To integrate the functionality of the SDK into a web-based resource, the SDK is downloaded from the interaction server 124 by the third-party server 112, or otherwise received by the third-party server 112. Once downloaded or received, the SDK is included as part of the application code of the web-based external resource. The code of the web-based resource can then call or activate certain functionality of the SDK to integrate the features of the interaction client 104 into the web-based resource.

[0061] The SDK stored on the interaction server system 110 effectively provides a bridge between the external resource (e.g., application 106 or widget) and the interaction client 104. This gives the user a seamless experience of communicating with other users on the interaction client 104 while also preserving the look and feel of the interaction client 104. To bridge the communication between the external resource and the interaction client 104, the SDK facilitates communication between the third-party server 112 and the interaction client 104. The WebView JavaScript Bridge running on the user system 102 establishes two one-way communication channels between the external resource and the interaction client 104. Messages are sent asynchronously between the external resource and the interaction client 104 via these communication channels. Each SDK function activation is sent as a message and a callback. Each SDK function is implemented by constructing a unique callback identifier and sending a message with the callback identifier.

[0062] By using the SDK, not all information from the interaction client 104 is shared with the third-party server 112. The SDK restricts which information is shared based on the needs of the external resource. Each third-party server 112 provides the interaction server 124 with an HTML5 file corresponding to the web-based external resource. The interaction server 124 can add a visual representation (e.g., a box diagram or other graphic) of the web-based external resource in the interaction client 104. Once the user selects the visual representation or indicates through the GUI of the interaction client 104 that the interaction client 104 is to access the features of the web-based external resource, the interaction client 104 obtains the HTML5 file and instantiates the resource for accessing the features of the web-based external resource.

[0063] The interaction client 104 presents a graphical user interface for the external resource (e.g., a landing page or title screen). During, before, or after presenting the landing page or title screen, the interaction client 104 determines whether the launched external resource has been previously authorized to access user data of the interaction client 104. In response to determining that the launched external resource has been previously authorized to access user data of the interaction client 104, the interaction client 104 presents another graphical user interface of the external resource that includes functionality and features of the external resource. In response to determining that the launched external resource has not been previously authorized to access user data of the interaction client 104, after displaying the landing page or title screen of the external resource for a display threshold period of time (e.g., 3 seconds), the interaction client 104 slides up a menu (e.g., animates the menu to emerge from the bottom of the screen to the middle or other portion of the screen) for authorizing the external resource to access user data. The menu identifies types of user data that the external resource will be authorized to use. In response to receiving a selection of an accept option by the user, the interaction client 104 adds the external resource to a list of authorized external resources and allows the external resource to access user data from the interaction client 104. The interaction client 104 authorizes the external resource to access user data under the OAuth 2 framework.

[0064] The interaction client 104 controls types of user data shared with external resources based on types of the external resources that are authorized. For example, external resources including full-scale applications (e.g., the application 106) are provided access to a first type of user data (e.g., two-dimensional avatars of users with or without different avatar characteristics). As another example, external resources including small-scale versions of applications (e.g., web-based versions of the application) are provided access to a second type of user data (e.g., payment information, two-dimensional avatars of users, three-dimensional avatars of users, and avatars with various avatar characteristics). Avatar characteristics include different ways of customizing the appearance and feel of an avatar (e.g., different poses, facial features, clothing, etc.).

[0065] The advertising system 232 is operable to enable third parties to purchase advertisements for presentation to end users via the interaction client 104 and also handles the delivery and presentation of these advertisements.

[0066] The image-to-image conversion system 234 supports or is equivalent to the image generator training component 902, the training set generator component 1102, the production training component 1302, and the generative neural network application 1508. The image-to-image conversion system 234 provides storage services and retrieval services for the image generator training component 902, the training set generator component 1102, the production training component 1302, and / or the generative neural network application 1508. The image-to-image conversion system 234 performs the functions for the image generator training component 902, the training set generator component 1102, the production training component 1302, and / or the generative neural network application 1508. The image-to-image conversion system 234 provides the reference images 904.

[0067] In addition, the image-to-image conversion system 234 can provide the generative neural network application 1508 to the user computing devices 114 and / or the head-wearable devices 116. The image-to-image conversion system 234 interacts with the social network platform 222 by providing a service such as a list of available generative neural network applications 1508 and delivering it to the computing devices 114.

[0068] Data Architecture

[0069] Figure 3 FIG. 3 is a schematic diagram illustrating a data structure 300 that can be stored in the database 304 of the interaction server system 110, in accordance with certain examples. While the contents of the database 304 are illustrated as including a plurality of tables, it will be appreciated that data can be stored in other types of data structures (e.g., object-oriented databases).

[0070] The database 304 includes message data stored within a message table 306. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. Reference is made below to FIG. 4 for a more detailed description of the message data. Figure 3 Further details regarding information that can be included in a message and included within the message data stored in the message table 306 are described.

[0071] The entity table 308 stores entity data and is linked (e.g., referentially) to the entity graph 310 and the profile data 302. Entities for which records are maintained within the entity table 308 can include individuals, corporate entities, organizations, objects, locations, events, and the like. Regardless of the entity type, any entity for which the interaction server system 110 stores data can be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0072] The entity graph 310 stores information about relationships and associations between entities. Such relationships can be social, professional (e.g., working at a common company or organization), interest-based, or activity-based, merely as examples. Certain relationships between entities can be one-way, such as a personal user's subscription to digital content of a business or publication user (e.g., a newspaper or other digital media channel or brand). Other relationships can be two-way, such as a "friend" relationship between individual users of the interaction system 100.

[0073] Certain permissions and relationships can be attached to each relationship, and can also be attached to each direction of a relationship. For example, a two-way relationship (e.g., a friend relationship between personal users) can include authorization for publication of digital content items between the personal users, but can impose certain restrictions or filtering on such publication of digital content items (e.g., based on content properties, location data, or time-of-day data). Similarly, a subscription relationship between a personal user and a business user can impose different degrees of restriction on publication of digital content from the business user to the personal user, and can significantly restrict or prevent publication of digital content from the personal user to the business user. As an example of an entity, a particular user can record certain restrictions in the record for that entity within the entity table 308 (e.g., through privacy settings). Such privacy settings can apply to all types of relationships in the context of the interaction system 100, or can selectively apply to certain types of relationships.

[0074] The profile data 302 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 302 can be selectively used and presented to other users of the interaction system 100. In the case of an entity being a person, the profile data 302 includes, for example, a username, a phone number, an address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or a collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations within the content of messages communicated via the interaction system 100 and on map interfaces displayed by the interaction client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a graphical representation of a status or activity that the user can select to communicate at a particular time.

[0075] In the case of an entity being a group, the profile data 302 for the group can similarly include one or more avatar representations associated with the group, in addition to the group name, members, and various settings (e.g., notifications) for the relevant group.

[0076] The database 304 also stores augmentation data, such as overlays or filters, in an augmentation table 312. The augmentation data is associated with videos (data for which is stored in a video table 314) and images (data for which is stored in an image table 316) and is applied to the videos and images.

[0077] In some examples, a filter is an overlay that is displayed as an overlay on an image or video during presentation to a recipient user. The filter can be of various types, including a user-selected filter from a set of filters presented to a sending user by the interactive client 104 when the sending user is composing a message. Other types of filters include geolocation filters (also referred to as geo-filters), which can be presented to the sending user based on geolocation. For example, a geolocation filter specific to a nearby or special location can be presented by the interactive client 104 within a user interface based on geolocation information determined by a global positioning system (GPS) unit of the user system 102.

[0078] Another type of filter is a data filter, which can be selectively presented to the sending user by the interactive client 104 based on other input or information collected by the user system 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the user system 102, or a current time.

[0079] Other augmentation data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to an application "lens" or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0080] A story table 318 stores data regarding collections of messages and associated image, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in the entity table 308). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the interactive client 104 can include a user-selectable icon to enable a sending user to add particular content to his or her personal story.

[0081] Collections can also constitute "live stories," which are collections of content from multiple users created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via a user interface of the interaction client 104. A live story can be identified to a user by the interaction client 104 based on the user's location. The end result is a "live story" told from a group perspective.

[0082] Another type of content collection is referred to as a "location story," which enables users whose user systems 102 are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some examples, contributions to a location story can employ secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0083] As mentioned above, the video table 314 stores video data that, in some examples, is associated with messages whose records are maintained within the message table 306. Similarly, the image table 316 stores image data that is associated with messages whose message data is stored in the entity table 308. The entity table 308 can associate various augmentations from the augmentation table 312 with various images and videos stored in the image table 316 and the video table 314.

[0084] The database 304 also includes a neural network table 318. Referring to Figure 11 and Figure 14 The neural network table 318 includes generated input and output pairs 1114, production-type generative neural networks 1402, etc.

[0085] Data Communication Architecture

[0086] Figure 4 is a schematic diagram showing the structure of a message 400 generated by an interaction client 104 for transmission to another interaction client 104 via an interaction server 124, in accordance with some examples. The content of a particular message 400 is used to populate the message table 306 stored within the database 304 that is accessible by the interaction server 124. Similarly, the content of the message 400 is stored in memory as "in-transit" or "in-flight" data for the user system 102 or the interaction server 124. The message 400 is shown to include the following example components: The message identifier 402 is a unique identifier that identifies the message 400. Message text payload 404 is text to be generated by a user via a user interface of user system 102 and included in message 400. Message image payload 406 is image data captured by a camera component of user system 102 or retrieved from a memory component of user system 102 and included in message 400. Image data for sent or received messages 400 can be stored in image table 316. Message video payload 408: video data captured by a camera component or retrieved from a memory component of user system 102 and included in message 400. Video data for sent or received messages 400 can be stored in image table 316. Message audio payload 410: audio data captured by a microphone or retrieved from a memory component of user system 102 and included in message 400. Message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) representing augmentations to be applied to message image payload 406, message video payload 408, or message audio payload 410 of message 400. Augmentation data for sent or received messages 400 can be stored in augmentation table 312. Message duration parameter 414 is a parameter value indicating, in seconds, an amount of time for content (e.g., message image payload 406, message video payload 408, message audio payload 410) of a message to be presented to or made accessible to a user via interaction client 104. Message geolocation parameter 416: geolocation data (e.g., latitude and longitude coordinates) associated with a content payload of a message. Multiple message geolocation parameter 416 values can be included in a payload, each of which is associated with a content item included in the content (e.g., a particular image within message image payload 406 or a particular video in message video payload 408). Message story identifier 418: an identifier value identifying one or more content collections (e.g.,“stories” identified in story table 318) associated with a particular content item in message image payload 406 of message 400. For example, multiple images within message image payload 406 can each be associated with multiple content collections using identifier values. Message tags 420: Each message 400 can be tagged with a plurality of tags, each of which indicates a subject of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value can be included within the message tags 420 that indicates the relevant animal. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition. Message sender identifier 422: An identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of the user system 102 on which the message 400 was generated and from which the message 400 was sent. Message recipient identifier 424: An identifier (e.g., messaging system identifier, email address, or device identifier) that indicates a user of the user system 102 to which the message 400 is addressed.

[0087] The content (e.g., values) of the various components of the message 400 can be pointers to locations within tables where the content data values are stored. For example, the image value in the message image payload 406 can be a pointer to a location within the image table 316 (or an address of a location within the image table 316). Similarly, the value within the message video payload 408 can point to data stored within the image table 316, the value stored within the message augmentation data 412 can point to data stored in the augmentation table 312, the value stored within the message story identifier 418 can point to data stored in the story table 318, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored within the entity table 308.

[0088] System with head wearable device

[0089] Figure 5 A system 500 including a head wearable device 116 with a selector input device is shown in accordance with some examples. Figure 5 is a high-level functional block diagram of an example head wearable device 116 communicatively coupled to computing devices 114 and various server systems 504 (e.g., interaction server system 110) via various networks 108.

[0090] The head wearable device 116 includes one or more cameras, each of which can be, for example, a visible light camera 506, an infrared emitter 508, and an infrared camera 510.

[0091] The computing device 114 connects with the head-wearable device 116 using both the low-power wireless connection 512 and the high-speed wireless connection 514. According to some examples, the computing device 114 also connects to the server system 504 and the network 516. The computing device 114 can be a portable computing device, such as a smartphone, a tablet computer, a laptop computer, or another type of computing device 114, such as a desktop computer, or another type of computing device 114.

[0092] The head-wearable device 116 also includes two image displays of an optical assembly of image displays 518. The two image displays of an optical assembly of image displays 518 include one image display associated with a left lateral side of the head-wearable device 116 and one image display associated with a right lateral side of the head-wearable device 116. The head-wearable device 116 also includes an image display driver 520, an image processor 522, low-power circuitry 524, and high-speed circuitry 526. The image displays of an optical assembly of image displays 518 are used to present images and video to a user of the head-wearable device 116, including images that can include a graphical user interface.

[0093] The image display driver 520 commands and controls the image displays of an optical assembly of image displays 518. The image display driver 520 can deliver image data directly to the image displays of an optical assembly of image displays 518 for presentation, or can convert the image data into a signal or data format suitable for delivery to the image display device. For example, the image data can be video data formatted according to a compression format such as H.264 (MPEG-4 Part 10), HEVC, Theora, Dirac, RealVideo RV40, VP8, VP9, or the like, while static image data can be formatted according to a compression format such as Portable Network Group (PNG), Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), or Exchangeable image file format (EXIF), or the like.

[0094] The head-wearable device 116 includes a frame and a stem (or temple) extending from a lateral side of the frame. The head-wearable device 116 also includes a user input device 528 (e.g., a touch sensor or button), including an input surface on the head-wearable device 116. The user input device 528 (e.g., a touch sensor or button) is used to receive input selections from a user for manipulating a graphical user interface of a presented image.

[0095] Figure 5The components shown for the head-worn wearable device 116 are located on one or more circuit boards (e.g., PCBs or flexible PCBs) in the frame or temples. Alternatively or additionally, the depicted components may be located in the modules, frame, hinges, or nose bridge of the head-worn wearable device 116. The left and right visible light camera devices 506 may include digital camera elements, such as complementary metal-oxide-semiconductor (CMOS) image sensors, charge-coupled devices, camera lenses, or any other corresponding visible light or light-capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0096] The head-mounted wearable device 116 includes a memory 502 that stores instructions for performing a subset or all of the functions described herein. The memory 502 may also include a storage device.

[0097] like Figure 5 As shown, the high-speed circuit system 526 includes a high-speed processor 530, a memory 502, and a high-speed wireless circuit system 532. In some examples, an image display driver 520 is coupled to the high-speed circuit system 526 and operated by the high-speed processor 530 to drive the left and right image displays of the image display 518 of the optical components. The high-speed processor 530 can be any processor capable of managing the high-speed communication and operation of any general-purpose computing system required by the head-worn device 116. The high-speed processor 530 includes the processing resources required to manage high-speed data transmission over the high-speed wireless connection 514 to a wireless local area network (WLAN) using the high-speed wireless circuit system 532. In some examples, the high-speed processor 530 executes the operating system of the head-worn device 116 (e.g., a LINUX operating system) or other such operating system, and the operating system is stored in the memory 502 for execution. Among other duties, the high-speed processor 530, which executes the software architecture of the head-worn device 116, manages data transmission with the high-speed wireless circuit system 532. In some examples, the high-speed wireless circuit system 532 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as WiFi. In some examples, the high-speed wireless circuit system 532 may implement other high-speed communication standards.

[0098] The low-power wireless circuitry system 534 and high-speed wireless circuitry system 532 of the head-mounted wearable device 116 may include a short-range transceiver (Bluetooth™) and a wireless wide-area network transceiver, a wireless local area network transceiver, or a wide-area network transceiver (e.g., cellular or WiFi). The computing device 114—including transceivers communicating via low-power wireless connection 512 and high-speed wireless connection 514—can be implemented using details of the architecture of the head-mounted wearable device 116, as can other components of the network 516.

[0099] Memory 502 includes any storage device capable of storing various data and applications, including camera data generated by the left and right visible light cameras 506, the infrared camera 510, and the image processor 522, as well as images generated for display on an image display 518 in the optical components via an image display driver 520. While memory 502 is shown as integrated with high-speed circuitry 526, in some examples, memory 502 may be a separate, independent component of the head-mounted wearable device 116. In some such examples, electrical wiring may provide a connection from the image processor 522 or the low-power processor 536 to memory 502 via a chip including a high-speed processor 530. In some examples, the high-speed processor 530 may manage addressing of memory 502 such that the low-power processor 536 will activate the high-speed processor 530 whenever a read or write operation involving memory 502 is required.

[0100] like Figure 5 As shown, the low-power processor 536 or high-speed processor 530 of the head-mounted wearable device 116 may be coupled to a camera device (visible light camera 506, infrared emitter 508 or infrared camera 510), an image display driver 520, a user input device 528 (e.g., a touch sensor or button), and a memory 502.

[0101] The head-mounted wearable device 116 is connected to a host computer. For example, the head-mounted wearable device 116 is paired with the computing device 114 via a high-speed wireless connection 514 or connected to the server system 504 via a network 516. The server system 504 may be one or more computing devices as part of a service or network computing system, for example, it includes a processor, memory, and network communication interfaces to communicate with the computing device 114 and the head-mounted wearable device 116 via the network 516.

[0102] The computing device 114 includes a processor and a network communication interface coupled to the processor. The network communication interface allows for communication over the network 516, the low-power wireless connection 512, or the high-speed wireless connection 514. The computing device 114 can also store at least a portion of the instructions for generating binaural audio content in a memory of the computing device 114 to implement the functionality described herein.

[0103] The output components of the head-wearable device 116 include visual components such as displays, e.g., liquid crystal displays (LCD), plasma display panels (PDP), light emitting diode (LED) displays, projectors, or waveguides. The image displays of the optical assembly are driven by an image display driver 520. The output components of the head-wearable device 116 also include acoustic components (e.g., speakers), haptic components (e.g., a vibrating motor), other signal generators, etc. The input components (e.g., user input devices 528) of the head-wearable device 116, the computing device 114, and the server system 504 can include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., physical buttons, touchscreens that provide location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0104] The head-wearable device 116 can also include additional peripheral device elements. Such peripheral device elements can include biometric sensors, additional sensors, or display elements integrated with the head-wearable device 116. For example, the peripheral device elements can include any I / O components including output components, motion components, positioning components, or any other such elements described herein.

[0105] For example, biometric components include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like. Motion components include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth. Positioning components include location sensor components (e.g., a Global Positioning System (GPS) receiver component), Wi-Fi or Bluetooth™ transceivers for generating positioning system coordinates, altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like. Such positioning system coordinates can also be received from the computing device 114 via the low-power wireless circuitry 534 or the high-speed wireless circuitry 532 over the low-power wireless connection 512 and the high-speed wireless connection 514.

[0106] Machine architecture

[0107] Figure 6is a diagrammatic representation of a machine 600 within which instructions 602 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 600 to perform any one or more of the methodologies discussed herein can be executed. For example, the instructions 602 can cause the machine 600 to execute any one or more of the methods described herein. The instructions 602 transform the general, non-programmed machine 600 into a particular machine 600 programmed to carry out the described and illustrated functions in the manner described. The machine 600 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 600 can operate in the capacity of a server machine or a client machine in server-client network environments, or it can act as a peer machine in peer-to-peer (or distributed) network environments. The machine 600 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 602, sequentially or otherwise, that specify actions to be taken by machine 600. Further, while a single machine 600 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 602 to perform any one or more of the methodologies discussed herein. For example, the machine 600 can include any of the user system 102 or multiple server devices forming part of the interaction server system 110. In some examples, the machine 600 can further include both a client system and a server system, where certain operations of a particular method or algorithm are performed on the server side and where certain operations of the particular method or algorithm are performed on the client side.

[0108] The machine 600 can include processors 604, memory 606, and input / output (I / O) components 608, which can be configured to communicate with each other via a bus 610. In an example, the processor(s) 604 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 612 and a processor 614 that execute the instructions 602. The term “processor” is intended to include a multi-core processor that can include two or more independent processors (sometimes referred to as “cores”) that can execute instructions contemporaneously. Although FIG. 6 shows the processor(s) 604 as a single entity, alternative systems can include plural processors 604. In some examples, the processor(s) 604 can be supplemented by, or incorporated within, special purpose logic circuitry. Hence, references to the processor(s) 604 shall be taken to include a single processor, multiple processors, or a single processor in combination with special purpose logic circuitry, as appropriate.Figure 6 Multiple processors 604 are shown, but a single processor with single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with single cores, multiple processors with multiple cores, or any combination thereof are also possible.

[0109] Memory 606 includes main memory 616, static memory 618, and storage unit 620, each of which can be accessed via bus 610 by processors 604. Main memory 616, static memory 618, and storage unit 620 store instructions 602 implementing any one or more of the methods or functions described herein. Instructions 602 can also reside, completely or any portion of them, within machine-readable medium 622 within main memory 616, within static memory 618, within storage unit 620, within at least one of processors 604 (e.g., within cache memory of the processors), or any suitable combination thereof, during execution thereof by machine 600.

[0110] I / O components 608 can include various components for receiving input, providing output, producing output, transmitting information, exchanging information, capturing measurements, and so on. The specific I / O components 608 included in a particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that I / O components 608 can include many other components that are not shown in FIG. 6. In various examples, I / O components 608 can include user output components 624 and user input components 626. User output components 624 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. User input components 626 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like. Figure 6

[0111] ​In further examples, I / O components 608 can include biometric components 628, motion components 630, environmental components 632, or positioning components 634, among a myriad of other components. For example, biometric components 628 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like.

[0112] Environmental components 632 include, for example, one or more cameras (with still and / or video capabilities), illumination sensors (e.g., photometers), temperature sensors (e.g., one or more thermometers that detect ambient temperature), humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., one or more microphones that detect background noise), proximity sensors (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a physical environment.

[0113] With respect to cameras, user system 102 can have a camera system that includes, for example, a front-facing camera on a front surface of user system 102 and a rear-facing camera on a rear surface of user system 102. The front-facing camera can be used, for example, to capture still and video images of a user of user system 102 (e.g., a “selfie”), which can then be enhanced with the above-described enhancement data (e.g., filters). The rear-facing camera can be used, for example, to capture still and video images in a more conventional camera mode, where the images are similarly enhanced with enhancement data. In addition to front- and rear-facing cameras, user system 102 can include a 360° camera for capturing 360° photos and videos.

[0114] Further, the camera system of user system 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even triple, quad, or penta rear-facing camera configurations on the front and rear sides of user system 102. These multi-camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0115] The positioning component 634 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0116] Communication can be implemented using a wide variety of technologies. The I / O component 608 also includes a communication component 636 operable to couple the machine 600, via the respective coupling or connection, to a network 638 or devices 640. For example, the communication component 636 can include a network interface component or another suitable device to interface with the network 638. In further examples, the communication component 636 can include wired ® communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® ® Low Energy components, Wi-Fi® ® components, and other communication components to provide communication via other modalities. The devices 640 can be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).

[0117] Moreover, the communication component 636 can detect identifiers or include components operable to detect identifiers. For example, the communication component 636 can include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify acoustic signals indicating a specific

[0118] Various memories (e.g., the main memory 616, the static memory 618, and the memory of the processor 604) and the storage unit 620 can store one or more sets of instructions and data structures (e.g., software) embodying or used by any one or more of the methodologies or functions described herein. These instructions (e.g., instructions 602), when executed by the processor 604, cause various operations to implement the disclosed examples.

[0119] The instructions 602 can be transmitted or received using a transmission medium via a network interface device and using any one of a number of well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)), as described above. Similarly, the instructions 602 can be transmitted or received using a transmission medium via a coupling (e.g., a peer-to-peer coupling) between the devices 640.

[0120] Software Architecture

[0121] Figure 7 is a block diagram 700 illustrating a software architecture 702, which can be installed on any one or more of the devices described herein. The software architecture 702 is supported by hardware, such as machine 704 that includes processors 706, memory 708, and I / O components 710. In this example, the software architecture 702 can be conceptualized as a stack of layers, where each layer provides particular functionality. The software architecture 702 includes, e.g., layers of an operating system 712, libraries 714, frameworks 716, and applications 718. Operationally, the applications 718 invoke API calls 720 through the software stack and receive messages 722 in response to the API calls 720.

[0122] The operating system 712 manages hardware resources and provides common services. The operating system 712 includes, e.g., a kernel 724, services 726, and drivers 728. The kernel 724 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 724 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 726 can provide other common services that the other software layers can use. The drivers 728 are responsible for controlling or interfacing with the underlying hardware. For instance, the drivers 728 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, and so forth.

[0123] The libraries 714 provide a common low-level infrastructure used by the applications 718. The libraries 714 can include system libraries 730 (e.g., C standard library) providing functionality such as memory allocation functions, string manipulation functions, mathematic functions, and the like. Further, the libraries 714 can include API libraries 732 such as media libraries (e.g., libraries to support presentation and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), graphics libraries (e.g., an OpenGL framework used to render two dimensional (2D) and three dimensional (3D) graphics on a display), database libraries (e.g., SQLite to provide various relational database functions), web libraries (e.g., WebKit to provide web browsing functionality), and the like. The libraries 714 also include a wide variety of other libraries 734 to provide many other APIs to the applications 718.

[0124] The frameworks 716 provide a common high-level infrastructure used by the applications 718. For example, the frameworks 716 provide various graphical user interface (GUI) functions, high-level resource management, and high-level location services. The frameworks 716 can provide a broad spectrum of other APIs that can be used by the applications 718, some of which can be specific to a particular operating system or platform.

[0125] In an example, the applications 718 include a home application 736, a contacts application 738, a browser application 740, a book reader application 742, a location application 744, a media application 746, a messaging application 748, a game application 750, and a broad assortment of other applications such as a third party application 752. The applications 718 are programs that execute functions defined in the programs. Programs can be written in various programming languages such as an object oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, the third party application 752 (e.g., an application developed using the ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of the particular platform) can be mobile software running on a mobile operating system such as IOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. In this example, the third party application 752 is able to use the API calls 720 provided by the operating system 712 to facilitate functionality described herein.

[0126] Figure 8 is a perspective view of a head-mountable device in the form of eyewear 800 according to some examples. The eyewear 800 is an eyewear article that includes electronics that operate within a network system for communicating image and video content. Figure 8 Examples of head-mountable devices 116 are shown. In some examples, the wearable electronic device is referred to as an augmented reality (AR), mixed reality (MR), virtual reality (VR) glasses. The eyewear 800 can include a frame 832 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. The frame 832 can have a front piece 833 that can include a first or left lens, display, or optical element holder 836 and a second or right lens, display, or optical element holder 837 connected by a bridge piece 838. The front piece 833 additionally includes a left end 841 and a right end 842. A first or left optical element 844 and a second or right optical element 843 can be disposed within the respective left optical element holder 836 and right optical element holder 837. Each of the optical elements 843, 844 can be a lens, a display, a display assembly, or a combination of the foregoing. In some examples, the eyewear 800 is provided with an integrated near-eye display mechanism that enables, for example, display of a preview image of visual media captured by a camera 869 of the eyewear 800 to a user.

[0127] The frame 832 additionally includes a left or temple piece 846 and a right or temple piece 847 that are coupled to the respective left end 841 and right end 842 of the front piece 833 by any suitable means, such as a hinge (not shown), so as to be coupled to or rigidly or fixedly secured to the front piece 833, so as to be integral with the front piece 833. Each of the temple pieces 846 and 847 can include a first portion 851 that is coupled to the respective end 841 or 842 of the front piece 833 and any suitable second portion 852, such as a bend or arc, for coupling to an ear of a user. In one example, the front piece 833 can be formed of a single piece of material to have a monolithic or unitary construction. In one example, the entire frame 832 can be formed of a single piece of material to have a monolithic or unitary construction.

[0128] Glasses 800 include a computing device such as computer 861, which can be of any suitable type to be carried by frame 832, and in one example has a suitable size and shape to be at least partially disposed in one or more of temple pieces 846 and 847. In one example, computer 861 has a size and shape similar to that of one of temple pieces 846, 847, and thus is disposed nearly entirely, if not entirely, within the structure and bounds of such temple piece 846, 847.

[0129] In one example, computer 861 can be disposed in both temple pieces 846, 847. Computer 861 can include one or more processors with memory, wireless communication circuitry, and a power source. Computer 861 includes low-power circuitry, high-speed circuitry, location circuitry, and a display processor. Various other examples can include these elements in different configurations or integrated together in different ways. Additional details of aspects of computer 861 can be implemented as described with reference to the following description.

[0130] Computer 861 additionally includes battery 862 or other suitable portable power supply. In one example, battery 862 is disposed in one of temple pieces 846 or 847. In Figure 8 In the illustrated glasses 800, battery 862 is shown disposed in left temple piece 846 and electrically coupled with connection 874 to the remainder of computer 861 disposed in right temple piece 847. One or more input devices and output devices can include a connector or port (not shown) accessible from the exterior of frame 832 suitable for charging battery 862, a wireless receiver, transmitter, or transceiver (not shown), or a combination of such devices.

[0131] Glasses 800 include digital cameras 869. Although two cameras 869 are depicted, other examples can contemplate the use of a single or additional (i.e., more than two) cameras 869. For ease of description, various features related to cameras 869 will be further described with reference to only a single camera 869, but it will be understood that these features can be applicable to both cameras 869 in suitable examples.

[0132] In various examples, the eyeglasses 800 can include any number of input sensors or peripherals in addition to the camera 869. The front piece 833 is provided with a front-facing, outward-facing, forward-facing, front or outer surface 866 that faces forward or away from the user when the eyeglasses 800 are placed on the user's face, and an opposite inward-facing, rear-facing, back or inner surface 867 that faces the user's face when the eyeglasses 800 are placed on the user's face. Such sensors can include inward-facing video sensors or digital imaging modules, such as the camera 869 that can be mounted on or disposed within the inner surface 867 of the front piece 833 or elsewhere on the frame 832 to face the user, and outward-facing video sensors or digital imaging modules, such as the camera 869 that can be mounted on or disposed within the outer surface 866 of the front piece 833 or elsewhere on the frame 832 to face away from the user. Such sensors, peripherals or peripheral devices can additionally include biometric sensors, location sensors, accelerometers or any other such sensors. In some examples, a projector (not shown) is used to project images on the inner surface of the optical elements 843, 844 (or lenses) to provide a mixed reality or augmented reality experience for the user of the eyeglasses 800.

[0133] The eyeglasses 800 also include an example of a camera control mechanism or user input mechanism that includes a camera control button mounted on the frame 832 for tactile or manual engagement by the user. The camera control button provides a dual-mode or single-action mechanism, i.e., the user can only set it between two states, an engaged state and a disengaged state. In this example, the camera control button is a button that is in the disengaged state by default, which can be pressed down by the user to place it in the engaged state. When the pressed-down camera control button is released, it automatically returns to the disengaged state.

[0134] In other examples, the single-action input mechanism can instead be implemented by a touch-sensitive button, e.g., including a capacitive sensor, mounted on the frame 832 adjacent to a surface of the frame to detect the presence of a user's finger to set the touch-sensitive button to the engaged state when the user touches a corresponding point on the outer surface 866 of the frame 832. It will be understood that the camera control button and capacitive touch button described above are merely two examples of tactile input mechanisms for single-action control of the camera 869, and other examples can employ different single-action tactile control arrangements.

[0135] The computer 861 is configured to perform the methods described herein. In some examples, the computer 861 is coupled to one or more antennas for receiving signals from a GNSS and circuitry for processing the signals, where the antennas and circuitry are housed in the eyeglasses 800. In some examples, the computer 861 is coupled to one or more wireless antennas and circuitry for transmitting and receiving wireless signals, where the antennas and circuitry are housed in the eyeglasses 800. In some examples, multiple sets of antennas and circuitry are housed in the eyeglasses 800. In some examples, the antennas and circuitry are configured to operate according to a communication protocol, such as Bluetooth™, Bluetooth™ Low Energy, IEEE 802, IEEE 802.11az / be, WiFi®, etc. In some examples, a PDR sensor is housed in the eyeglasses 800 and coupled to the computer 861. In some examples, the eyeglasses 800 are a VR headset, where the optical elements 843, 844 are opaque screens for displaying images to a user of the VR headset. In some examples, the computer 861 is coupled to user interface elements, such as a slider or touchpad 876 and buttons 878. A long press of the buttons 878 resets the eyeglasses 800. The slider or touchpad 876 and buttons 878 are used for a user to provide input to the computer 861 and / or other electronic components of the eyeglasses 800. The eyeglasses 800 include one or more microphones 882 coupled to the computer 861. The eyeglasses 800 include one or more gyroscopes 880.

[0136] Generating ground truth for generative AI

[0137] Figure 9 An image generator training component 902 is shown in accordance with some examples. Figure 10 An example of a system 1000 for generating ground truth for a generative neural network is shown in accordance with some examples. Figure 9 And Figure 10 are discussed in conjunction with one another. The image generator training component 902 trains an image generator neural network component 1008. The reference images 904 are images that have attributes of ground truth outputs or target output images for the image generator neural network 1402. For example, the reference images 904 can be full-body augmented reality (AR) style images. AR style images are images that can combine real-world images with digitally created images. The pose generator component 906 generates a pose 1002 from a reference image 1004 that the image generator training component 902 selects from the reference images 904. The pose 1002 captures some keypoints in the reference image 1004 to capture a pose 1002 of a person in the reference image 1004. The pose 1002 is determined using a neural network trained to extract the pose 1002 from the reference image 1004 or using other methods that can apply traditional image processing.

[0138] Text prompt 1016 is a description of reference image 1004. For example, "full body pose of a person wearing a pink dress" 1022. One or more regions of reference image 1004 are occluded or blacked out to form one or more inpainting regions 1012. Inpainting selector component 912 selects inpainting regions 1012 or inpainting masks or input regions. In some examples, inpainting selector component 912 randomly selects inpainting regions 1012 during training while ensuring that different parts of reference image 1004 are selected. The values of the pixels within inpainting regions 1012 are changed to different values, for example, all the same value indicating black, white, or other color. In some implementations, a Gaussian distribution is used to determine the change in values of the pixels.

[0139] Image generator training component 902 trains image generator neural network component 1008 by providing three inputs including pose 1002, reference image 1004, and inpainting regions 1012. Discriminator component 908 compares generated image 1018 to reference image 1004 without inpainting regions 1012 and determines a difference. Weight update component 910 makes changes to weights 1014 based on the difference. In some examples, another input is a keyword 1013 describing the person or object depicted in reference image 1004, where example keywords are the gender, race, skin color, etc. of the person in reference image 1004.

[0140] Neural network structure 1010 includes convolutional layers, pooling layers, input layers, output layers, attention layers, etc. Neural network structure 1010 combines a diffusion model for updating weights 1014 with an additional block structure for updating weights 1014. In the diffusion model, weight update component 910 uses Gaussian noise to change reference image 1004, where discriminator component 908 determines a difference between generated image 1018 and reference image 1004. Weight update component 910 then adjusts weights 1014 based on the output of discriminator component 908 using backpropagation. Image generator neural network component 1008 learns to generate reference image 1004 from reference image 1004 with added Gaussian noise. Image generator training component 902 can repeat this method for different sets of reference images 904, for example, "full body pose of a person wearing a pink dress," "upper body pose of a person wearing a black t-shirt," "back pose of a person wearing a halter dress," "full body AR style of a person wearing a black t-shirt," "portrait of a person in cartoon style," etc. Each of the sets of reference images 904 can be a number such as five to fifty or other number. In some examples, reference images 904 can have a cartoon style, which can be used to convert images of people to cartoon images of people.

[0141] Additionally, the weight update component 910 divides the neural network structure 1010 into blocks, where the weights 1014 of some blocks are fixed, while the weights of other blocks are adjusted by the weight update component 910. The blocks whose weights 1014 are adjusted by the weight update component 910 learn conditions such as for different reference images 904 (e.g., for the “pink dress” reference image 904). The blocks whose weights 1014 are fixed retain what the image generator neural network component 1008 has already learned. For example, some of the fixed weights 1014 can be retained after the image generator training component 902 uses a set of reference images 904 for “full body poses of people wearing pink dresses.” Additionally, other fixed weights 1014 blocks can be used for different keywords 1013, sets of reference images 904, and / or text prompts 1016. The pose 1002 facilitates the image generator neural network component 1008 associating the pose 1002 with the reference image 1004 to generate a generated image 1018 having the same pose 1002. The discriminator component 908 generates the pose 1002 for the generated image 1018 as part of the loss determination. In some examples, random augmentations are generated for the reference image 1004, and the image generator training component 902 trains the image generator neural network component 1008 by generating random augmentations that are added to the reference image 1004, in which case the discriminator uses the original reference image 1004 to determine the difference. This training with random augmentations avoids model overfitting. When the image generator training component 902 completes training of the image generator neural network component 1008, the image generator neural network component 1008 produces an arbitrary person having the characteristics of the reference image 904, such as a person wearing a “pink dress” 1006.

[0142] Figure 11 A training set generator component 1102 is shown in accordance with some examples. Figure 12 A system 1200 for generating ground truth for a generative neural network is shown in accordance with some examples. Figure 11 and Figure 12 are discussed in conjunction with one another. The training set generator component 1102 uses the trained image generator neural network component 1008 and images 1104 to generate generated input and output pairs 1114 that can be referred to as ground truth. The images 1104 are images that match the descriptions for which the image generator neural network component 1008 was trained. For example, the images include full bodies of people. Continuing the example of Figure 10 The images 1104 include full bodies of people. An image with a body part mask 1204 is an example of an image 1104. There can be objects in the images 1104 other than people that are to be transformed. For example, a building can be transformed to have a modern appearance, where the transformation preserves some features of the building, such as a logo, doors, etc.

[0143] Figure 12 Three examples 1202 of the generated input and output pair generating training set generator component 1102 are shown in FIG. 12. The mask generator component 1106 takes as input pair an image 1104 (e.g., an image associated with a body part mask 1204) and generates as output pair a generated image 1214.

[0144] The mask generator component 1106 generates a body part mask 1204, which can include a mask around the head, hair, hands, feet, and other body parts of a person in the input image 1104 or can be used for different masks of objects, such as a mask for identifying a logo on an object. According to some examples, the mask generator component 1106 uses a neural network to identify portions of a person or object within the input image 1104.

[0145] The body bounding box generator component 1108 takes the image 1104 and generates a bounding box around the person or object within the image 1104. The body bounding box generator component 1108 uses a neural network trained to identify a person or object.

[0146] The inpainting mask generator 1110 generates an inpainting mask 1208 as an exception to the body part mask 1204. The inpainting mask 1208 enables features such as an arm 1221 of the original input image 1205 to be retained in the generated image 1214. The keyword generator component 1112 generates a keyword 1212 for the image 1104 and can use a trained neural network to determine the keyword 1212. In some examples, the keyword 1212 is the same or similar to the keyword 1013. The pose generator component 906 generates a pose from the image 1104. According to some examples, the text cue 1016 indicates one of the translations for which the image generator neural network component 1008 is trained by the image generator training component 902. Other text cues 1016 can be used, where the image generator neural network component 1008 interprets the text cue 1016 according to other text cues 1016 for which it has been trained to recognize.

[0147] The training set generator component 1102 generates the generated input and output pair 1114 by repeating the pose 1210, the inpainting mask 1208, the text cue 1016, and in some examples the keyword 1212 into the human generator neural network component 1108. The generated image 1214 matches the input image 1104. Figure 12 Examples 1202 one, two, and three are shown, so three generated input and output pairs 1114 are generated. The number of generated input and output pairs 1114 can be in the thousands.

[0148] After the image generator neural network component 1008 generates the image 1214, the training set generator component 1102 augments the generated image 1214 by fusing the body part mask 1204 with the generated image 1214. The mask generator component 1106 identifies features or characteristics, such as hair, arms, neck, feet, gloves being worn by the person, shoes being worn by the person, etc. The mask generator component 1106 identifies some features, such as the hair 1206 extending too far beyond the bounding box of the head 1207, so the mask generator component 1106 identifies the portion of the hair 1206 that extends into the repair mask 1208. The training set generator component 1102 can then utilize the identified hair 1206 to fuse the hair 1206 into the generated image 1214. In some examples, the input image 1205 depicts objects associated with the person, such as a stethoscope, a mobile device, glasses, etc. The training set generator component 1102 can then utilize the identified objects to fuse them into the generated image 1214.

[0149] In some examples, the fusing is performed within the repair mask 1208 to enable, for example, a person with long hair to have their hair flow over the “pink dress” 1020. The fusing improves the final generated input and output pair 1114 by incorporating actual portions of the input image into the output image and enables portions of the input image that are part of the body part mask 1204 to be fused into the repair mask 1208. The fusing can be performed using a neural network trained to fuse the body part mask 1204, such as a neural network to fuse hair. Additionally, the repair mask 1208 indicates areas to be filled in by the image generator neural network component 1008 based on the repair region 1012 utilizing Figure 10 The repair mask 1208 enables the background of the image of the example 1202 to be preserved in the generated image 1214.

[0150] In some examples, the training set generator component 1102 examines the generated input and output pairs 1114 and removes, excludes, or discards those that do not match the pose 1210. For example, the training set generator component 1102 generates a pose for the generated image 1214 and compares the pose to the pose 1210 and excludes those generated images 1214 where the difference between the poses exceeds a threshold difference. In some examples, the training set generator component 1102 additionally analyzes the generated image 1214 to determine whether the generated image 1214 matches the text prompt 1016. For example, the training set generator component 1102 uses a neural network to determine whether the generated image 1214 is determined to be wearing the “pink dress” 1020, and the generated image 1214 is removed, excluded, or discarded if the generated image 1214 does not exceed a threshold for wearing the “pink dress” 1020.

[0151] The image generator neural network component 1008 enables the training set generator component 1102 to generate ground truth or input and output pairs 1114 to train another neural network. The pose 1210 enables the image generator neural network component 1008 to more reliably match the pose 1210 of the input image 1104. The keywords 1212 enable the image generator neural network component 1008 to more reliably match the input image 1104. In some examples, the training set generator component 1102 randomly selects (e.g., picks from the internet or samples from a publicly available image database) a large number of input images to generate the generated input and output pairs 1114.

[0152] Figure 13 A production training component 1302 is shown in accordance with some embodiments. Figure 14 Training of a production generative neural network 1402 is shown in accordance with some examples. The production training component 1302 uses the generated input and output pairs 1114 to train the weights 1406 of the production generative neural network 1402. There are three examples 1408 shown as one, two, and three. Each example 1408 is a generated input and output pair 1114. The production training component 1302 inputs an input image 1410 to the generative neural network 1402, which generates a generated output image 1412. The discriminator component 1304 determines a loss based on a difference between the generated output image 1412 and an output image 1414 for the corresponding input image 1410 of the generated input and output pair 1114. The generated input and output pairs 1114 enable the production training component 1302 to train the production generative neural network 1402. The weight update component 1306 updates the weights 1406 based on the loss determined by the discriminator component 1304 using a method such as backpropagation. In some examples, the neural network structure 1404 is an image-to-image deep learning model with supervised learning.

[0153] Figure 15 An example generative neural network application is shown in accordance with some examples. The computing device 114 includes a camera 1504 and a screen 1506. In some examples, the input image 1510 is an image captured by the camera 1504. A user of the computing device 114 is provided with the option to select the generative neural network application 1508 to convert the input image 1510 to an output image 1512. The reference image 1509 shows the image conversion performed by the generative neural network application 1508. The user can select to switch back to the input image 1510 by selecting the switch back 1514. The output image 1512 is generated in accordance with the generative neural network application 1508. Figure 14The training described in the middle transforms the input image 1510. In this example, the full-body pose of the woman is transformed into a "full-body pose of a person wearing a pink dress." In some examples, the computing device 114 is a head wearable 116, where the output image 1512 is displayed on the screen 1506 of the head wearable 116. The input image 1510 can be a video or a series of images, which can be transformed into the output image 1512 in real-time, which can be a video or a series of images. The output image 1512 can be shared on the social networking platform 222.

[0154] Figure 16 A method 1600 for generating an image generator neural network application is shown, in accordance with some embodiments. The method 1600 begins at operation 1602 with collecting reference images. For example, as discussed in connection with Figure 9 and Figure 10 the reference images 904, e.g., an image of a "full-body pose of a person wearing a pink dress," are collected. This collection can be performed by a user or a computing device.

[0155] The method 1600 continues at operation 1604 with performing an image generator training method. For example, as discussed in connection with Figure 9 and Figure 10 the image generator training component 902 can train the image generator neural network component 1008.

[0156] The method 1600 continues at operation 1606 with performing a training set generator method. For example, as discussed in connection with Figure 11 and Figure 12 the training set generator component 1102 generates the generated input and output pairs 1114.

[0157] The method 1600 continues at operation 1608 with generating an image generator neural network application. For example, as discussed in connection with Figure 13 and 14 the production training component 1302 trains the production generative neural network 1402, and the image generator neural network application component 1702 generates an application that includes the functionality of the production generative neural network 1402. In some examples, the image generator neural network application component 1702 generates the application by providing the reference image 1408 to illustrate the image transformation performed and providing the weights 1406 to an application, such as the generative neural network application 1508, or the image generator neural network application component 1702 can generate the generative neural network application 1508 and can make it available on the social networking platform 222.

[0158] In some examples, the method 1600 can optionally include one or more additional operations, which can be performed in a different order, or one or more of the operations of the method 1600 can be optional. The method 1600 is performed, in whole or in part, by one or more computing devices having computing circuitry configured to perform or implement the functionality of the corresponding components described herein. Figures 1 to 18 The disclosed components of the computing device are performed by one or more apparatuses of the components, such as the functionality of the production training component 1302 is performed or implemented by one or more computing devices or processing circuitry configured to perform or implement the functionality of the corresponding component.

[0159] Figure 17 A generate image generator neural network application component is shown, according to some embodiments. In some examples, the generate image generator neural network application component 1702 is configured to perform the method 1600. The input image 1104 is a reference image 904 as described in connection with Figure 9 and Figure 10 The user of the social network platform 222 can find some images that each depict a “full body pose of a person wearing a pink dress,” for example. The textual prompt 1016 and / or the keywords 1013 are determined by the generate image generator neural network application component 1702, can be associated with the images 1104 by the generate image generator neural network application component 1702, or the user can provide the textual prompt 1016 and / or the keywords 1013.

[0160] The generate image generator neural network application component 1702 then performs the method 1600 and generates the neural network structure 1404, the weights 1406, the reference image 1509, and / or the image generative neural network application 1508. In some examples, the neural network structure 1404 is the same for each generative neural network application 1508 generated, or is different for each image generative neural network application 1508. The reference image 1408 is selected by the user or the image generator neural network application component 1702. The reference image 1408 is an image 1509 that is used to illustrate the image transformation performed by the generative neural network application 1508. The generate image generator neural network application component 1702 is an application that is available to the user in a similar manner to the way that the generative neural network application 1508 is available to the user, in which the user is prompted to provide an image 1104 or a description of an image, after which the generate image generator neural network application component 1702 retrieves from, for example, the internet or an image repository.

[0161] Figure 18 A method 1800 for generating ground truth for generative neural networks is shown, according to some embodiments. The method 1800 begins at operation 1802, where a computing device determines a pose of a person depicted in a plurality of input images. For example, as described in connection with Figure 9 andFigure 10 The discussed pose generator component 906 determines a pose 1002 for each of the reference images 1004.

[0162] The method 1800 continues at operation 1804 with changing values of pixels within a region of the plurality of input images. For example, the inpainting selector component 912 changes values of pixels within the inpainting region 1012. The method 1800 continues at operation 1806 with inputting the pose, the region, and the textual cue describing the plurality of input images into a neural network to generate a plurality of output images. For example, the image generator training component 902 processes the image generator neural network component 1008 with input including the pose 1002, the reference image 1004 with the inpainting region 1012 or the inpainting region 1012, and the textual cue 1016.

[0163] The method 1800 continues at operation 1808 with determining a loss between the plurality of output images and the plurality of input images. For example, the discriminator component 908 determines a loss between the generated image 1018 and the reference image 1004 without the inpainting region 1012. The method 1800 continues at operation 1810 with updating weights of the neural network based on the loss. For example, the weight update component 910 updates the weights 1014 with backpropagation based on the determined loss.

[0164] The method 1800 can optionally include one or more additional operations, the operations of the method 1800 can be executed in a different order, or one or more operations of the method 1800 can be optional. The method 1800 can be performed wholly or partly by one or more computing devices or one or more apparatuses of one or more computing devices having the functionality described herein in connection with Figures 1 to 18 The disclosed one or more computing devices or one or more apparatuses of one or more computing devices to perform. The functionality of components such as the production training component 1302 is performed or implemented by one or more computing devices or processing circuitry configured to perform or implement the functionality of the corresponding component.

[0165] Glossary

[0166] “Carrier signal” means any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals. The instructions can be transmitted or received over a network via a network interface device using a transmission medium.

[0167] A "client device" refers to, for example, any machine that interfaces with a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistants (PDAs), smart phones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0168] A "communications network" refers to, for example, one or more portions of a network that can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, other appropriate type of network, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless or cellular network, and coupling can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other type of cellular or wireless coupling. In this example, coupling can enable any of a variety of types of data transfer technology, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer technology.

[0169] A "component" or "module" refers to, for example, a logic or an apparatus, a physical entity that can be implemented in either a hardware abstract or a software abstract that has boundaries defined by a function or a subroutine call, a branch point, an API, or a other tangible mechanisms that provide an interface between such entity and other components or modules. A component or module can be a hardware component that is designed for use with other components in a specific architecture and typically provides a particular functionality to other components via an interface. A component or module can be a software component, which can be executed by a hardware processor, that performs related functions. A "hardware component" or "hardware module" is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. A hardware component can include software configured to permanently configure or temporarily configure programmable logic forThe software accordingly configures one or more specific processors to constitute a particular hardware component at one time and different hardware components at different times. Hardware components or modules can provide information to and receive information from other hardware components. Therefore, the described hardware components or modules can be considered communicatively coupled. In the case of multiple hardware components present simultaneously, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In examples where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessible to the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Another hardware component or module can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., collections of information). The various operations of the example methods described herein can be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute components of a processor implementation that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented, at least in part, by processors, where a particular one or more processors are examples of hardware. For example, at least some of the various operations of the methods can be performed by one or more processors or components implemented by processors. Furthermore, one or more processors can also operate to support the execution of relevant operations in a "cloud computing" environment or as a "Software as a Service" (SaaS) operation. For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), where these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing only within a single machine, but deployed across multiple machines. In some examples, the processor or processor implementation components may be located in a single geographic location (e.g., within a home environment, an office environment, or a server cluster). In other examples, the processor or processor implementation components may be distributed across multiple geographic locations.

[0170] "Computer-readable storage media" refers to both machine-storage media and transmission media. Thus, the terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure.

[0171] "Transient message" refers to a message that is accessible, for example, for a time-limited duration. A transient message can be text, image, video, etc. The access time for a transient message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0172] "Machine-storage medium" refers to a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid state memory, as well as optical and magnetic media, including memory internal or external to a processor. Specific examples of machine-storage media, computer-storage media, and device-storage media include non-volatile memory, including, for example, semiconductor memory devices, e.g., Erasable Programmable Read-Only Memories (EPROM), Electrically Erasable Programmable Read-Only Memories (EEPROM), FPGAs, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine-storage medium," "device-storage medium," "computer-storage medium," mean the same thing and can be used interchangeably in this disclosure. The terms "machine-storage medium," "computer-storage medium," and "device-storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are, by their nature, ephemeral rather than non-volatile.

[0173] "Non-transitory computer-readable storage medium" refers to, for example, a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0174] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals. The term "signal medium" shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

[0175] A“user device” refers to a device that is accessed by, controlled by, or owned by a user and with which the user interacts to perform actions or interactions with other users or computer systems. Other claimed subject matter includes the following:

[0176] Example 1 is a computing device comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations comprising: determining a pose of a person depicted in a plurality of input images; changing values of pixels within a region of the plurality of input images; inputting the pose, the region, and a textual prompt describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss.

[0177] In Example 2, the subject matter of Example 1 includes, wherein changing the values of the pixels comprises: changing the values of the pixels to indicate a same value.

[0178] In Example 3, the subject matter of Examples 1-2 includes, wherein determining the pose for the plurality of input images comprises: identifying key points of the person depicted in each of the plurality of input images; and connecting the key points.

[0179] In Example 4, the subject matter of Examples 1-3 includes, wherein the textual prompt describing the plurality of input images describes one or more common features of the plurality of input images.

[0180] In Example 5, the subject matter of Examples 1-4 includes, wherein the operations further comprise: determining one or more keywords for each of the plurality of input images; and wherein the input further comprises: inputting the one or more keywords.

[0181] In Example 6, the subject matter of Examples 1-5 includes, wherein the operations further comprise: determining keywords for the plurality of input images, wherein the keywords describe features of the person depicted within a corresponding input image of the plurality of input images, and wherein the input to the neural network further comprises the keywords.

[0182] In Example 7, the subject matter of Example 6 includes, wherein the keywords comprise at least one of: a gender of the person, a skin color of the person, a hair color of the person, an indication of whether the person is wearing gloves, an indication of whether the person is wearing glasses, and an indication of a color of shoes worn by the person.

[0183] In Example 8, the subject matter of Examples 1-7 includes, wherein the pose is a first pose, and wherein the operations further comprise: determining body part masks for the plurality of ground truth input images; determining bounding boxes for the plurality of ground truth input images, each bounding box comprising a person depicted within a corresponding ground truth input image of the plurality of ground truth input images; determining regions comprising a difference between the bounding boxes and the corresponding body part masks for the plurality of ground truth input images; changing values of pixels within the regions of the plurality of ground truth input images; determining a second pose for the plurality of ground truth input images; and inputting the second pose, the regions, and the textual cues into the neural network with the updated weights to generate the plurality of ground truth output images.

[0184] In Example 9, the subject matter of Example 8 includes, wherein the values of the pixels are changed to the same values as the values of the pixels within the regions of the plurality of input images.

[0185] In Example 10, the subject matter of Examples 8-9 includes, wherein the operations further comprise: fusing the body part masks with the plurality of ground truth output images.

[0186] In Example 11, the subject matter of Example 10 includes, wherein the operations further comprise: extracting body parts from the plurality of ground truth input images; and fusing the extracted body parts with corresponding ground truth outputs of the plurality of ground truth output images.

[0187] In Implementation 12, the subject matter of Implementation 11 includes, wherein the fusing is performed outside of the regions.

[0188] In Example 13, the subject matter of Examples 8-12 includes, wherein the operations further comprise: determining a third pose for the plurality of ground truth output images; comparing the second pose to the corresponding third pose; and responsive to a difference between the second pose of the ground truth input image and a third pose of the third pose of the ground truth output image of the plurality of ground truth output images exceeding a threshold, removing the corresponding ground truth output image.

[0189] In Example 14, the subject matter of Examples 8-13 includes, wherein the operations further comprise: determining keywords for the plurality of ground truth input images, wherein the keywords describe features of the person depicted within a corresponding ground truth input image of the plurality of ground truth input images, and wherein the input to the neural network further comprises the keywords.

[0190] In Example 15, the subject matter of Examples 8-14 includes, wherein the neural network is a first neural network, the plurality of input images is a plurality of first input images, and wherein the operations further comprise: inputting a plurality of ground truth input images for generating a plurality of second output images into a second neural network to generate the plurality of second output images; determining a loss between the plurality of second output images and corresponding ground truth output images of the plurality of ground truth output images; and updating weights of the second neural network based on the loss.

[0191] Example 16 is a non-transitory computer-readable storage medium comprising instructions that, when processed by one or more processors, configure the one or more processors to perform operations comprising: determining a pose of a person depicted in a plurality of input images; changing values of pixels within a region of the plurality of input images; inputting the pose, the region, and a textual prompt describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss.

[0192] In Example 17, the subject matter of Example 16 includes, wherein changing the values of the pixels comprises: changing the values of the pixels within the region to indicate a same value.

[0193] In Example 18, the subject matter of Examples 16-17 includes, wherein determining the pose for the plurality of input images comprises: identifying keypoints of the person depicted within the plurality of input images; and connecting the keypoints.

[0194] Example 19 is a method comprising: determining a pose of a person depicted in a plurality of input images; changing values of pixels within a region of the plurality of input images; inputting the pose, the region, and a textual prompt describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss.

[0195] In Example 20, the subject matter of Example 19 includes, wherein changing the values of the pixels comprises: changing the values of the pixels within the region to indicate a same value.

[0196] Example 21 is at least one machine readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement any of Examples 1-20.

[0197] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0198] Example 23 is a system for implementing any of Examples 1-20.

[0199] Example 24 is a method for implementing any of Examples 1-20.

Claims

1. A computing device comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, configure the one or more processors to perform operations comprising: determining poses of a person depicted in a plurality of input images; changing values of pixels within a region of the plurality of input images; inputting the poses, the region, and a textual cue describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss. changing the values of the pixels comprises:

2. The computing device of claim 1, wherein, changing the values of the pixels to indicate a same value. determining the poses for the plurality of input images comprises:

3. The computing device of claim 1, wherein, identifying keypoints of the person depicted in each input image of the plurality of input images; and connecting the keypoints. the textual cue describing the plurality of input images describes one or more common features of the plurality of input images.

4. The computing device of claim 1, wherein, the operations further comprise:

5. The computing device of claim 1, wherein, determining one or more keywords for each input image of the plurality of input images; and wherein the input further comprises: inputting the one or more keywords. the operations further comprise:

6. The computing device of any of claims 1 to 5, wherein, determining a keyword for the plurality of input images, wherein the keyword describes a feature of the person depicted within a corresponding input image of the plurality of input images, and wherein the input to the neural network further comprises the keyword. the keyword comprises at least one of: a gender of the person, a skin color of the person, a hair color of the person, an indication of whether the person is wearing gloves, an indication of whether the person is wearing glasses, and an indication of a color of shoes worn by the person.

7. The computing device of claim 6, wherein, the pose is a first pose, and wherein the operations further comprise:

8. The computing device of any of claims 1 to 5, wherein, determining body part masks for a plurality of ground truth input images; determining bounding boxes for the plurality of ground truth input images, each bounding box comprising the person depicted within a corresponding ground truth input image of the plurality of ground truth input images; determining regions comprising a difference between the bounding boxes and corresponding body part masks for the plurality of ground truth input images; changing values of pixels within the regions of the plurality of ground truth input images; determining second poses for the plurality of ground truth input images; and inputting the second poses, the regions, and the textual cue into the neural network with the updated weights to generate a plurality of ground truth output images. the values of the pixels are changed to a same value as values of pixels within the regions of the plurality of input images.

9. The computing device of claim 8, wherein, the operations further comprise:

10. The computing device of claim 8, wherein, fusing the body part masks with the plurality of ground truth output images. the operations further comprise:

11. The computing device of claim 10, wherein, extracting body parts from the plurality of ground truth input images; and fusing the extracted body parts with corresponding ones of the plurality of ground truth output images. the fusing is outside of the regions.

12. The computing device of claim 11, wherein, the operations further comprise:

13. The computing device of claim 8, wherein, determining third poses for the plurality of ground truth output images; comparing the second poses to corresponding third poses; and ​ remove the corresponding ground truth output image in response to a difference between the second pose of the ground truth input image and a third pose of a third pose of the ground truth output image of the plurality of ground truth output images exceeding a threshold.

14. The computing device of claim 8, wherein, The operations further include: determining a keyword for the plurality of ground truth input images, wherein the keyword describes a feature of a person depicted within a corresponding ground truth input image of the plurality of ground truth input images, and wherein the input to the neural network further includes the keyword.

15. The computing device of claim 8, wherein, The neural network is a first neural network, the plurality of input images is a plurality of first input images, and wherein the operations further include: inputting the plurality of ground truth input images used to generate a plurality of second output images into a second neural network to generate a plurality of second output images; determining a loss between the plurality of second output images and a corresponding ground truth output image of the plurality of ground truth output images; and updating weights of the second neural network based on the loss.

16. A non-transitory computer-readable storage medium comprising instructions that, when processed by one or more processors, configure the one or more processors to perform operations comprising: determining a pose of a person depicted in a plurality of input images; changing a value of a pixel within a region of the plurality of input images; inputting the pose, the region, and a textual prompt describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss.

17. The non-transitory computer-readable storage medium of claim 16, wherein, changing the value of the pixel includes: changing the value of the pixel within the region to indicate a same value.

18. The non-transitory computer-readable storage medium of claim 16 or 17, wherein, determining the pose for the plurality of input images includes: identifying key points of the person depicted within the plurality of input images; and connecting the key points.

19. A method comprising: determining a pose of a person depicted in a plurality of input images; changing a value of a pixel within a region of the plurality of input images; inputting the pose, the region, and a textual prompt describing the plurality of input images into a neural network to generate a plurality of output images; determining a loss between the plurality of output images and the plurality of input images; and updating weights of the neural network based on the loss.

20. The method of claim 19, wherein, changing the value of the pixel includes: changing the value of the pixel within the region to indicate a same value.

Images