Generating visual expressions
The use of a machine learning model on computing devices generates photorealistic images from user inputs, addressing the inconvenience of manual image creation and enhancing image quality and efficiency.
Patent Information
- Application Number
- PCT/US2024/028181
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-14
AI Technical Summary
Users face challenges in creating refined and detailed images, as manually modifying or creating images requires significant skill and time, and using input devices for precise tasks can be inconvenient.
A computing device employs a machine learning model to generate or modify images based on user input, such as drawings or text prompts, producing photorealistic and high-quality images that adhere to the user's creative vision, reducing the need for manual editing and improving image quality.
The solution decreases the time and effort required to create images, enabling users to send refined and detailed images quickly, thereby improving the quality of content shared and reducing interactions with the device.
Smart Images

Figure US2024028181_14112024_PF_FP_ABST
Abstract
Description
GENERATING VISUAL EXPRESSIONSCROSS-REFERENCE TO RELATED APPLICATION
[0001] This application claims the benefit of US Provisional Patent Application No. 63 / 500,838, filed May 8, 2023, which is incorporated by reference herein in its entirety.BACKGROUND
[0002] Users may communicate with each other via one or more computing devices. In some cases, the communications (e.g., an email, a text message, a meeting request, etc.) may include text, images, etc. A computing device may enable a user of a computing device to respond to an incoming communication by allowing the user to include an image and send the image to other computing devices as a response.SUMMARY
[0003] Techniques are described by which a computing device may use a machine learning model to generate or modify visual expressions that users may share with each other. For example, a computing device associated with a user may apply a machine learning model to generate a modified image based on input by the user. The input may include an image (e.g., a picture, a drawing, etc.), a text prompt describing a desired visual expression (including a modification to an existing image), gesture input for drawing a visual expression, etc. In general, the modified image may adhere to the creative vision of the user (e.g., by incorporating elements requested by the user into the modified image) and, at the same time, be a realistic and high-quality image that may be difficult to distinguish from real photos (e.g., photorealistic). In this way, the techniques of this disclosure may provide various advantages, such as saving time and improving image quality. In turn, the techniques of this disclosure may reduce in fewer interactions with and less resources (e.g., power, processor cycles, memory space, memory bus bandwidth, wireless bandwidth) expended by the first computing device.
[0004] In some examples, a method includes obtaining, by a computing device, a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device. The method further includes generating, by the computing device and based on at least the portion of the image, a modified image by at least applying a machine learning model. The method further includes outputting, by thecomputing device and for display, the modified image, and providing, by the computing device and to a second user, a message that includes the modified image.
[0005] In some examples, a computing device includes one or more processors. The computing device further includes a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device. The instructions further cause the one or more processors to generate, based on at least the portion of the image, a modified image by at least applying a machine learning model. The instructions further cause the one or more processors to output, for display, the modified image, and provide, to a second user, a message that includes the modified image.
[0006] In some examples, a computer-readable storage device is encoded with instructions that, when executed by one or more processors of a first computing device, cause the one or more processors to obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device. The instructions further cause the one or more processors to generate, based on at least the portion of the image, a modified image by at least applying a machine learning model. The instructions further cause the one or more processors to output, for display, the modified image, and provide, to a second user, a message that includes the modified image.
[0007] In some examples, a computer program product for image generation includes at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with a computing device. The one or more instructions further cause the at least one processor to generate, based on at least the portion of the image, a modified image by at least applying a machine learning model. The one or more instructions further cause the at least one processor to output, for display, the modified image, and provide, to a second user, a message that includes the modified image.
[0008] In some examples, an apparatus includes means for obtaining, by a computing device, a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device. The apparatus further includes means for generating, by the computing device and based on at least the portion of the image, a modified image by at least applying a machine learning model. The apparatus furtherincludes means for outputting, by the computing device and for display, the modified image, and providing, by the computing device and to a second user, a message that includes the modified image.
[0009] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS
[0010] FIG. 1 A is a conceptual diagram illustrating an example computing environment for generating images in accordance with one or more techniques of this disclosure.
[0011] FIG. IB is a conceptual diagram illustrating a graphical user interface for generating images in accordance with one or more techniques of this disclosure.
[0012] FIG. 2 is a block diagram illustrating one example of a computing device in accordance with one or more techniques of this disclosure.
[0013] FIG. 3 is a block diagram illustrating an example operation of a machine learning model in accordance with techniques of this disclosure.
[0014] FIGS. 4A-4D are conceptual diagrams illustrating graphical user interfaces for generating images in accordance with one or more techniques of this disclosure.
[0015] FIG. 5 is a flow diagram illustrating an example operation for generating images in accordance with one or more techniques of this disclosure.DETAILED DESCRIPTION
[0016] FIG. 1 A is a conceptual diagram illustrating an example computing environment 1 and a graphical user interface 116 (“GUI 116”) for generating images that users may share with each other, in accordance with one or more techniques of this disclosure. As shown in the example of FIG. 1 A, computing environment 1 includes one or more computing devices, such as computing devices 100A-100N (collectively, “computing devices 100”). Examples of computing devices 100 may include, but are not limited to, portable, mobile, or other devices, such as mobile phones (including smartphones), wearable devices (including smartwatches), laptop computers, desktop computers, tablet computers, vehicle head units (e.g., infotainment systems), smart television platforms, server computers, mainframes, etc. In some examples, computing devices 100 may represent a cloud computing system that provides one or more services via the network.That is, in some examples, computing devices 100 may be a distributed computing system.
[0017] As shown in the example of FIG. 1A, computing device 100 A, includes one or more user interface (UI) devices (“UI device 102”). UI device 102 of computing device 100A may be configured to function as an input device and / or an output device for computing device 100 A. UI device 102 may be implemented using various technologies. For instance, UI device 102 may be configured to receive input from a user through tactile, audio, and / or video feedback. Examples of input devices include a presencesensitive or touch-sensitive input device (e.g., a presence-sensitive display), a mouse, a keyboard, a voice responsive system, video camera, microphone or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display may include a touch-sensitive or presence-sensitive input screen, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitance touchscreen, a pressure sensitive screen, an acoustic pulse recognition touchscreen, or another presence-sensitive technology. That is, UI device 102 of computing device 100 A may include a presence-sensitive device that may receive tactile input from a user of computing device 100 A. UI device 102 may receive indications of the tactile input by detecting one or more gestures from the user (e.g., when the user touches or points to one or more locations of UI device 102 with a finger or a stylus pen).
[0018] UI device 102 may additionally or alternatively be configured to function as an output device by providing output to a user using tactile, audio, or video stimuli. Examples of output devices include a sound card, a video graphics adapter card, or any of one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, microLED, miniLED, organic light-emitting diode (OLED) display, e-ink, or similar monochrome or color display capable of outputting visible information to a user of computing device 100A. Additional examples of an output device include a speaker, a haptic device, or other device that can generate intelligible output to a user. For instance, UI device 102 may present output to a user of computing device 100 A as a graphical user interface that may be associated with functionality provided by computing device 100A. In this way, UI device 102 may present various user interfaces of applications executing at or accessible by computing device 100A (e.g., an electronic message application, an Internet browser application, etc.). A user ofcomputing device 100 A may interact with a respective user interface of an application to cause computing device 100 A to perform operations relating to a function.
[0019] In some examples, UI device 102 of computing device 100 A may detect two- dimensional and / or three-dimensional gestures as input from a user of computing device 100A. For instance, a sensor of UI device 102 may detect the user’s movement (e.g., moving a hand, an arm, a pen, a stylus, etc.) within a threshold distance of the sensor of UI device 102. UI device 102 may determine a two- or three-dimensional vector representation of the movement and correlate the vector representation to a gesture input (e.g., a hand-wave, a pinch, a clap, a pen stroke, etc.) that has multiple dimensions. In other words, UI device 102 may, in some examples, detect a multidimensional gesture without requiring the user to gesture at or near a screen or surface at which UI device 102 outputs information for display. Instead, UI device 102 may detect a multidimensional gesture performed at or near a sensor which may or may not be located near the screen or surface at which UI device 102 outputs information for display.
[0020] In the example of FIG. 1 A, computing device 100 A includes user interface (UI) module 104, application module 106A-106N (collectively “application module 106”). Modules 104 and 106 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at computing device 100A. Computing device 100 A may execute modules 104 and 106 with one processor or with multiple processors. In some examples, computing device 100 A may execute modules 104 and 106 as a virtual machine executing on underlying hardware. Modules 104 and 106 may execute as one or more services of an operating system or computing platform or may execute as one or more executable programs at an application layer of a computing platform.
[0021] UI module 104, as shown in the example of FIG. 1A, may be operable by computing device 100 A to perform one or more functions, such as receive input and send indications of such input to other components associated with computing device 100 A, such as application module 106. UI module 104 may also receive data from components associated with computing device 100A such as application module 106. Using the data received, UI module 104 may cause other components associated with computing device 100 A, such as UI device 102, to provide output based on the data. For instance, UI module 104 may receive data from one of application modules 106 to display a GUI.
[0022] Application modules 106, as shown in the example of FIG. 1 A, may include functionality to perform any variety of operations on computing device 100 A. Forinstance, application modules 106 may include a word processor, a text application, a web browser, a multimedia player, a calendar application, an operating system, a distributed computing application, a graphic design application, a video editing application, a web development application, or any other application. One of application modules 106 (e.g., application module 106 A) may be a text messaging or Short Message Service (SMS) application. For example, application module 106 A may include functionality to compose outgoing text message communications, receive incoming text message communications, respond to incoming text message communications, and other functions. Application module 106 A, in various examples, may provide data to UI module 104 causing UI device 102 to display a GUI.
[0023] In some examples, one or more of application module 106 may be operable to receive incoming communications from other devices (e.g., via the network). For instance, one or more of application module 106 may receive text messages for an account associated with a user of computing device 100 A, calendar alerts or meeting requests for a user of computing device 100 A, or other incoming communications.
[0024] In general, communications may include information (e.g., generated in response to input by users of the other devices). Examples of information include text (e.g., any combination of letters, words, numbers, punctuation, etc.), emoji, emoticons, images, animations, video, audio, or any other content that may be included in a communication. In the example of FIG. 1 A, application module 106 A may receive an incoming communication (e.g., a text message) from another computing device (e.g., computing device 100N).
[0025] Computing devices 100 may communicate with each other via a network. The network may include any public or private communication network, such as a cellular network, Wi-Fi network, a direct cell-to-satellite communication network, or other type of network for transmitting data between computing devices. In some examples, the network may represent one or more packet switched networks, such as the Internet. Computing devices 100 may send and receive data across the network using any suitable communication techniques. For example, computing devices 100 may each be operatively coupled to the network using respective network links. The network may include network hubs, network switches, network routers, etc., that are operatively inter-coupled thereby providing for the exchange of information between computing devices 100. In some examples, network links of the network may be Ethernet, ATM or other network connections. Such connections may include wireless and / or wired connections.
[0026] In general, a user may use a communication platform provided by application module 106 to electronically communicate with another user via computing devices 100 associated with the user and the other user. For example, the user may use computing device lOOAto send messages to and receive messages from the other user (who may also be using computing device 100 A or another computing device, such as computing device 100N). In some examples, the user may want to send a message that includes an image. When doing so, the user may seek to create or modify an image. However, manually creating or modifying an image may require considerable skill and time in order to produce a refined and detailed image. Furthermore, the user may find using UI device 102 to perform precise tasks (e.g., drawing) particularly challenging and inconvenient.
[0027] In accordance with techniques of this disclosure, computing device 100A (and other computing devices 100) may include a suggestion module 108 that generates a modified image. Suggestion module 108 may represent or otherwise apply an ML model 138 trained to generate a modified image based on user input. In some examples, the user input may include a drawing of which ML model 138 generates a photorealistic representation. In this way, the techniques of this disclosure may help manifest the imagination of a user of computing device 100 A, allowing for greater and more effective expression. In turn, the techniques may reduce the number of interactions with computing device 100A and the time required to achieve an image satisfying the creative vision of a user.
[0028] Computing device 100 A may include suggestion module 108, which may include or otherwise apply ML model 138. ML model 138 is described in greater detail below (e.g., with respect to FIG. 2). Briefly, however, ML model 138 may be trained on a large amount of data in order to generate images. In some examples, ML model 138 may use complex algorithms and neural networks to analyze the underlying patterns, textures, and structures that make images realistic and / or visually appealing. ML model 138 may perform various computer vision tasks, such as image classification, object detection, semantic segmentation, pose estimation, face detection and recognition, image synthesis and style transfer, depth estimation and 3D reconstruction, etc.
[0029] Suggestion module 108 may obtain an image 110 (shown as part of GUI 116A) for a message by a user associated with computing device 100 A. The message may be for communication to a different user (e.g., associated with computing device 100N). The message may be exchanged between two or more users within a communication platform(e.g., provided by application modules 106), such as a messaging application, a social media platform, etc.
[0030] Image 110 may include one or more drawings by a user of computing device 100 or a pre-existing image. For example, the user may provide an input that draws at least a portion of image 110. In some examples, the user may draw the entirety of image 110. In other examples, the user may provide an input that selects a pre-existing image (e.g., an image stored by computing device 100A), and the user may also provide an input that draws a set of elements in relation to the pre-existing image. In these examples, image 110 may include both the set of elements and the pre-existing image, irrespective of whether the set of elements and the pre-existing image exist in the same image layer (e.g., a separate level or plane within an image that can be independently manipulated without affecting other image layers). Image 110 may constitute or otherwise represent the overlapping image formed by the pre-existing image and the set of elements drawn by the user.
[0031] For example, a user may provide an input that selects a pre-existing image of a person (e.g., a photograph of the user). The user may provide an input that draws a portion of image 110 that includes a set of elements (e.g., a hat, a person, a cat, a tree, a sun, etc.) in relation to the pre-existing image. For example, the user may draw a hat on top of the head of the person in the pre-existing image. Image 110 may therefore be the overlapping image formed by the pre-existing image of the person and the hat drawn on top of the head of the person. That said, in the example of FIG. 1 A, the entirety of image 110 is a drawing of a mountain by the user.
[0032] In some examples, GUI 116Amay include area 118 for drawing. The user may provide an input that draws at least a portion of image 110 within area 118. Additionally, UI module 104 may output, for display by UI device 102, image 110 in area 118, thereby enabling the user to see (e.g., in real-time) the effect of the input on image 110. The user may draw the entirety of image 110 using area 118. Alternatively, the user may select a pre-existing image, causing GUI 116Ato include the pre-existing image in area 118. The user may then draw a set of elements in relation to the pre-existing image. The relationship between the set of elements and the pre-existing image may be at least based on the position of the set of elements relative to the position of the pre-existing image.
[0033] Suggestion module 108 may provide image 110 to ML model 138, which ML model 138 may use as input to generate a modified image 112. In other words, ML model 138 may generate modified image 112 based on image 110, particularly the portion ofimage 110 that includes drawings by the user. For example, ML model 138 may render (e.g., synthesize, create, etc.) modified image 112, which is a photorealistic representation of a mountain based on image 110. The images generated by ML model 138 may be high- quality in various respects. For example, modified image 112 may be photorealistic (e.g., accurately represent color, lighting, real -world subjects, etc.), include intricate and complex patterns and textures, etc. Additionally, ML model 138 may produce one or more unique images (e.g., different variations), apply an artistic style (e.g., cubism, impressionism, minimalism, contemporary, abstract, baroque, etc.), etc.
[0034] In general, suggestion module 108 may provide information (e.g., image 110, messages between users, etc.) to ML model 138 only if computing device 100 receives permission from the user to send the information. For example, in situations discussed here in which computing device 100 may collect, transmit, or may make use of personal information about a user, the user may be provided with an opportunity to control whether programs or features of computing device 100 can collect user information (e.g., information about a user’s communications, a user’s social network, a user’s social actions or activities, a user’s profession, a user’s preferences, or a user’s current location), or to control whether and / or how computing device 100 may store and share user information.
[0035] In addition, certain data may be treated in one or more ways before it is stored, transmitted, or used so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined about the user. Thus, the user may have control over how information is collected about the user and stored, transmitted, and / or used in accordance with techniques of this disclosure.
[0036] In some examples, prior to generating modified image 112, ML model 138 may perform image classification to classify (e.g., label, characterize, identify, describe, etc.) image 110. Additionally or alternatively, the user may provide a prompt 114 (which may be based on, e.g., a text input, a voice input, etc.) that describes one or more of image 110 (e.g., the set of elements drawn by the user) or an intended modification to image 110. In any case, responsive to classifying image 110, ML model 138 may output modified image 110 based on image 110 as well as prompt 114. For example, modified image 110 may broadly incorporate properties of image 110 such as the number of mountains, the size of the mountains, the shape of the mountains, etc. At the same time, modified image 110 may incorporate patterns, textures, and colors based on the classification of ‘mountain’that result in a photorealistic representation (or any other preferred stylistic representation) of a mountain.
[0037] As shown in FIG. IB, UI module 104 may output, for display by UI device 102, modified image 112 in area 118 of GUI 116B. The user may review modified image 112 and, if desirable, perform an action. For example, the user may provide an input that causes ML model 138 to generate variations of modified image 112 without changing the input (e.g., image 110) to ML model 138. In another example, the user may edit modified image 112 (e.g., by using area 118 to draw a set of elements), and ML model 138 may use the edited modified image as input to generate another modified image. In yet another example, the user may provide an input that causes computing device 100 to provide modified image 112 to another user (e.g., by sending modified image 112 to another user as a message, by attaching modified image 112 to a message, etc.).
[0038] The techniques of this disclosure include one or more advantages. For example, instead of sharing simple, rough sketches, the user may instead use computing device 100 to send refined and detailed images produced by ML model 138. Furthermore, ML model 138 may be able to quickly (e.g., relative to a human) produce the refined and detailed images based on the user drawings or other inputs, in this way expeditiously realizing the creative vision of the user. As a result, the techniques of this disclosure may not only decrease the time to compose messages, but also improve the overall quality of the content of the messages sent by the user.
[0039] FIG. 2 is a block diagram illustrating a computing device 200. Computing device 200 may be one example of computing devices 100 in accordance with one or more techniques of this disclosure. FIG. 2 illustrates only one particular example of computing device 200, and many other examples of computing device 200 may be used in other instances and may include a subset of the components included in example computing device 200 or may include additional components not shown in FIG. 2.
[0040] As shown in FIG. 2, computing device 200 may include one or more user interface devices 202 (“UI device 202”), one or more processors 222 (“processor 222”), one or more storage devices 234 (“storage device 224”), one or more communication units 226 (“communication unit 226”). As also shown in FIG. 2, UI device 202 may include one or more input devices 228 (“input device 228”) and one or more output devices 230 (“output device 230”). As also shown in FIG. 2, storage device 224 may include a user interface module 204 (“UI module 204”), one or more application modules206 (“application module 206”), a suggestion module 208, an operating system 234 (“OS 234”), a database 236, a ML model 238, and a model trainer 240.
[0041] In some examples, UI device 202 may be a presence-sensitive display configured to detect input (e.g., touch and non-touch input) from a user of respective computing device 200. UI device 202 may output information to a user in the form of a UI, which may be associated with functionality provided by computing device 200. Such UIs may be associated with computing platforms, operating systems, applications, and / or services executing at or accessible from computing device 200 (e.g., electronic message applications, chat applications, Internet browser applications, mobile or desktop operating systems, social media applications, electronic games, menus, and other types of applications).
[0042] Processor 222 may implement functionality and / or execute instructions within computing device 200. For example, processor 222 may receive and execute instructions that provide the functionality of modules 204-208 and OS 234. These instructions executed by processor 222 may cause computing device 200 to store and / or modify information within storage device 224 of processor 222 during program execution. Processor 222 may execute instructions of modules 204-208 and OS 234 to perform one or more operations. That is Modules 204-208 and OS 234 may be operable by processor 222 to perform various functions described herein.
[0043] Storage device 224 within computing device 200 may store information for processing during operation of computing device 200 (e.g., computing device 200 may store data accessed by modules 204-208 and OS 234 during execution at computing device 200). In some examples, storage device 224 may be a temporary memory, meaning that a primary purpose of storage device 224 is not long-term storage. Storage device 224 on computing device 200 may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.
[0044] Storage device 224 may include one or more computer-readable storage media. Storage device 224 may be configured to store larger amounts of information than volatile memory. Storage device 224 may further be configured for long-term storage of information as non-volatile memory space and retain information after power on / off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs,floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage device 224 may store program instructions and / or information (e.g., within database 236) associated with modules 204-208 and OS 234.
[0045] Communication unit 226 of computing device 200 may communicate with one or more external devices via one or more wired and / or wireless networks by transmitting and / or receiving network signals on the one or more networks. Examples of communication units 226 include a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and / or receive information. Other examples of communication unit 226 may include short wave radios, cellular data radios, wireless network radios, as well as universal serial bus (USB) controllers.
[0046] Input device 228 of computing device 200 may receive input. Examples of input are tactile, audio, and video input. Input device 228 of computing device 200, in one example, includes a presence-sensitive display, a fingerprint sensor, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine.
[0047] Input devices 228 may include one or more sensors. Numerous examples of sensors exist and include any input component configured to obtain environmental information about the circumstances surrounding computing device 200 and / or physiological information that defines the activity state and / or physical well-being of a user of computing device 200. In some examples, a sensor may be an input component that obtains physical position, movement, and / or location information of computing device 200. For instance, sensors may include one or more location sensors (e.g., GNSS components, Wi-Fi components, cellular components), one or more temperature sensors, one or more motion sensors (e.g., multi-axial accelerometers, gyros), one or more pressure sensors (e.g., barometer), one or more ambient light sensors, and one or more other sensors (e.g., microphone, camera, infrared proximity sensor, hygrometer, and the like). Other sensors may include a heart rate sensor, magnetometer, glucose sensor, hygrometer sensor, olfactory sensor, compass sensor, step counter sensor, to name a few other non-limiting examples.
[0048] Output device 230 of computing device 200 may generate one or more outputs. Examples of outputs are tactile, audio, and video output. Output device 230 of computing device 200, in one example, includes a presence-sensitive display, sound card, videographics adapter card, speaker, liquid crystal display (LCD), or any other type of device for generating output to a human or machine.
[0049] Communication channels 232 (“COMM channel 232”) may interconnect each of the components 202, 222, 224, and 226 for inter-component communications (physically, communicatively, and / or operatively). In some examples, communication channel 232 may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.
[0050] Computing device 200 may include OS 234. OS 234 may control the operation of components of computing device 200. For example, OS 234 may facilitate the communication of modules 204-208 with processor 222, storage device 224, and communication units 226. In some examples, OS 234 may manage interactions between software applications (e.g., application module 206) and a user of computing device 200. OS 234 may have a kernel that facilitates interactions with underlying hardware of computing device 200 and provides a fully formed application space capable of executing a wide variety of software applications having secure partitions in which each of the software applications executes to perform various operations. In some examples, UI module 204 may be considered a component of OS 234.
[0051] In the example of FIG. 2, suggestion module 208 may include a hardware device having various hardware, firmware, and software components. However, FIG. 2 illustrates only one particular example of suggestion module 208, and many other examples of suggestion module 208 may be used in accordance with techniques of this disclosure. In some examples, components of suggestion module 208 may be located in a singular location. In other examples, one or more components of suggestion module 208 may be in different locations (e.g., connected via the network of FIG. 1). That is, in some examples suggestion module 208 may be part of a conventional computing device, while in other examples, suggestion module 208 may be part of a distributed or “cloud” computing system. Further, suggestion module 208 may be located in and executed by a computing system remote from and communicatively coupled to computing device 200. In such examples, computing device 200 may send and receive messages, data, or other otherwise exchange information with the remote computing system such that the remote computing system provides computing device 200 with the functionality of suggestion module 208. In some examples, suggestion module 208 may be functionally split between computing device 200 and one or more remote computing systems such that a portion of the functionality provided by suggestion module 208 is performed locally at computingdevice 200 and other portions of the functionality are performed by the one or more remote computing systems.
[0052] Suggestion module 208 may apply a machine learning model, such as ML model 238, to generate suggested messages relevant to a message thread. In some examples, ML model 238 can be or include one or more artificial neural networks (also referred to simply as neural networks). A neural network can include a group of connected nodes, which also can be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be referred to as “deep” networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer. The nodes of the neural network can be connected or non-fully connected.
[0053] ML model 238 can be or include one or more generative neural networks. A generative neural network may include a generator that generates new data points or samples in the form of images, text, audio, etc. The generator may be trained to create realistic, synthetic data samples (e.g., images) by analyzing the underlying patterns and structures in a given training dataset. For example, the generator may transform a lowdimensional input by applying a series of layers (e.g., convolutional layers for image generation) to produce a higher-dimensional output that resembles the target data.
[0054] In some examples, the generator may learn to produce better-quality samples by participating in an adversarial training process with another neural network, such as a discriminator. This type of neural network may be referred to as a generative adversarial network (GAN). In a GAN, the aim of the generator may be to create fake samples that the discriminator determines to be real, while the aim of the discriminator may be to correctly distinguish between real and fake samples. During training, the generator may receive feedback from the discriminator in the form of gradients, which may update parameters of the generator and in turn improve output quality.
[0055] In some examples, ML model 238 may be or include an autoencoder. The aim of an autoencoder may be to learn a representation (e.g., a lower- dimensional encoding) for a set of data, typically for the purpose of dimensionality reduction. For example, in some examples, an autoencoder can seek to encode the input data and then provide output data that reconstructs the input data from the encoding. In some examples, the autoencoder can include additional losses beyond reconstructing the input data. ML model 238 may be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines, deep belief networks, stacked autoencoders, etc. Any of the neuralnetworks described herein can be combined (e.g., stacked) to form more complex networks.
[0056] In some examples, ML model 238 can be or include one or more convolutional neural networks. In some examples, a convolutional neural network can include one or more convolutional layers that perform convolutions over input data using learned filters. Filters can also be referred to as kernels. Convolutional neural networks can be especially useful for vision problems such as when the input data includes imagery such as still images or video. It should be understood that the configurations of ML model 238 discussed herein are provided as examples and that other example configurations are contemplated by this disclosure.
[0057] In some examples, ML model 238 may be or include a diffusion model. A diffusion model may be trained to generate synthetic images by adding random noise to an image and then removing the noise to reconstruct the image. For example, the diffusion model may add Gaussian noise, salt and pepper noise, etc. The diffusion model may modify the amount of noise that is added to control the level of detail in the generated image. The diffusion model may apply the noise in multiple steps (e.g., according to a set of noise schedules).
[0058] ML model 238 may be trained to reverse the diffusion process by denoising the image, in this way transforming the noise into a high-quality synthetic image. For example, ML model 238 may apply a denoising neural network that gradually (e.g., in multiple steps) removes noise that ML model 238 predicts the forward diffusion process added. Diffusion models may be used for a variety of applications, including image generation, image editing, image restoration, etc. Thus, diffusion models may advantageously generate high-quality images with a wide range of variations. It should be understood that the examples of ML models described herein are not intended to limit the scope of this disclosure and that ML model 238 may be or include ML models in addition or alternative to the ML models described herein.
[0059] Computing device 200 may include model trainer 240 that trains (e.g., pre-train, fine-tune, etc.) ML model 238. Model trainer 240 may pre-train ML model 238 on a large and diverse dataset of images to perform a variety of computer vision tasks, such as image classification, image generation, etc. The images may be annotated with labels corresponding to a myriad of object categories (e.g., to facilitate supervised learning). Model trainer 240 may train ML model 238 to optimize an objective function. The objective function may be or include a loss function, such as cross-entropy loss, thatmeasures the discrepancy between predictions or other output by ML model 238 and the true labels of training data. For example, model trainer 240 may train ML model 238 to correctly classify images in the training data as often as possible (in turn training ML model 238 to classify a drawing by a user).
[0060] In some examples, model trainer 240 may fine-tune ML model 238 by using the feedback in the training process. For example, UI device 202 may receive a user input that selects feedback (e.g., thumbs up, thumbs down, etc.) relating to output (e.g., a generated image, a classification, etc.) by ML model 238. In some examples, the feedback may indicate whether the image classification was accurate or inaccurate, whether the generated image was high quality or low quality, etc. Model trainer 240 may obtain this feedback from the user of computing device 200 (as well as feedback from other users) and use the feedback for training. For example, model trainer 240 may convert the feedback into labeled data for supervised training. Additionally or alternatively, model trainer 240 may fine-tune ML model 238 by monitoring the relationship between the performance of ML model 238 and user feedback, and iterate the fine-tuning process as necessary (e.g., to receive more positive user feedback and less negative user feedback). In this way, the techniques of this disclosure may establish a feedback loop that continuously improves the quality of the output of ML model 238.
[0061] By leveraging ML model 238, suggestion module 208 may decrease the time and / or effort required to create an image worthy of sending to another user. That is, by generating images on behalf of a user of computing device 200, the techniques of this disclosure may reduce the likelihood that the user will need to manually edit an image, which may be deficient because of practical or other constraints.
[0062] FIG. 3 is a block diagram illustrating an example operation of a machine learning model in accordance with techniques of this disclosure. As shown in FIG. 3, an example training process 350 is used to train a ML model 338, which may be similar if not substantially similar to ML model 238. ML model 338 is trained on training data 351 that includes input data 352 that has labels 353. Training processes 350 is one example training process; other training processes may be used as well.
[0063] In some examples, ML model 338 may be trained by optimizing an objective function, such as objective function 355. For example, in some examples, objective function 355 may be or include a loss function, such as cross-entropy loss. One or more of various optimization techniques may be performed to optimize objective function 355. For example, the optimization technique(s) may minimize or maximize objective function355. Example optimization techniques include Hessian-based techniques and gradientbased techniques, such as, for example, coordinate descent; gradient descent (e.g., stochastic gradient descent); subgradient methods; etc. Other optimization techniques include black box optimization techniques and heuristics.
[0064] In some examples, backward propagation of errors may be used in conjunction with an optimization technique (e.g., gradient based techniques) to train ML model 338 (e.g., when a machine-learned model is a multi-layer model such as an artificial neural network). For example, an iterative cycle of propagation and model parameter (e.g., weights) update may be performed to train ML model 338. Example backpropagation techniques include truncated backpropagation through time, Levenberg- Marquardt backpropagation, etc.
[0065] In some examples, ML model 338 described herein may be trained using unsupervised learning techniques to increase the effectiveness of ML model 338. Unsupervised learning may include inferring a function to describe hidden structure from unlabeled data. For example, a classification or categorization may not be included in the data. Unsupervised learning techniques may be used to produce machine-learned models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks.
[0066] ML model 338 may be trained to increase the effectiveness of ML model 338 using semi-supervised techniques which combine aspects of supervised learning and unsupervised learning. ML model 338 may be trained or otherwise generated through evolutionary techniques or genetic algorithms. In some examples, ML model 338 described herein may be trained using reinforcement learning. In reinforcement learning, an agent (e.g., model) may take actions in an environment and learn to maximize rewards and / or minimize penalties that result from such actions. Reinforcement learning may differ from the supervised learning problem in that correct input / output pairs are not presented, nor sub-optimal actions explicitly corrected.
[0067] In some examples, ML model 338 described herein may include or otherwise be impacted by a number of hyperparameters, such as, for example, learning rate, number of layers, number of nodes in each layer, number of leaves in a tree, number of clusters; etc. Hyperparameters may affect model performance. Hyperparameters may be hand selected or may be automatically selected through application of techniques such as, for example, grid search; black box optimization techniques (e.g., Bayesian optimization, random search, etc.); gradient-based optimization; etc. Example techniques and / or tools forperforming automatic hyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE); etc.
[0068] In some examples, various techniques may be used to optimize and / or adapt the learning rate when the model is trained. Example techniques and / or tools for performing learning rate optimization or adaptation include Adagrad; Adaptive Moment Estimation (ADAM); Adadelta; RMSprop; etc. In some examples, transfer learning techniques may be used to provide an initial model from which to begin training of ML model 338 described herein.
[0069] In some examples, ML model 338 described herein may be included in different portions of computer-readable code on a computing device. In one example, ML model 338 may be included in a particular application or program and used (e.g., exclusively) by such particular application or program. Thus, in one example, a computing device may include a number of applications and one or more of such applications may contain its own respective machine learning library and machine-learned model(s).
[0070] In another example, ML model 338 described herein may be included in an operating system of a computing device (e.g., in a central intelligence layer of an operating system) and may be called or otherwise used by one or more applications that interact with the operating system. In some examples, each application may communicate with the central intelligence layer (and model(s) stored therein) using an application programming interface (API) (e.g., a common, public API across all applications).
[0071] In some examples, the central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized repository of data for the computing device. The central device data layer may communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some examples, the central device data layer may communicate with each device component using an API (e.g., a private API).
[0072] FIGS. 4A-4D are conceptual diagrams illustrating example GUIs for providing images modified by a ML model in accordance with techniques of this disclosure. The example of FIGS. 4A-4D are described below within the context of FIG 2.
[0073] As described above, ML model 238 may generate modified image 412 based on image 410 that suggestion module 208 provides to ML model 238. For example, a user may select a pre-existing image of the user, draw a set of elements 450 in relation to the pre-existing image of the user, and add a prompt 414 of ‘girlfriend.’ ML model 238 mayobtain image 410, including the pre-existing image of the user and set of elements 450, and output, for display as part of GUI 416B, modified image 412. As shown in FIG. 4B, ML model 238 may render a modified representation 452 of the portion of image 410 that includes set of elements 450.
[0074] In some examples, ML model 238 may generate modified image 412 by scaling a size of at least a subset of elements of set of elements 450 to a modified size in relation to image 410. For example, as shown in FIG. 4B, modified representation 452 may be larger in size than set of elements 450 to make the proportions of the user and modified representation 452 more similar (e.g., which may be necessary for photorealism). In other words, ML model 238 may scale the size of set of elements 450 based on the size of various portions of or elements in image 410.
[0075] For instance, in the example of FIGS. 4A-4B, ML model 238 may determine that the user’s head in the pre-existing image is associated with set of elements 450. For example, ML model 238 may perform image classification to classify at least a portion of the pre-existing image as a portrait of the user and pattern recognition to predict associations (e.g., contextual, logical, functional, etc.) between the portrait of the user and set of elements 450 classified as ‘girlfriend’. ML model 238 may in turn determine that the desired output is a realistic portrait of the user and a ‘girlfriend’ based on set of elements 450. Accordingly, ML model 238 may scale the size of set of elements 450 based on the size of the user’s head in the pre-existing image to create the appearance that the user and modified representation 452 are similar in size (which may typically be the case for a boyfriend and a girlfriend).
[0076] Additionally or alternatively, ML model 238 may generate modified image 412 by repositioning a position of at least a subset of elements of set of elements 450 to a modified position in relation to image 410. For example, as shown in FIG. 4B, modified representation 452 may be closer to the representation of the user in modified image 412 to suggest a more affectionate disposition, and the head portrayed in modified representation 452 may be tilted for a more fashionable look. As such, ML model 238 may reposition set of elements 450 based on the position of various portions of or elements in image 410.
[0077] For example, although not shown, the user may draw another set of elements intended to be a hat placed on the top of the user’s head in the pre-existing image. In addition to classifying the set of elements as a hat, ML model 238 may classify a portion of image 410 as the user’s head and predict that the set of elements corresponding to thehat is associated with the user’s head. ML model 238 may then generate a modified representation of the hat on the user’s head that is realistically positioned. For example, ML model 238 may align the position and orientation of the hat with the user’s head, place some parts of the hat in the foreground of modified image 412 and other parts of the hat in the background of modified image 412 (e.g., by determining the appropriate image layer for each element of the modified representation of the hat), etc.
[0078] In some examples, another user may collaborate with the user in creating a modified image in accordance with techniques of this disclosure. For example, as shown in FIG. 4C, the other user may use modified image 412 as a starting point for an updated image 454 that suggestion module 208 provides as input to ML model 238. The other user may provide an input that draws another set of elements 456 in relation to modified image 412 to create updated image 454. ML model 238 may then generate an updated modified image 458. As shown in FIG. 4D, ML model 238 may render a modified representation 460 of the portion of updated image 454 that includes set of elements 456. As demonstrated by the example of FIG. 4D, even without the other user providing a prompt, ML model 238 may classify set of elements 456 (e.g., as a ‘sun’) to render modified representation 460.
[0079] FIG. 5 is a flowchart illustrating example operations of an example computing device in accordance with one or more aspects of the present disclosure. Although the example operation of FIG. 5 is described as being performed by computing device 200 of FIG. 2 in the context of FIG. 1, in other examples some or all of the example operation may be performed by another computing device.
[0080] Suggestion module 208 may obtain (e.g., via UI module 204), a user input that draws at least a portion of image 110 (500). In some examples, the user of computing device 200 may draw the entirety of image 110 or use a pre-existing image and draw a set of elements in relation to the pre-existing image. In any case, with the explicit consent of the user, suggestion module 108 may provide image 110 to ML model 138, which ML model 138 may use as input to generate modified image 112 (502). For example, ML model 138 may generate modified image 112 based on at least a part of image 110. In some examples, the user may provide prompt 114 (which may be based on, e.g., a text input, a voice input, etc.) that classifies image 110. In those examples, ML model 138 may generate modified image 110 based on image 110 as well as prompt 114.
[0081] UI module 204 may output, for display by UI device 102, modified image 112 in area 118 of GUI 116B. The user may review modified image 112 and, if desirable,perform an action. For example, the user may provide an input that causes ML model 138 to generate variations of modified image 112 without changing the input (e.g., image 110) to ML model 138. In another example, the user may edit modified image 112 (e.g., by using area 118 to draw a set of elements), and ML model 138 may use the edited modified image as input to generate another modified image. In yet another example, the user may provide an input to send modified image 112 to another user, which application module 106 may do (e.g., by sending modified image 112 as its own message, by attaching modified image 112 to another message, etc.).
[0082] In some examples, ML model 238 may generate modified image 112 by scaling a size of at least a subset of elements of the set of elements drawn by the user in relation to image 110. For example, ML model 238 may scale the size of a generated hat larger or smaller (e.g., compared to the set of elements drawn by the user) to achieve more realistic proportions. Additionally or alternatively, ML model 238 may reposition a position of at least a subset of elements of the set of elements in relation to image 110. For example, ML model 238 may reposition the hat relative to a person in image 110 on which the user drew the set of elements that ML model 238 classified as a ‘hat’ (e.g., to make the hat appear to sit more naturally on the person’s head).
[0083] UI module 204 may output, for display by UI device 202, modified image 112 (504). For example, GUI 116 may include modified image 112 in area 118. In some examples, model trainer 240 may receive user input that selects feedback relating to modified image 112. Model trainer 240 may train ML model 238 using the feedback.
[0084] The user may send modified image 112 to another user (506). In some examples, the other user may collaborate with the user in creating a modified image in accordance with techniques of this disclosure. For example, the other user may use modified image 112 as a starting point for an updated image that suggestion module 208 provides as input to ML model 238. The other user may provide an input that draws another set of elements in relation to the updated image. ML model 238 may then generate an updated modified image based on the updated image (and, optionally, a prompt provided by the other user). If satisfied with the output, the other user may send the updated modified image to the user and / or other users for their amusement.
[0085] This disclosure includes various examples, such as the following examples.
[0086] Example 1 : A method includes obtaining, by a computing device, a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generating, by the computing device and based on at least theportion of the image, a modified image by at least applying a machine learning model; outputting, by the computing device and for display, the modified image; and providing, by the computing device and to a second user, a message that includes the modified image.
[0087] Example 2: The method of example 1, wherein generating the modified image includes rendering a photorealistic representation of at least the portion of the image.
[0088] Example 3 : The method of example 1 or 2, wherein the method further includes obtaining, by the computing device, a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
[0089] Example 4: The method of example 3, wherein generating the modified image includes scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the pre-existing image.
[0090] Example 5: The method of example 4, wherein scaling the size of at least the subset of elements includes determining, by the computing device, a classification of at least a portion of the pre-existing image; and predicting, by the computing device and based on the classification, one or more associations between the portion of the preexisting image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
[0091] Example 6: The method of any of examples 3 through 5, wherein generating the modified image includes repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
[0092] Example 7: The method of example 6, wherein repositioning the position of at least the subset of elements includes determining, by the computing device, a classification of at least a portion of the pre-existing image; and predicting, by the computing device and based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified position is based on a position of the portion of the pre-existing image.
[0093] Example 8: The method of any of examples 3 through 7, wherein the user input is a first user input, wherein the set of elements is a first set of elements, and wherein the method further includes obtaining, by the computing device, a second user input by the second user that draws a second set of elements in relation to the modified image; and generating, by the computing device, an updated modified image based on the second set of elements.
[0094] Example 9: The method of any of examples 1 through 8, wherein the method further includes obtaining, by the computing device, a prompt that describes one or more of the image or a modification to the image, and wherein generating the modified image is further based on the prompt.
[0095] Example 10: The method of example 9, wherein the prompt is based on one or more of a text input or a voice input provided by the first user.
[0096] Example 11 : The method of any of examples 1 through 10, wherein the method further includes receiving, by the computing device, a user input that selects feedback relating to the modified image; and providing, by the computing device, the machine learning model with the feedback.
[0097] Example 12: A computing device includes: one or more processors; a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generate, based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
[0098] Example 13: The computing device of example 12, wherein the instructions cause the one or more processors to generate the modified image by rendering a photorealistic representation of at least the portion of the image.
[0099] Example 14: The computing device of example 12 or 13, wherein the instructions cause the one or more processors to obtain a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
[0100] Example 15: The computing device of example 14, wherein the instructions cause the one or more processors to generate the modified image by scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the preexisting image.
[0101] Example 16: The computing device of example 15, wherein the instructions cause the one or more processors to scale the size of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the preexisting image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
[0102] Example 17: The computing device of any of examples 14 through 16, wherein the instructions cause the one or more processors to generate the modified image by repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
[0103] Example 18: The computing device of example 17, wherein the instructions cause the one or more processors to reposition the position of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the preexisting image and at least the subset of elements, wherein the modified position is based on a position of the portion of the pre-existing image.
[0104] Example 19: The computing device of any of examples 14 through 18, wherein the user input is a first user input, wherein the set of elements is a first set of elements, wherein the instructions further cause the one or more processors to obtain a second user input by the second user that draws a second set of elements in relation to the modified image; and generate an updated modified image based on the second set of elements.
[0105] Example 20: The computing device of any of examples 12 through 19, wherein the instructions further cause the one or more processors to obtain a prompt that describes one or more of the image or a modification to the image, and wherein generating the modified image is further based on the prompt.
[0106] Example 21 : The computing device of example 20, wherein the prompt is based on one or more of a text input or a voice input provided by the first user.
[0107] Example 22: The computing device of any of examples 12 through 21, wherein the instructions further cause the one or more processors to receive a user input that selects feedback relating to the modified image; and provide the machine learning model with the feedback.
[0108] Example 23: A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors of a first computing device, cause the one or more processors to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generate, based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
[0109] Example 24: The non-transitory computer-readable storage medium of example 23, wherein the instructions cause the one or more processors to generate the modified image by rendering a photorealistic representation of at least the portion of the image.
[0110] Example 25: The non-transitory computer-readable storage medium of example 23 or 24, wherein the instructions cause the one or more processors to obtain a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
[0111] Example 26: The non-transitory computer-readable storage medium of example25, wherein the instructions cause the one or more processors to generate the modified image by scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the pre-existing image.
[0112] Example 27: The non-transitory computer-readable storage medium of example26, wherein the instructions cause the one or more processors to scale the size of at least the subset of elements by determining a classification of at least a portion of the preexisting image; and predicting, based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
[0113] Example 28: The non-transitory computer-readable storage medium of any of examples 25 through 27, wherein the instructions cause the one or more processors to generate the modified image by repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
[0114] Example 29: The non-transitory computer-readable storage medium of example 28, wherein the instructions cause the one or more processors to reposition the position of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified position is based on a position of the portion of the pre-existing image.
[0115] Example 30: The non-transitory computer-readable storage medium of any of examples 25 through 29, wherein the user input is a first user input, wherein the set of elements is a first set of elements, wherein the instructions further cause the one or more processors to obtain a second user input by the second user that draws a second set of elements in relation to the modified image; and generate an updated modified image based on the second set of elements.
[0116] Example 31 : The non-transitory computer-readable storage medium of any of examples 23 through 30, wherein the instructions further cause the one or more processors to obtain a prompt that describes one or more of the image or a modification to the image, and wherein generating the modified image is further based on the prompt.
[0117] Example 32: The non-transitory computer-readable storage medium of example 31, wherein the prompt is based on one or more of a text input or a voice input provided by the first user.
[0118] Example 33: The non-transitory computer-readable storage medium of any of examples 23 through 32, wherein the instructions further cause the one or more processors to receive a user input that selects feedback relating to the modified image; and provide the machine learning model with the feedback.
[0119] Example 34: A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors of a first computing device, cause the one or more processors to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generate, based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
[0120] Example 35: The non-transitory computer-readable storage medium of example 34, wherein the instructions cause the one or more processors to generate the modified image by rendering a photorealistic representation of at least the portion of the image.
[0121] Example 36: The non-transitory computer-readable storage medium of example 34 or 35, wherein the instructions cause the one or more processors to obtain a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
[0122] Example 37: The non-transitory computer-readable storage medium of example36, wherein the instructions cause the one or more processors to generate the modified image by scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the pre-existing image.
[0123] Example 38: The non-transitory computer-readable storage medium of example37, wherein the instructions cause the one or more processors to scale the size of at least the subset of elements by determining a classification of at least a portion of the preexisting image; and predicting, based on the classification, one or more associationsbetween the portion of the pre-existing image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
[0124] Example 39: The non-transitory computer-readable storage medium of any of examples 36 through 38, wherein the instructions cause the one or more processors to generate the modified image by repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
[0125] Example 40: The non-transitory computer-readable storage medium of example 39, wherein the instructions cause the one or more processors to reposition the position of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified position is based on a position of the portion of the pre-existing image.
[0126] Example 41 : The non-transitory computer-readable storage medium of any of examples 36 through 40, wherein the user input is a first user input, wherein the set of elements is a first set of elements, wherein the instructions further cause the one or more processors to obtain a second user input by the second user that draws a second set of elements in relation to the modified image; and generate an updated modified image based on the second set of elements.
[0127] Example 42: The non-transitory computer-readable storage medium of any of examples 34 through 41, wherein the instructions further cause the one or more processors to obtain a prompt that describes one or more of the image or a modification to the image, and wherein generating the modified image is further based on the prompt.
[0128] Example 43: The non-transitory computer-readable storage medium of example 42, wherein the prompt is based on one or more of a text input or a voice input provided by the first user.
[0129] Example 44: The non-transitory computer-readable storage medium of any of examples 34 through 43, wherein the instructions further cause the one or more processors to receive a user input that selects feedback relating to the modified image; and provide the machine learning model with the feedback.
[0130] Example 34: A computer program product for image generation, the computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generate,based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
[0131] Example 35: The computer program product of example 34, wherein the one or more instructions cause the at least one processor to generate the modified image by rendering a photorealistic representation of at least the portion of the image.
[0132] Example 36: The computer program product of example 34 or 35, wherein the one or more instructions cause the at least one processor to obtain a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
[0133] Example 37: The computer program product of example 36, wherein the one or more instructions cause the at least one processor to generate the modified image by scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the pre-existing image.
[0134] Example 38: The computer program product of example 37, wherein the one or more instructions cause the at least one processor to scale the size of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
[0135] Example 39: The computer program product of any of examples 36 through 38, wherein the one or more instructions cause the at least one processor to generate the modified image by repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
[0136] Example 40: The computer program product of example 39, wherein the one or more instructions cause the at least one processor to reposition the position of at least the subset of elements by determining a classification of at least a portion of the pre-existing image; and predicting, based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified position is based on a position of the portion of the pre-existing image.
[0137] Example 41 : The computer program product of any of examples 36 through 40, wherein the user input is a first user input, wherein the set of elements is a first set of elements, wherein the one or more instructions further cause the at least one processor to obtain a second user input by the second user that draws a second set of elements inrelation to the modified image; and generate an updated modified image based on the second set of elements.
[0138] Example 42: The computer program product of any of examples 34 through 41, wherein the one or more instructions further cause the at least one processor to obtain a prompt that describes one or more of the image or a modification to the image, and wherein generating the modified image is further based on the prompt.
[0139] Example 43: The computer program product of example 42, wherein the prompt is based on one or more of a text input or a voice input provided by the first user.
[0140] Example 44: The computer program product of any of examples 34 through 43, wherein the one or more instructions further cause the at least one processor to receive a user input that selects feedback relating to the modified image; and provide the machine learning model with the feedback.
[0141] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
[0142] By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer- readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0143] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.
[0144] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.
[0145] Various examples have been described. These and other examples are within the scope of the following claims.
Claims
CLAIMS:
1. A method comprising: obtaining, by a computing device, a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generating, by the computing device and based on at least the portion of the image, a modified image by at least applying a machine learning model; outputting, by the computing device and for display, the modified image; and providing, by the computing device and to a second user, a message that includes the modified image.
2. The method of claim 1, wherein generating the modified image comprises rendering a photorealistic representation of at least the portion of the image.
3. The method of claim 1 or 2, further comprising: obtaining, by the computing device, a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image.
4. The method of claim 3, wherein generating the modified image comprises scaling a size of at least a subset of elements from the set of elements to a modified size in relation to the pre-existing image.
5. The method of claim 4, wherein scaling the size of at least the subset of elements comprises: determining, by the computing device, a classification of at least a portion of the pre-existing image; and predicting, by the computing device and based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified size is based on a size of the portion of the pre-existing image.
6. The method of any of claims 3 through 5, wherein generating the modified image comprises repositioning a position of at least a subset of elements from the set of elements to a modified position in relation to the pre-existing image.
7. The method of claim 6, wherein repositioning the position of at least the subset of elements comprises: determining, by the computing device, a classification of at least a portion of the pre-existing image; and predicting, by the computing device and based on the classification, one or more associations between the portion of the pre-existing image and at least the subset of elements, wherein the modified position is based on a position of the portion of the preexisting image.
8. The method of any of claims 3 through 7, wherein the user input is a first user input, wherein the set of elements is a first set of elements, and wherein the method further comprises: obtaining, by the computing device, a second user input by the second user that draws a second set of elements in relation to the modified image; and generating, by the computing device, an updated modified image based on the second set of elements.
9. The method of any of claims 1 through 8, wherein the method further comprises: obtaining, by the computing device, a prompt that describes one or more of the image or a modification to the image, wherein the prompt is based on one or more of a text input or a voice input provided by the first user, and wherein generating the modified image is further based on the prompt.
10. The method of any of claims 1 through 9, wherein the method further comprises: receiving, by the computing device, a user input that selects feedback relating to the modified image; and providing, by the computing device, the machine learning model with the feedback.
11. A computing device comprising:one or more processors; and a memory that stores instructions that, when executed by the one or more processors, cause the one or more processors to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with the computing device; generate, based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
12. The computing device of claim 11, wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to: obtain a selection of a pre-existing image, wherein the portion of the image includes a set of elements drawn in relation to the pre-existing image; determine a classification of at least a portion of the pre-existing image; predict, based on the classification, one or more associations between the portion of the pre-existing image and at least a subset of elements from the set of elements; scale, based on the one or more associations and a size of the portion of the preexisting image, a size of at least the subset of elements to a modified size in relation to the pre-existing image; and reposition, based on the one or more associations and a position of the portion of the pre-existing image, a position of at least the subset of elements to a modified position in relation to the pre-existing image.
13. The computing device of claim 11 or 12, wherein the user input is a first user input, wherein the set of elements is a first set of elements, and wherein the instructions that, when executed by the one or more processors, further cause the one or more processors to: obtain a second user input by the second user that draws a second set of elements in relation to the modified image; and generate an updated modified image based on the second set of elements.
14. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform any of the methods of claim 1-10.
15. A computer program product for image generation, the computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to: obtain a user input that draws at least a portion of an image, wherein the user input is by a first user associated with a computing device; generate, based on at least the portion of the image, a modified image by at least applying a machine learning model; output, for display, the modified image; and provide, to a second user, a message that includes the modified image.
