Text-to-image converter

Abstract

The invention provides a text-to-image converter. A method of modifying an image is described. The method may include the step of receiving, via a user interface of a service, a reference image and an input including text associated with the reference image. The method may also include the step of determining one or more features of the reference image by a trained machine learning (ML) model. The method may also include the step of modifying the reference image based on the determined one or more features and the received input by trained one or more potential diffusion models (LDMs). Any one or more of a background of the reference image, a region of the reference image, or a pattern of the reference image may be modified. The method may even further include the step of causing the modified image to be displayed via a user interface of the service.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 585,476, filed on September 26, 2023, entitled “Text to Image Changer,” U.S. Provisional Application No. 63 / 585,517, filed on September 26, 2023, entitled “Text ToScene Generation,” and U.S. Non-Provisional Patent Application No. 18 / 882,458, filed on September 11, 2024, the contents of which are incorporated by reference in their entirety. Technical Field

[0003] The present application generally relates to methods, systems, and computer program products for modifying images using artificial intelligence (AI). Background Art

[0004] Text-to-image models include advanced artificial intelligence (AI) systems designed to generate visual content from text descriptions. With the introduction of deep learning techniques, significant progress has been made in the field of text-to-image generation. Latent diffusion models (LDMs) have become a powerful tool for generating high-quality images from text descriptions, enabling a wide range of applications in digital art, design, and entertainment. These models leverage multiple capabilities of neural networks to understand and process visual and textual data to create images that are closely aligned with the given textual cues.

[0005] However, it remains a challenge to preserve the identity of the subject in the reference image and the true output while incorporating textual cues. Moreover, ensuring the privacy of the subject in the modified image also remains a challenge. Summary of the invention

[0006] The subject technology is an architecture for generating modified images that strikes a balance between preserving subject identity, following complex text cues, and maintaining visual quality.

[0007] One aspect of the subject technology relates to a method for modifying an image. The method may include the steps of receiving a reference image and input including text associated with the reference image via a user interface of a service. The reference image may include any one or more of an object or a person. The method may also include the steps of determining one or more features of the reference image through a trained machine learning (ML) model. The method may also include the steps of modifying the reference image based on the determined one or more features and the received input through one or more trained latent diffusion models (LDMs). Any one or more of the background of the reference image, the area of ​​the reference image, or the style of the reference image may be modified. The method may also include the steps of displaying the modified image via the user interface of the service.

[0008] Another aspect of the subject technology relates to a system. The system may include a processor and a non-volatile memory operably coupled to the processor, the non-volatile memory having instructions stored thereon. One of the multiple instructions executed may include: receiving a reference image and an input including text associated with the reference image via a user interface of a service. The reference image may include any one or more of an object or a person. Another executed instruction may include: determining one or more features of the reference image through a trained ML model. Another executed instruction may include: modifying the reference image based on the determined one or more features and the received input through one or more trained LDMs. Any one or more of the background of the reference image, the area of ​​the reference image, or the style of the reference image may be modified. Further executed instructions may include: causing the modified image to be displayed via the user interface of the service.

[0009] Yet another aspect of the subject technology relates to a computer readable medium (CRM). The CRM may include program instructions that implement a series of steps when executed by a processor. One of these instructions may include: receiving a reference image and input including text associated with the reference image via a user interface of a service. The reference image may include any one or more of an object or a person. Another of these instructions may include: determining one or more features of the reference image through a trained ML model. Even another of these instructions may include: modifying the reference image based on the determined one or more features and the received input through one or more trained LDMs. Any one or more of the background of the reference image, the area of ​​the reference image, or the style of the reference image may be modified. Yet another of these instructions may include: causing the modified image to be displayed via the user interface of the service.

[0010] Other advantages will be partially described in the following description, or can be learned through practice. These advantages will be realized and obtained through the elements and combinations specifically pointed out in the appended claims. It should be understood that, as claimed, the above general description and the following detailed description are only exemplary and explanatory, not restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The Summary of the Invention and the Detailed Description below may be further understood when read in conjunction with the accompanying drawings. Examples of the disclosed subject matter are shown in the drawings for the purpose of illustrating the disclosed subject matter; however, the disclosed subject matter is not limited to the specific methods, compositions, and apparatus disclosed. In addition, the drawings are not necessarily drawn to scale. In the drawings:

[0012] Figure 1 A schematic diagram of an exemplary network environment is shown in accordance with one or more example aspects of the subject technology.

[0013] Figure 2 A schematic diagram of an exemplary communication device according to one or more example aspects of the subject technology is shown.

[0014] Figure 3 An exemplary computing system in accordance with one or more example aspects of the subject technology is shown.

[0015] Figure 4 A machine learning and training model framework according to multiple example aspects of the present disclosure is shown.

[0016] Figure 5 A system for generating a modified image according to one or more example aspects of the subject technology is shown.

[0017] Figure 6 Modifying the style of an image by restyling a model according to one or more example aspects of the subject technology is shown.

[0018] Fig. 7A Modification of the background of an image via a background model is shown in accordance with one or more example aspects of the subject technology.

[0019] Figure 7B Another modification of an image via a background model is shown in accordance with one or more example aspects of the subject technology.

[0020] Figure 8 A flow chart illustrating an example process of the subject technology is shown.

[0021] These figures depict various examples for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative examples of the structures and methods shown herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0022] Some examples of the subject technology will be described more fully below with reference to the accompanying drawings, in which some but not all examples of the subject technology are shown. In fact, various examples of the subject technology can be embodied in a variety of different forms and should not be construed as being limited to the examples set forth herein. The same reference numerals always refer to the same elements.

[0023] As used herein, according to examples of the present disclosure, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that can be sent, received, and / or stored. In addition, the term "exemplary" as used herein is not intended to express any qualitative assessment, but merely to express an illustration of an example. Therefore, the use of any such terms should not be viewed as limiting the spirit and scope of the examples of the present disclosure.

[0024] As defined herein, "computer-readable storage media" may be distinguished from "computer-readable transmission media," which refers to non-transitory physical or tangible storage media (e.g., volatile storage devices or non-volatile storage devices) and "computer-readable transmission media," which refers to electromagnetic signals.

[0025] As used herein, an "application" may refer to a computer software package that can perform specific functions for a user and / or in some cases for another application or applications. One or more applications can be run using an operating system (OS) and other supporting programs. In some examples, one or more applications can request one or more services from other entities via an application programming interface (API) and communicate with these entities.

[0026] As described herein, a Metaverse may represent an immersive virtual space or world in which devices may be used in a network in which one or more social connections may, but need not, exist between users in the network or with environments in the Metaverse or world. A Metaverse or Metaverse network may be associated with a three-dimensional (3D) virtual world, an online game (e.g., a video game), one or more content items (e.g., images, videos, non-fungible tokens (NFTs)), and wherein the content items may, for example, be purchased with digital currency (e.g., cryptocurrency) and other suitable currencies. In some examples, a Metaverse or Metaverse network may allow for the generation and provision of an immersive virtual space in which remote users may socialize, collaborate, learn, shop, and / or participate in various other activities within the virtual space, including through the use of augmented reality / virtual reality / mixed reality.

[0027] As described herein, one or more resources or one or more external resources may refer to any entity or source accessible by such a program or system: the program or system may be run, executed, or implemented on a communication device and / or a network. Some examples of resources may include, but are not limited to, HyperText Markup Language (HTML) pages, web pages, images, videos, scripts, style sheets, other types of files (e.g., multimedia files) that can be accessed via a network (e.g., the Internet), and other files that can be stored locally and / or accessed by a communication device.

[0028] It should be understood that the methods and systems described herein are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is only used for the purpose of describing specific embodiments and is not intended to be limiting.

[0029] Exemplary System Architecture

[0030] Reference now Figure 1 , Figure 1 FIG. 1 is a block diagram of a system according to an exemplary embodiment. Figure 1As shown, system 100 may include one or more communication devices 105, 110, 115, and 120 and a network device 160. In addition, system 100 may include any suitable network, such as network 140. In some examples, network 140. In other examples, network 140 may be any suitable network capable of providing content and / or facilitating communication between entities within network 140 or between entities associated with network 140. As an example and not limitation, one or more portions of network 140 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 140 may include one or more networks 140.

[0031] Links 150 may connect communication devices 105, 110, 115, and 120 to network 140, network device 160, and / or to each other. The present disclosure contemplates any suitable links 150. In some exemplary embodiments, one or more links 150 may include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In some exemplary embodiments, one or more links 150 may each comprise an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link, or a combination of two or more such links 150. The links 150 are not necessarily all the same throughout the system 100. One or more first links may differ from one or more second links in one or more aspects.

[0032] In some exemplary embodiments, the communication device 105, 110, 115, 120 may be an electronic device that includes hardware, software, or embedded logic components, or a combination of two or more such components, and is capable of performing appropriate functions implemented or supported by the communication device 105, 110, 115, 120. By way of example and not limitation, the communication device 105, 110, 115, 120 may be a computer system, such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer (e.g., a smart tablet), an e-book reader, a Global Positioning System (GPS) device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, smart glasses, an augmented / virtual reality device, a smart watch, a charging case, or any other suitable electronic device, or any suitable combination thereof. The communication device 105, 110, 115, 120 may enable one or more users to access the network 140. The communication devices 105 , 110 , 115 , 120 may enable one or more users to communicate with other users at other communication devices 105 , 110 , 115 , 120 .

[0033] Other components of the system 100 can either directly access the network device 160 or can access the network device 160 via the network 140. By way of example and not limitation, the communication devices 105, 110, 115, 120 can use a web browser or a local application associated with the network device 160 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof) to either directly access the network device 160 or to access the network device 160 via the network 140. In certain exemplary embodiments, the network device 160 may include one or more servers 162. Each server 162 may be a single server, or a distributed server across multiple computers or multiple data centers. The server 162 may be of various types, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof. In certain exemplary embodiments, each server 162 may include hardware, software, or embedded logic components, or a combination of two or more such components, for performing the appropriate functions implemented or supported by the server 162. In certain exemplary embodiments, the network device 160 may include one or more data repositories 164. The data repository 164 may be used to store various types of information. In certain exemplary embodiments, the information stored in the data repository 164 may be organized according to a specific data structure. In certain exemplary embodiments, each data repository 164 may be a relational database, a columnar database, an associated database, or other suitable database. Although the present disclosure describes or illustrates a specific type of database, the present disclosure contemplates any suitable type of database. Certain exemplary embodiments may provide such an interface: the interface enables the communication device 105, 110, 115, 120 and / or another system (e.g., a third-party system) to manage, retrieve, modify, add or delete the information stored in the data repository 164.

[0034] Network device 160 may provide users of system 100 with the ability to communicate and interact with other users. In certain exemplary embodiments, network device 160 may provide users with the ability to take actions on various types of items or objects supported by network device 160. In certain exemplary embodiments, network device 160 may be able to link various entities. By way of example and not limitation, network device 160 may enable users to interact with each other and receive content from other systems (e.g., third-party systems) or other entities, or allow users to interact with these entities through an application programming interface (API) or other communication channels.

[0035] It should be pointed out that although Figure 1One network device 160 and four communication devices 105, 110, 115, and 120 are shown, but any suitable number of network devices 160 and communication devices 105, 110, 115, and 120 may be used without departing from the spirit and scope of the present disclosure. Figure 1 part of the system.

[0036] Exemplary Communication Devices

[0037] Figure 2 A block diagram of an exemplary hardware / software architecture of a communication device (e.g., user equipment (UE) 30) is shown. In some exemplary aspects, UE 30 can be any one of communication devices 105, 110, 115, 120. In some exemplary aspects, UE 30 can be a computer system, such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer (e.g., a smart tablet), an e-book reader, a GPS device, a camera, a personal digital assistant, a handheld electronic device, a cellular phone, a smart phone, smart glasses, an augmented / virtual reality device, a smart watch, a charging case, or any other suitable electronic device. Figure 2 As shown, UE 30 (also referred to herein as node 30) may include a processor 32, a non-removable memory 44, a removable memory 46, a speaker / microphone 38, a display, a touchpad and / or one or more user interfaces 42, a power supply 48, a GPS chipset 50, and other peripherals 52. In some exemplary aspects, the display, touchpad, and / or one or more user interfaces 42 may be referred to herein as a display / touchpad / one or more user interfaces 42. The display / touchpad / one or more user interfaces 42 may include a user interface that is capable of presenting one or more content items and / or collecting input of one or more user interactions / actions associated with the user interface. The power supply 48 may be capable of receiving power to supply power to the UE 30. For example, the power supply 48 may include an alternating current to direct current (AC to DC) converter that allows the power supply 48 to be connected / inserted into an AC electrical outlet and / or a Universal Serial Bus (USB) port for receiving power. The UE 30 may also include a camera 54. In an exemplary embodiment, the camera 54 may be a smart camera configured to sense images / videos that appear within one or more bounding boxes. The UE 30 may also include communication circuitry, such as a transceiver 34 and a transmit / receive element 36. It should be appreciated that the UE 30 may include any sub-combination of the aforementioned elements while remaining consistent with the embodiment.

[0038] The processor 32 may be a dedicated processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and a state machine, etc. In general, the processor 32 may execute computer executable instructions stored in a memory (e.g., a non-removable memory 44 and / or a removable memory 46) of the node 30 to perform various required functions of the node. For example, the processor 32 may perform signal encoding, data processing, power control, input / output processing, and / or any other function that enables the node 30 to operate in a wireless environment or a wired environment. The processor 32 may run an application layer program (e.g., a browser) and / or a radio access layer (RAN) program and / or other communication programs. The processor 32 may also perform security operations such as authentication, security key protocols, and / or encryption operations, for example, at the access layer and / or application layer. The non-removable memory 44 and / or the removable memory 46 may be computer-readable storage media. For example, the non-removable memory 44 may include non-transitory computer-readable storage media and transitory computer-readable storage media.

[0039] Processor 32 is coupled to its communication circuitry (eg, transceiver 34 and transmit / receive element 36). Processor 32 may control the communication circuitry by executing computer-executable instructions to enable node 30 to communicate with other nodes via a network to which it is connected.

[0040] The send / receive element 36 can be configured to send signals to other nodes or network devices, or receive signals from other nodes or network devices. For example, in an exemplary embodiment, the send / receive element 36 can be an antenna configured to send and / or receive radio frequency (RF) signals. The send / receive element 36 can support various networks and air interfaces, such as wireless local area network (WLAN), wireless personal area network (WPAN) and cellular, etc. In another exemplary embodiment, the send / receive element 36 can be configured to send and / or receive both RF signals and optical signals. It will be appreciated that the send / receive element 36 can be configured to send and / or receive any combination of wireless signals or wired signals.

[0041] The transceiver 34 may be configured to modulate signals to be transmitted by the transmit / receive element 36 and to demodulate signals received by the transmit / receive element 36. As described above, the node 30 may have multi-mode functionality. Thus, the transceiver 34 may include multiple transceivers for enabling the node 30 to communicate via multiple radio access technologies (RATs), such as universal terrestrial radio access (UTRA) and Institute of Electrical and Electronics Engineer (IEEE) (IEEE 802.11).

[0042] The processor 32 can access information from any type of suitable memory such as non-removable memory 44 and / or removable memory 46, and store data in these memories. For example, as described above, the processor 32 can store session context in its memory (e.g., non-removable memory 44 and / or removable memory 46). The non-removable memory 44 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 46 may include a subscriber identity module (SIM) card, a memory stick, and a secure digital (SD) memory card, etc. In other exemplary embodiments, the processor 32 can access information from a memory that is not physically located on the node 30 (e.g., located on a server or a home computer), and store data in the memory.

[0043] The processor 32 may receive power from a power source 48 and may be configured to distribute power to and / or control power to other components in the node 30. The power source 48 may be any suitable device for powering the node 30. For example, the power source 48 may include one or more dry cells (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc. The processor 32 may also be coupled to a GPS chipset 50, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the node 30. It will be appreciated that the node 30 may obtain location information by any suitable location determination method while being consistent with the exemplary embodiment.

[0044] Exemplary Computing System

[0045] Figure 3 is a block diagram of an exemplary computing system 300. In some exemplary embodiments, the network device 160 may be the computing system 300. The computing system 300 may include a computer or server and may be primarily controlled by computer-readable instructions, which may be in the form of software, regardless of where such software is stored or accessed, or regardless of how such software is stored or accessed. Such computer-readable instructions may be executed within a processor (e.g., a central processing unit (CPU) 91) to cause the computing system 300 to operate. In many workstations, servers, and personal computers, the central processing unit 91 may be implemented by a single-chip CPU known as a microprocessor. In other machines, the central processing unit 91 may include multiple processors. The coprocessor 81 may be an optional processor different from the main CPU 91 that performs additional functions or assists the CPU 91.

[0046] In operation, the CPU 91 fetches, decodes and executes instructions, and transmits information to and receives information from other resources via the computer's main data transmission path (system bus 80). Such a system bus connects the various components in the computing system 300 and defines the medium for data exchange. The system bus 80 typically includes data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus. An example of such a system bus 80 is a peripheral component interconnect (PCI) bus.

[0047] The memory coupled to the system bus 80 includes RAM 82 and ROM 93. Such memory may include circuits that allow information to be stored and retrieved. ROM 93 generally contains stored data that is not easily modified. The data stored in RAM 82 can be read or changed by CPU 91 or other hardware devices. Access to RAM 82 and / or ROM 93 can be controlled by a memory controller 92. The memory controller 92 can provide an address translation function that converts a virtual address into a physical address when an instruction is executed. The memory controller 92 can also provide a memory protection function that isolates multiple processes within the system and isolates system processes from user processes. Therefore, a program running in the first mode can only access memory mapped by the virtual address space of its own process; it cannot access memory within the virtual address space of another process unless memory sharing has been established between the processes.

[0048] In addition, the computing system 300 may include a peripheral device controller 83 that is responsible for transmitting instructions from the CPU 91 to peripheral devices, such as a printer 94 , a keyboard 84 , a mouse 95 , and a disk drive 85 .

[0049] The display 86 controlled by the display controller 96 can be used to display the visual output generated by the computing system 300. Such visual output can include text, graphics, animated graphics, and video. The display 86 can also include a user interface or be associated with a user interface. The user interface can be capable of presenting one or more content items and / or collecting input of one or more user interactions associated with the user interface. The display 86 can be implemented with a cathode ray tube (CRT)-based video display, a liquid crystal display (LCD)-based flat panel display, a gas plasma-based flat panel display, or a touch pad. The display controller 96 includes the electronic components required to generate the video signal sent to the display 86.

[0050] In addition, computing system 300 may include communication circuitry, such as network adapter 97, which may be used to connect computing system 300 to an external communication network (e.g., Figure 2 network 12) to enable the computing system 300 to communicate with other nodes of the network (eg, UE 30).

[0051] Figure 4 4 shows a machine learning and training model according to an example of the present disclosure. The machine learning framework 400 associated with the machine learning model can be hosted remotely. Alternatively, the machine learning framework 400 can be located at Figure 1 The machine learning model 410 may be communicatively coupled to training data 420 stored in a memory or database (e.g., ROM, RAM) such as a training database 422. In some examples, the machine learning model 410 may be associated with the operation of any one or more of the multiple systems / architectures depicted in subsequent figures of the present application. In some other examples, the machine learning model 410 may be associated with other operations. The machine learning model 410 may be implemented by one or more machine learning models and / or another device (e.g., a server and / or a computing system). In some embodiments, the machine learning model 410 may be a student model trained by a teacher model, and the teacher model may be included in the training database 422.

[0052] Image Modification

[0053] According to one aspect of the described subject technology, new methods for improving instruction and control of text-to-image modification are described in this application.

[0054] In general, a diffusion model can be a generative AI model that gradually transforms random noise into a structured output, such as an image or an audio clip, through a series of learning steps. The architecture of a diffusion model can be centered on a deep neural network that can use convolutional layers when processing images, or recursive layers for sequence data such as audio or text. The operation of the model can include two main stages: a forward diffusion process and a reverse generation process. In forward diffusion, the model can gradually add noise (e.g., Gaussian noise) to the data over a series of timesteps, thereby transforming the original data into pure noise. This is done in such a way that each step of adding noise is statistically tractable, allowing the model to learn how the data is corrupted at each timestep.

[0055] Diffusion models can be generated based on the concept of knowledge distillation, where the goal is to transfer knowledge from a complex model (teacher) to a simpler model (student). To train a student diffusion model through a distillation process, a well-trained, high-performance teacher model must first be generated or accessed. The teacher model may have learned how to effectively perform the task at hand, such as image generation, through a series of forward (e.g., adding noise) diffusion steps and reverse (e.g., removing noise) diffusion steps as described above. In some embodiments, the teacher model may be a pre-trained model.

[0056] According to an example embodiment of this aspect, a method for modifying an image using a machine learning model and a latent diffusion model is described. More specifically, the application provides a user-friendly interface for receiving a reference image and an input containing text associated with the reference image. The user interface of the service is intuitive and easy to use, allowing a user to modify an image quickly and efficiently.

[0057] The method may be particularly useful for modifying images used in marketing and advertising campaigns. The ability to modify the background, specific areas of an image, or the style of an image can help create more visually appealing and effective marketing materials. The method may also be used in a variety of other contexts, such as graphic design, web development, and social media marketing. For professionals in these fields, the ability to quickly and easily modify images may be a valuable tool.

[0058] According to an embodiment of this aspect, Figure 5An example system architecture 500 for generating a modified image is shown. The modified image may be generated by one or more inputs including a reference image, a textual cue, and a mask of an area or location within the reference image.

[0059] In such Figure 5 In an embodiment of the depicted aspect, a reference image 501 may be received via a user interface of a service (e.g., an application). An image description model 505 may be employed to digest the reference image 501 and understand its relevant features. The trained model employed in the architecture may be able to recognize and analyze various features of the reference image. These features of the reference image may include, for example, color, texture, shape, and other visual characteristics. In another example, the reference image may include an object, such as an ice cream cone. In yet another example, the reference image may include one or more people. According to an embodiment, it is contemplated that features associated with the identity of the person extracted by the trained model are retained in the modified image.

[0060] According to one embodiment, the service may send suggestions to the user via a user interface. The suggestions may provide the ability to: edit a local area of ​​a reference image (e.g., the foreground) through local editing; reshape the entire reference image; or modify the background of the reference image. For example, in the case of local editing, the suggestion from the service may include a mask or mark of a specific area in the image (e.g., a bounding box) with the option to be modified based on feedback from the user. An indication of feedback may be received via the user interface. An indication of feedback from the user may include, for example, clicking on an area of ​​the reference image so that a mask or mark appears on the clicked area. The indication may also include a verbal (e.g., speak to type) prompt or a keyboard-initiated prompt. The feedback may add or replace an object or move an object within the foreground of the reference image. For example, if the reference image includes a hand holding an ice cream cone (e.g., Figure 6 ), and the user provides feedback, either text-based or mask-based, to replace the ice cream cone with flowers, the service can process the request and apply the directional changes to the selected region of the reference image and output the modified image.

[0061] As described above, the suggestion may include a query for modifying the background or reshaping the reference image. As understood in the present application, the background may include everything in the image except the foreground including the object. Respectively, the reshaping may include global changes to the image pixels. For example, the reshaping may include, but is not limited to, aesthetic themes (e.g., warmth, nature, simplicity, candid) and cinematic perspective. The reshaping may also include changing a modern image to one that displays rural features.

[0062] Further references Figure 5, a hint / text 502 and / or a mask 503 may be received via a user interface of the service. The hint / text 502 is typically associated with a reference image. The input may provide additional information that may be used to modify the reference image.

[0063] In one example, a natural language processing (NLP) model 510 can evaluate the context of a prompt 502 and / or a mask 503. In other words, whether the context affects the editing of a local region of a reference image, the reshaping of a reference image, or the background of a reference image. Based on the evaluation, the NLP model 510 can send the prompt 502 and / or the mask 503 to one or more LDMs, including, for example, a local editing model 511, a reshaping model 512, or a background model 513. In one embodiment, it is contemplated that one or more models can be fine-tuned one or more times with the selected common image generation data. In doing so, a realistic output of the modified image can be obtained.

[0064] Next, the reference image 501 may be modified by one or more of the local editing model 511, the reshaping model 512, or the background model 513. The LDM used in the method is trained to modify the image based on features identified by the machine learning model. These models may modify the background, region, or style of the reference image to create a modified image that is more suitable for the user's needs.

[0065] In one embodiment, one or more of the plurality of models can enlarge a retained area of ​​a reference image, such as a person's face or an area where a new object is to be added through local editing. Changes can be applied to a specific area. This can preserve the high image resolution of the area. The edited area of ​​the image can be pasted back into the image.

[0066] According to another embodiment, the criteria for upscaling can include: the resolution of the full image is more than twice the resolution that the model can handle. Another criterion can include: the size of the area that needs to be edited. For example, if the subject is less than 5% of the image, it can be imagined that upscaling may be required. In doing so, the full image quality can be preserved.

[0067] like Figure 5 As further shown in FIG. 5 , a spatial feature output 520 representing a modified image 530 (eg, a target) may be obtained through a service. In one embodiment, the modified output image 530 may be sent to a user via a user interface.

[0068] Alternatively, before sending the modified image 530 through the user interface, the service can wait before sending the modified image. That is, a security check is performed on the spatial feature output 520 through the security check model 525. The security check model 525 can evaluate whether the output image sent to the user and / or the public via the user interface is safe based on predetermined criteria. The predetermined criteria may include any one or more of the following that may affect privacy, morality, or laws and regulations: degree of violence, degree of profanity, or degree of nudity, etc. In other words, the security check model 525 ensures that the reference image 530 meets privacy, morality, and legal considerations based on predetermined criteria before sending. The security check model can be continuously updated based on relevant updates to laws or regulations.

[0069] According to another embodiment, if the predetermined criteria are not met, the service sends a request to the user via the user interface to modify the reference image and / or prompt. The user is then prompted via the user interface to provide a new image or new input including the prompt. In doing so, the entire process described above is rerun before the modified image is output on the user interface.

[0070] According to yet another embodiment, a user can refine an area of ​​a reference image. For example, a user can click on an area of ​​a reference image or enter a mask and annotations such as "change the color to blue" in an area of ​​a reference image. The user can also provide a text prompt, for example, "add flowers to the man's hand". The mask can include a hollow box or a filled box or any shape surrounding an area, object, or person in the reference image. If the prompt provided by the user via the user interface is to perform a local edit on the foreground of the image, the trained local editing model will ensure that the foreground is semantically suitable for the new background and output a smooth and harmonious modified image. It is envisioned that the service can output one or more options where the user can enter more modifications to the local area, style, or background as needed. In some examples, users can choose to provide feedback on whether they are satisfied with the generation to help improve the learning rate of one or more models of the service.

[0071] According to another embodiment of this aspect, a user may reshape any image with any visual aesthetic based on text entered via the user interface. According to the present application, reshaping involves modification of all pixels of the image. In another example, again with respect to an image of a hand holding an ice cream cone, a user may enter a prompt or alternatively be provided with a suggestion via the service. One prompt may include reshaping the image. Figure 6 As shown, the reference image of a hand holding an ice cream cone can be reshaped based on a phrase agreed to by the user (e.g., "Van Gogh"). In other words, the image can be reimagined in any visual style in the described manner. As shown in the middle user interface, the service's trained model can take the received image and search the database for the hint "Van Gogh". Figure 6As further shown in FIG, the reference image is reshaped in the style of Van Gogh, and the modified image on the user interface (right side) shows a hand holding flowers.

[0072] According to yet another embodiment, a user may select a photo of themselves in an environment. For example, the environment may be their garage or somewhere outdoors. The user may provide a prompt through the user interface or agree to a suggestion provided by the service. The prompt may involve modifying the background of the image. The background of the image may be changed to any imaginable scene through the prompt. Fig. 7A As shown, the cue may be “being chased by a dinosaur.” A model trained for background modification may search a database based on the provided cue and apply the modification while preserving the person.

[0073] like Figure 7B As shown, the received prompt may be: "Showing up on the red carpet at the Oscars." The trained model may search the database and apply modifications to the reference image (e.g., a group of people in a garage). Here, it is contemplated that one or more of local editing, reshaping, or background modification may be performed in combination. Figure 7B As further depicted in , local editing can involve replacing a person's casual clothing with formal attire including tuxedos and dresses.

[0074] According to another embodiment of the present application, a method for modifying a reference image based on input received via a user interface of a service is described. The reference image may contain an object or a person. A trained ML model may be used to determine one or more features of the image.

[0075] Based on the determined features and the received input, the trained one or more LDMs may then be used to modify the image. The modification of the image may include a change to one or more of the background, local area, or style of the image. The modified image may be displayed via a user interface of the service.

[0076] In some embodiments, the input may include a mask or bounding box of an area in the reference image. Prior to receiving the input via the user interface, the service may send suggestions for masks or bounding boxes of specific areas in the image to be modified. Additionally or alternatively, the service may provide suggestions for affecting the background or style of the reference image. It is contemplated in the present application that the identity of objects or people in the reference image is preserved during the modification process.

[0077] According to an embodiment, the service can be configured to preserve image quality. For example, if the resolution of the reference image exceeds the capacity of the LDM, the service can determine the portion of the image to be preserved. The service can then edit the enlarged portion of the image before pasting it into the reference image. Thus, the quality of the output image is preserved.

[0078] According to another embodiment of the method, one or more of the reference image, the received input, or the modified image may be evaluated to see whether they meet predetermined criteria. The predetermined criteria may include, for example, violence, profanity, or nudity. The evaluation may be performed by a privacy LDM, etc. Based on the evaluation, a request to update the reference image or the received input may be sent to the user via a user interface. Subsequently, the user may receive the updated reference image or input via the user interface for further processing according to the above-mentioned protocols.

[0079] According to yet another embodiment, Figure 8 A flow chart showing an example process of the subject application is shown. In some embodiments, Figure 8 One or more of the process blocks in may be performed by a device. Figure 8 As shown, process 800 may include step 802, which involves receiving a reference image and an input having text associated with the reference image via a user interface of a service. The reference image may include any one or more of an object or a person. Process 800 may also include step 804, which involves determining one or more features of the reference image through a trained ML model. Process 800 may also include step 806, which involves modifying the reference image through one or more trained LDMs based on the determined features and the received input. Any one or more of the background of the reference image, the area of ​​the reference image, or the style of the reference image may be modified. Process 800 may even further include step 808, which involves causing the modified image to be displayed via the user interface of the service. Although Figure 8 Several example blocks of process 800 are shown, but in some implementations, process 800 may include Figure 8 Additional boxes different from those boxes depicted in Figure 8 Fewer boxes than those depicted in Figure 8 In addition or alternatively, two or more of the multiple blocks of process 800 may be executed in parallel.

[0080] Alternative Embodiments

[0081] The foregoing description of the embodiments has been presented for the purpose of illustration; the foregoing description is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. It can be appreciated by those skilled in the relevant art that many modifications and variations are possible in light of the above disclosure.

[0082] Some parts of this specification sheet describe multiple embodiments in terms of the application and symbolic representation of information operations. The technical personnel in the field of data processing usually use these application descriptions and representations to effectively convey the essence of their work to other technical personnel in the field. Although these operations are described functionally, computationally or logically, these operations should be understood to be implemented by computer programs or equivalent circuits, or microcodes, etc. In addition, without loss of generality, it is sometimes convenient to refer to the arrangement of these operations as components. The described operations and their associated components can be implemented in software, firmware, hardware or any combination thereof.

[0083] Any of the steps, operations or processes described herein may be performed or implemented individually or in combination with other devices using one or more hardware or software components. In one embodiment, the software component is implemented using a computer program product including a computer-readable medium, the computer-readable medium including computer program code, which may be executed by a computer processor to perform any or all of the steps, operations or processes described.

[0084] Embodiments may also relate to an apparatus for performing the operations herein. The apparatus may be specially constructed for the desired purpose, and / or the apparatus may include a computing device that is selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored in a non-transitory tangible computer-readable storage medium that can be coupled to a computer system bus, or in any type of medium suitable for storing electronic instructions. In addition, any computing system mentioned in this specification may include a single processor, or may be an architecture that employs a multi-processor design for increased computing power.

[0085] Embodiments may also be directed to a product produced by the computing process described herein. Such a product may include information obtained from the computing process, wherein the information is stored on a non-transitory tangible computer-readable storage medium and may include any example of a computer program product or other data combination described herein.

[0086] Finally, the language used in this specification is selected primarily for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. Therefore, it is intended that the scope of the patent right is not limited by the specific embodiment, but by any claims published in this application based thereon. Therefore, the disclosure of the embodiments is intended to illustrate, not limit, the scope of the patent right set forth in the appended claims.

Claims

1. A method comprising: receiving, via a user interface of a service, an input of a reference image and text associated with the reference image, wherein the reference image includes any one or more of an object or a person; Determining one or more features of the reference image by a trained machine learning (ML) model; Modifying the reference image based on the determined one or more features and the received input by the trained one or more latent diffusion models (LDMs), wherein any one or more of a background of the reference image, a region of the reference image, or a pattern of the reference image is modified; and The modified image is caused to be displayed via the user interface of the service.

2. The method according to claim 1, wherein: The input comprises a mask of a region in the image.

3. The method according to claim 1, further comprising: prior to receiving the input, sending, via the user interface, any one or more of: a suggestion of a mask for the region in the image; Or a suggestion affecting the background or the style of the reference image.

4. The method according to claim 1, wherein: The identity of the object or person is preserved.

5. The method according to claim 1, further comprising: detecting whether a resolution of the reference image exceeds a capacity of the one or more LDMs; determining a retained portion of the reference image; as well as The retained enlarged portion of the reference image is edited and the edited portion is pasted into the reference image.

6. The method according to claim 1, further comprising: Before the reference image is displayed on the user interface of the service, any one or more of the reference image, the received input, or the modified image is evaluated by a privacy LDM to determine whether predetermined criteria are met, wherein the predetermined criteria include any one or more of violence, profanity, or nudity.

7. The method according to claim 6, further comprising: Based on the evaluation, sending, via the user interface, a request to update any one or more of: the reference image or the received input; as well as Any one or more of the following is received via the user interface: an updated reference image or an updated input.

8. A system comprising: A non-transitory memory having instructions stored therein; as well as A processor operably coupled to the non-transitory memory and configured to execute the following instructions: receiving, via a user interface of a service, an input including a reference image and text associated with the reference image; Determining one or more features of the reference image by a trained machine learning (ML) model; Modifying the reference image based on the determined one or more features and the received input by the trained one or more latent diffusion models (LDMs), wherein any one or more of a background of the reference image, a region of the reference image, or a pattern of the reference image is modified; as well as The modified image is caused to be displayed via the user interface of the service.

9. The system according to claim 8, wherein: The input comprises a mask of a region in the image.

10. The system according to claim 8, wherein: The processor is also configured to execute the following instructions: Prior to receiving the input, any one or more of: a suggestion for a mask for the area in the image; or a suggestion to affect the background or the style of the reference image is sent via the user interface.

11. The system of claim 8, wherein: The reference image includes any one or more of an object or a person; and The identity of the object or person is preserved.

12. The system according to claim 8, wherein: The processor is also configured to execute the following instructions: detecting whether a resolution of the reference image exceeds a capacity of the one or more LDMs; determining a retained portion of the reference image; as well as The retained enlarged portion of the reference image is edited and the edited portion is pasted into the reference image.

13. The system according to claim 8, wherein: The processor is also configured to execute the following instructions: Before the reference image is displayed on the user interface of the service, whether any one or more of the reference image, the received input, or the modified image meets predetermined criteria is evaluated by a privacy LDM.

14. The system according to claim 13, wherein: The predetermined criteria include any one or more of violence, profanity or nudity.

15. The system of claim 13, wherein: The processor is also configured to execute the following instructions: Based on the evaluation, sending, via the user interface, a request to update any one or more of: the reference image or the received input; and Any one or more of the following is received via the user interface: an updated reference image or an updated input.

16. A computer-readable medium, the computer-readable medium comprising program instructions stored thereon, the program instructions, when executed by a processor, implementing: receiving, via a user interface of a service, a reference image and input including text associated with the reference image, wherein: The reference image includes any one or more of an object or a person; Determining one or more features of the reference image by a trained machine learning (ML) model; Modifying the reference image based on the determined one or more features and the received input by the trained one or more latent diffusion models (LDMs), wherein any one or more of a background of the reference image, a region of the reference image, or a pattern of the reference image is modified; as well as The modified image is caused to be displayed via the user interface of the service.

17. The computer-readable medium of claim 16, wherein: When the program instructions are executed by the processor, they also implement: Prior to receiving the input, any one or more of: a suggestion for a mask for the area in the image; or a suggestion to affect the background or the style of the reference image is sent via the user interface.

18. The computer-readable medium of claim 16, wherein: When the program instructions are executed by the processor, they also implement: detecting whether a resolution of the reference image exceeds a capacity of the one or more LDMs; determining a retained portion of the reference image; as well as The retained enlarged portion of the reference image is edited and the edited portion is pasted into the reference image.

19. The computer-readable medium of claim 16, wherein: When the program instructions are executed by the processor, they also implement: Before the reference image is displayed on the user interface of the service, any one or more of the reference image, the received input, or the modified image is evaluated by a privacy LDM to determine whether predetermined criteria are met, wherein the predetermined criteria include any one or more of violence, profanity, or nudity.

20. The computer-readable medium of claim 16, wherein: When the program instructions are executed by the processor, they also implement: Based on the evaluation, sending, via the user interface, a request to update any one or more of: the reference image or the received input; and Any one or more of the following is received via the user interface: an updated reference image or an updated input.

Images