Learning continuous control for 3D perceptual image generation on text-to-image diffusion model

Abstract

A method, apparatus, non-transitory computer-readable medium and system for image processing includes obtaining a textual cue describing an element and an attribute value of a contiguous attribute of the element; embedding a text prompt to obtain text embedding in the text embedding space; embedding the attribute value to obtain attribute embedding in the text embedding space; and generating a composite image based on the text embedding and the attribute embedding, wherein the composite image depicts a continuous attribute of the element based on the attribute value.

Description

Background Art

[0001] The following generally relates to image processing, and more specifically to image generation using machine learning models. Image processing refers to the use of computers to edit images using algorithms or processing networks. In some cases, image processing software can be used for various image processing tasks, such as image restoration, image detection, image synthesis, image editing, and image generation. For example, image generation includes using machine learning models to generate images based on text prompts.

[0002] In some cases, image generation models can be used to generate images that have the appearance of depth. That is, a two-dimensional (2D) image can have the appearance of three-dimensional (3D) properties such as depth or perspective. Summary of the Invention

[0003] Aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for image processing. Aspects of the present disclosure include a continuous control model trained to generate attribute embeddings based on input attributes. In one aspect, the input attributes include 3-dimensional characteristics of an element described by a text prompt. In some aspects, a text embedding model generates a text embedding based on the text prompt. In some aspects, the text embedding and the attribute embedding are combined as input embeddings to a text encoder to generate a guided embedding for an image generation model. The image generation model generates a synthetic image based on the guided features, wherein the synthetic image includes the element described by the text prompt and depicts the continuous attributes of the element based on the attribute values.

[0004] A method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining text hints describing elements and attribute values of continuous attributes of the elements; embedding the text hints to obtain text embeddings in a text embedding space; embedding the attribute values using a continuous control model to obtain attribute embeddings in a text embedding space; and generating a composite image based on the text embeddings and the attribute embeddings using an image generation model, wherein the composite image depicts the continuous attributes of the element based on the attribute values.

[0005] A method, apparatus, non-transitory computer-readable medium, and system for image processing include: initializing a machine learning model; obtaining a training set comprising a plurality of training images, each of which depicts an object having a plurality of values of a continuous attribute; training an image generation model using the training set to generate a composite image having the plurality of values of the continuous attribute; and training a continuous control model using the training set to generate an input to the image generation model corresponding to the continuous attribute.

[0006] An apparatus and system for image processing include: at least one processor; at least one memory storing instructions executable by the at least one processor; a continuous control model including parameters stored in the at least one memory and trained to embed attribute values of continuous attributes to obtain attribute embeddings in a text embedding space; and an image generation model including parameters stored in the at least one memory and trained to generate a composite image based on text embeddings and attribute embeddings of text prompts, wherein the composite image depicts the continuous attribute based on the attribute values. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An example of an image processing system according to aspects of the present disclosure is shown.

[0008] Figure 2 An example of a method for text-to-image generation according to aspects of the present disclosure is shown.

[0009] Figure 3 and Figure 4 An example of hybrid text-to-image generation according to aspects of the present disclosure is shown.

[0010] Figure 5 An example of image interpolation using attribute values according to aspects of the present disclosure is shown.

[0011] Figure 6 An example of a method for generating a composite image based on a text prompt according to aspects of the present disclosure is shown.

[0012] Figure 7 An example of an image processing apparatus according to aspects of the present disclosure is shown.

[0013] Figure 8 An example of a machine learning model according to aspects of the present disclosure is shown.

[0014] Figure 9 An example of a diffusion model according to aspects of the present disclosure is shown.

[0015] Figure 10 An example of a method for generating a composite image based on embedding according to aspects of the present disclosure is shown.

[0016] Figure 11 An example of a method for training a machine learning model according to aspects of the present disclosure is shown.

[0017] Figure 12 An example of a first stage training according to aspects of the present disclosure is shown.

[0018] Figure 13An example of second stage training according to aspects of the present disclosure is shown.

[0019] Figure 14 An example of a computing device according to aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Aspects of the present disclosure provide methods, non-transitory computer-readable media, devices, and systems for image processing. Aspects of the present disclosure include a continuous control model trained to generate attribute embeddings based on input attributes. In one aspect, the input attributes include 3-dimensional characteristics of an element described by a text prompt. In some aspects, a text embedding model generates a text embedding based on the text prompt. In some aspects, the text embedding and the attribute embedding are combined as input embeddings to a text encoder to generate a guided embedding for an image generation model. The image generation model generates a synthetic image based on the guided features, wherein the synthetic image includes the element described by the text prompt and depicts the continuous attributes of the element based on the attribute values.

[0021] According to some aspects, input attributes include 3D properties of the element described by the textual hint. For example, input attributes include the element's orientation, lighting direction, non-rigid shape transformations, scaling effects, or object pose. However, expressing these 3D properties in a textual description is challenging and laborious. According to some aspects, input attributes are integrated into user controls to allow users to easily control the values of the input attributes of the element to be generated in the composite image.

[0022] A subfield within image processing involves text-to-image generation. Text-to-image generative models are able to generate 2D images that closely resemble real photographs. However, the textual inputs used to generate these 2D images are inherently limited to high-level descriptions, which is a far cry from the detailed control of actual photography. In some cases, conventional models are trained using limited datasets, for example, limited descriptions of training images with precise object motion and camera parameters. In some cases, training images can be rendered at a fine-grained scale using predefined camera parameters, object motion, or non-rigid shape transformations. However, generating these training images can be inefficient and computationally expensive.

[0023] Conventional text-to-image generation models use large-scale text-image datasets to guide the image generation process. In some cases, conventional models utilize memory-efficient strategies to enhance performance by incorporating latent space diffusion methods. In some cases, conventional models use zero convolution to condition text and image data (e.g., depth maps, canny maps, and sketches). However, these conventional models cannot control the properties of the elements depicted in the image, such as lighting direction or object orientation.

[0024] In some cases, conventional models first generate an adjusted image based on text input and then use text instructions to perform the edits. For example, a user can edit the generated image by modifying the text prompt while preserving some aspects of the original image. However, due to the limited ability of users to describe the characteristics of the elements through text, conventional methods are limited in their ability to control the details of the elements depicted in the image. For example, describing an 11° angle change in the lighting direction in 3D space poses considerable challenges.

[0025] In some cases, conventional models can be trained on 3D data of an element (e.g., various viewpoints of a 3D rendering). Conventional models enable viewpoint editing given an image depicting the element. In some cases, conventional models rely on extensive 3D datasets to perform edits to the object orientation of the element depicted in the image. However, these edits are performed in a post-processing phase (e.g., on the generated image). Consequently, conventional models are inefficient at generating synthetic images with controllable 3D properties.

[0026] Thus, the present disclosure describes a method and system for generating a composite image depicting desired attributes of an element based on continuous attribute input including attribute values and textual prompts describing the element. In one aspect, the textual prompts and the continuous attribute input are combined and input into an image generation model to generate the composite image. In one aspect, the continuous attribute input includes 3D characteristics of the element, such as orientation, lighting direction, non-rigid shape transformations, object pose, scaling effects, etc. In one aspect, the continuous attribute input is integrated into a user control of a user interface that allows a user to easily input desired attributes to generate the composite image. In one aspect, the continuous attribute input includes variable inputs rather than specific values.

[0027] According to some aspects, a generative image model is trained using a two-stage training process. The first training stage trains the generative image model to learn the identities of components described by textual cues. For example, the generative image model generates a composite image based on training images depicting the component and a textual cue describing the component. The generative image model is then fine-tuned using a reconstruction loss calculated based on the training and composite images. By fine-tuning the generative image model using the reconstruction loss in the first stage, the generative image model can learn the identities of components described by the textual cues.

[0028] According to some aspects, the second training phase is to train the image generation model to learn the attributes of the element based on the continuous attribute input and the text prompt. For example, the continuous control model receives the continuous attribute input to generate an attribute embedding. The attribute embedding is combined with the text embedding of the text prompt to generate an input embedding for the image generation model. The image generation model generates a synthetic image based on the training image and the input embedding. In one aspect, the training image depicts the element and includes the attributes of the continuous attribute input. The image generation model is fine-tuned using a reconstruction loss calculated based on the training image and the synthetic image. By fine-tuning the image generation model using the reconstruction loss in the second phase, the image generation model can learn the attributes of the element from the continuous attribute input.

[0029] According to some aspects, a continuous control model is trained to generate attribute embeddings based on attribute values. For example, the continuous control model includes a multilayer perceptron (MLP). In one aspect, the MLP can receive a continuous input or inputs (e.g., attribute inputs) and generate a continuous output (e.g., attribute embeddings). Thus, the continuous control model can generate attribute embeddings based on the attribute values of the continuous attribute inputs, wherein the attribute embeddings are used as input to the image generation model.

[0030] According to some aspects, an image generation model is configured to generate synthetic images based on negative cues. In one aspect, the negative cues are used to guide the image generation model away from generating elements described by the negative cues. For example, the negative cues include elements depicted in training images. By using negative cues to generate synthetic images, the image generation model can be generalized to new, unseen data.

[0031] refer to Figure 1 and Figure 14 An example system of the present invention in image processing is provided. Figure 2-5 An example application of the present invention in image processing is provided. Figure 7-9 Provides details about the architecture of the image processing device. Figure 6 and Figure 10 Provides examples of procedures used for image processing. Figure 11-13 Provides a description of the example training process.

[0032] Embodiments of the present disclosure include systems and methods for improving conventional image generation models by generating more accurate synthetic images given target continuous properties, including 3D properties such as camera viewpoint and lighting conditions. For example, an image generation model can be trained to generate synthetic images with a target viewpoint based on a textual prompt describing an object and an input specifying the target properties. Improved accuracy can be achieved by training an attribute encoder (i.e., a continuous control model) that converts the continuous properties into a textual embedding space. Furthermore, by combining the output of the continuous control model with the textual prompt, the image generation model can generate synthetic images with the target continuous properties more efficiently (i.e., in a single generation process).

[0033] In some examples, a two-stage training process is used to train the image generation model. For example, the first training stage enables the image generation model to learn modified attributes of a specific object (i.e., pose or perspective). The second training stage enables the image generation model to learn continuous attributes of an element from continuous attribute input. Therefore, by using a two-stage training process to train image generation, the image generation model is able to separate attributes from object identity and thus enhance the quality of image generation.

[0034] Image processing

[0035] exist Figure 1-6 and Figure 10 In the present invention, a method, apparatus, non-transitory computer-readable medium, and system for image processing include: obtaining a text hint describing an element and an attribute value of a continuous attribute of the element; embedding the text hint to obtain a text embedding in a text embedding space; embedding the attribute value using a continuous control model to obtain an attribute embedding in the text embedding space; and generating a composite image based on the text embedding and the attribute embedding using an image generation model. In some cases, the composite image depicts the continuous attribute of the element based on the attribute value.

[0036] In some aspects, the continuous attribute comprises a 3-dimensional property of the element. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further comprise dividing the text prompt into a plurality of tokens. Some examples further comprise embedding each token of the plurality of tokens using a text embedding model. In some aspects, the text prompt comprises a random number token corresponding to the attribute value. In some aspects, the text prompt comprises a word corresponding to the continuous attribute.

[0037] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include encoding the text embedding and the attribute embedding to obtain guidance information for the image generation model. In some cases, a synthetic image is generated based on the guidance information. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include performing a diffusion process on the noise input to obtain the synthetic image.

[0038] In some aspects, an image generation model is trained using a training set comprising a plurality of training images, each of the plurality of training images depicting an object having a plurality of values for a continuous attribute. Some examples of the methods, apparatuses, non-transitory computer-readable media, and systems further include identifying a negative cue based on the object from the plurality of training images. In some cases, a composite image is generated based on the negative cue.

[0039] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining additional attribute values corresponding to the additional continuous attribute. In some cases, a composite image is generated to depict the additional attribute values.

[0040] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a plurality of attribute values of the continuous attribute. Some examples further include using an image generation model to generate a plurality of synthetic images based on the same random input and the plurality of attribute values.

[0041] Figure 1 An example of an image processing system according to aspects of the present disclosure is shown. The example shown includes a user 100, a user device 105, an image processing apparatus 110, a cloud 115, and a database 120. The image processing apparatus 110 is a reference Figure 7 Examples of corresponding elements described or including reference Figure 7 Aspects of the corresponding elements are described.

[0042] refer to Figure 1 , user 100 provides a textual hint describing elements and attributes to image processing apparatus 110 via user device 105 and cloud 115. For example, the textual hint states "photo of a horse". In some cases, the textual hint includes a random number token corresponding to an attribute. For example, the textual hint states "a photo of a horse".<V*> Photos”, including<V*> Represents a random number token. In some cases, one or more attributes are provided to the image processing device 110. For example, the attributes include 3D characteristics of an element. In some cases, for example, the attributes include a 3D orientation or 3D lighting of an element (such as a horse) described by a text prompt. In some cases, the attributes are integrated into user controls in a user interface, where the values of the attributes can be easily modified using the user controls. The image processing device 110 generates a composite image based on the text prompt and the attributes. For example, the composite image depicts a horse described by the text prompt and a 3D orientation and / or 3D lighting based on the attributes. In some cases, the image processing device 110 displays the composite image to the user 100 via the user device 105 and the cloud 115.

[0043] User device 105 can be a personal computer, laptop computer, mainframe computer, palmtop computer, personal assistant, mobile device or any other suitable processing device. In some examples, user device 105 includes software that incorporates an image processing application. In some examples, the image processing application on user device 105 can include the functionality of image processing device 110.

[0044] The user interface can enable user 100 to interact with user device 105. In some embodiments, the user interface can include an audio device, such as an external speaker system, an external display device, such as a display screen, or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the user interface can be represented by code, where the code is sent to user device 105 and rendered locally by a browser. Figure 2 The process of using the image processing apparatus 110 is further described.

[0045] The image processing device 110 is a reference Figure 7 Examples of corresponding elements described or including reference Figure 7 According to some aspects, the image processing apparatus 110 includes a computer-implemented network comprising a machine learning model, a text embedding model, a continuous control model, a text encoder, and an image generation model. The image processing apparatus 110 further includes a processor unit, a memory unit, an I / O module, a training component, and a data preparation component. In some cases, the data preparation component includes training the image generation model. In some embodiments, the image processing apparatus 110 further includes a communication interface, a user interface component, and a bus, as described in reference Figure 14 Furthermore, the image processing apparatus 110 communicates with the user device 105 and the database 120 via the cloud 115. Figure 2 More details are provided regarding the operation of the image processing apparatus 110 .

[0046] In some cases, image processing apparatus 110 is implemented on a server. The server provides one or more functions to the user through one or more links in various networks. In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling the various aspects of the server. In some cases, the server uses a microprocessor and protocol to exchange data with other devices / users on one or more networks via Hypertext Delivery Protocol (HTTP) and Simple Mail Delivery Protocol (SMTP), but other protocols such as File Delivery Protocol (FTP) and Simple Network Management Protocol (SNMP) can also be used. In some cases, the server is configured to send and receive files (e.g., for displaying web pages) in Hypertext Markup Language (HTML) format. In various embodiments, the server includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0047] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources (such as data storage and computing power). In some examples, cloud 115 provides resources without the need for active management by users (e.g., user 100). The term cloud is sometimes used to describe a data center that is available to many users via the Internet. Some large cloud networks have functionality distributed across multiple locations from a central server. If a server has a direct or close connection to a user, the server is designated as an edge server. In some cases, cloud 115 is limited to a single organization. In other examples, cloud 115 is available to many organizations. In one example, cloud 115 includes a multi-layer communication network that includes multiple edge routers and core routers. In another example, cloud 115 is based on a local aggregation of switches in a single physical location.

[0048] According to some aspects, database 120 stores training data (or training sets) including a plurality of training images, wherein the plurality of training images depict objects having a plurality of values for a continuous attribute. Database 120 is an organized collection of data. For example, database 120 stores data in a specified format called a schema. Database 120 can be configured as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the storage and processing of data in database 120. In some cases, a user (e.g., user 100) interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0049] Figure 2An example of a method 200 for text-to-image generation according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0050] refer to Figure 2 , users (e.g. reference Figure 1 The user described) to the image processing device (eg, reference Figure 1 and Figure 7 The image processing device described herein provides text hints and attributes. For example, the text hint states "horse's<V*> Photo". In some cases, the random number token<V*> Added to text prompts by machine learning models. In some cases, random number tokens<V*> Not displayed to the user and processed by the machine learning model. In some cases, the random number token<V*> The textual hint is replaced by attributes. The attributes describe the 3D characteristics of the object described by the textual hint. For example, the attributes describe orientation, lighting, pose, and scale. The image processing device generates a text embedding based on the textual hint and an attribute embedding based on the attributes. In some cases, the text embedding and the attribute embedding are combined to generate an input embedding for a text encoder of a machine learning model. The text encoder generates a guided embedding based on the input embedding to guide the image generation model to generate a synthetic image. The synthetic image depicts the horse described by the textual hint and the 3D characteristics based on the attributes.

[0051] At operation 205, the system provides text hints and attributes. In some cases, the operation of this step involves reference to Figure 1 The user described may be referred to as Figure 1 For example, the user performs the following operations on the user device (e.g., reference Figure 1 A user interface provided on a user device (described in the example embodiment) provides the image processing apparatus with a textual prompt "Photo of a Horse" and attributes. In some cases, for example, the attributes are integrated into user controls where they can be easily modified by the user. In some cases, the attributes include 3D characteristics of the element described by the textual prompt (e.g., orientation, pose, and lighting).

[0052] At operation 210, the system embeds the attribute to obtain the attribute embedding. In some cases, the operation of this step involves as reference Figure 1 and Figure 7 The image processing apparatus described may be configured as shown in FIG. Figure 1 and Figure 7 In some cases, the operation of this step involves the image processing apparatus described in reference Figure 7 、 Figure 8 and Figure 13 The continuous control model described can be obtained by referring to Figure 7 、 Figure 8 and Figure 13 The continuous control model described herein performs. In some cases, for example, the continuous control model includes a multi-layer perceptron (MLP) trained to embed attributes to obtain attribute embeddings. In some cases, the machine learning model embeds text cues to obtain text embeddings. In some cases, the attribute embeddings are added to regions of the sequence of text embeddings.

[0053] At operation 215, the system encodes the text hint and attribute embedding to obtain guidance information. In some cases, the operation of this step involves Figure 1 and Figure 7 The image processing device may be configured as shown in FIG. Figure 1 and Figure 7 In some cases, the operation of this step involves the following steps: Figure 7-9 The text encoder described can also be used as reference Figure 7-9 The described text encoder performs. In some embodiments, the text encoder receives text embeddings (including attribute embeddings) to generate guided embeddings (e.g., guidance information). The guided embeddings are used to guide the image generation model to generate a synthetic image.

[0054] At operation 220, the system generates a composite image based on the guidance information. In some cases, the operation of this step involves Figure 1 and Figure 7 The image processing apparatus described may be configured as shown in FIG. Figure 1 and Figure 7 In some cases, the operation of this step involves performing the image processing described in the reference Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The image generation model described can also be used as reference Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The image generation model described herein is executed. In some embodiments, the image generation model receives a noise input (e.g., a noise map) and a guided embedding to generate a composite image. In some cases, the composite image includes elements described by a text prompt and attributes from a user input. In some cases, the composite image is displayed on a user device via an image processing device and a user interface of the cloud.

[0055] Figure 3An example of hybrid text-to-image generation according to aspects of the present disclosure is shown. The example shown includes a text prompt 300, attributes 305, a machine learning model 310, and a composite image 315. In some embodiments, the example shown is integrated into a user interface.

[0056] refer to Figure 3 , machine learning model 310 receives text prompt 300 and attributes 305 to generate composite image 315. For example, text prompt 300 states "Photo of a race car on a road." In some cases, text prompt 300 includes a placeholder for attribute 305. For example, attribute 305 can be placed at the beginning, middle, or end of text prompt 300. Attribute 305 includes 3-dimensional characteristics of the element described by text prompt 300. For example, attribute 305 includes lighting direction. In some embodiments, attribute 305 is integrated into a user control in a user interface, where a user can easily modify the attribute value of attribute 305. For example, user controls may include scroll bars, buttons, text input controls, drop-down lists, sliders, progress bars, switches, tabs, drop-down menus, etc.

[0057] In some cases, composite image 315 includes one or more composite images that depict the element described by text prompt 300 and 3-dimensional characteristics from attributes 305. For example, the composite image 315 on the left depicts a car (i.e., the element described by text prompt 300) and a lighting direction specified by, for example, a user (i.e., a 3-dimensional characteristic from attributes 305). In some cases, the lighting direction is depicted by the car's shadow. For example, the lighting direction shows that the light source is located at the upper right corner of the element. As a result, the shadow is reflected on the other side of the light source, for example, at the lower left corner of the element. The composite image 315 in the middle depicts a car (e.g., a different car) and a second lighting direction. The composite image 315 on the right depicts a car (e.g., a different car) and a third lighting direction.

[0058] Text prompt 300 is a reference Figure 4 、 Figure 5 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 5 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 Attribute 305 is a reference to the corresponding element. Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 8 、 Figure 12 and Figure 13The machine learning model 310 is a reference to the corresponding elements of the description. Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The composite image 315 is a reference to the corresponding elements of the Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Aspects of the corresponding elements are described.

[0059] Figure 4 An example of hybrid text-to-image generation according to aspects of the present disclosure is shown. The example shown includes a text prompt 400, a first attribute 405, a second attribute 410, a machine learning model 415, and a composite image 420. In some embodiments, the example shown is integrated into a user interface.

[0060] refer to Figure 4 , machine learning model 415 receives text prompt 400, first attribute 405, and second attribute 410 to generate composite image 420. For example, text prompt 400 states "photo of an owl." In some cases, text prompt 400 includes placeholders for first attribute 405 and second attribute 410. For example, first attribute 405 and second attribute 410 can be placed at the beginning, middle, or end of text prompt 400. In some embodiments, first attribute 405 and second attribute 410 are integrated into a single user control. In some embodiments, first attribute 405 and second attribute 410 are integrated into two different user controls. First attribute 405 includes a wing pose of an element. Second attribute 410 includes a 3-dimensional orientation of an element.

[0061] In some cases, composite image 420 includes one or more composite images that depict the element described by text prompt 400 and 3-dimensional characteristics from first attribute 405 and second attribute 410. For example, composite image 420 on the left depicts an owl (i.e., the element described by text prompt 400), a wing pose (i.e., a 3-dimensional characteristic from first attribute 405), and a 3-dimensional orientation specified by, for example, a user (i.e., a 3-dimensional characteristic from second attribute 410). The left, center, and right composite images 420 depict different combinations of first attribute 405 and second attribute 410. For example, composite image 420 on the left depicts a first wing pose and a first 3-dimensional orientation, composite image 420 in the center depicts a second wing pose and a second 3-dimensional orientation, and composite image 420 on the right depicts a third wing pose and a third 3-dimensional orientation. In some cases, machine learning model 415 can generate composite images 420 with a fixed first attribute 405 and a changing second attribute 410, or vice versa. For example, the left composite image 420 may depict a first wing pose and a first 3-dimensional orientation, the middle composite image 420 may depict the first wing pose and a second 3-dimensional orientation, and the right composite image 420 may depict the first wing pose and a third 3-dimensional orientation.

[0062] Text prompt 400 is a reference Figure 3 、 Figure 5 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 5 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 The machine learning model 415 is a reference to the corresponding elements of the description. Figure 3 、 Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The composite image 420 is a reference to the corresponding elements of the Figure 3 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 8 、 Figure 12 and Figure 13Aspects of the corresponding elements are described.

[0063] Figure 5 An example of image interpolation using attribute values according to aspects of the present disclosure is shown. The example shown includes a text prompt 500, a first attribute value 505, a second attribute value 510, a machine learning model 515, a first composite image 520, an intermediate composite image 525, and a final composite image 530. In some embodiments, the example shown is integrated into a user interface.

[0064] refer to Figure 5 , the machine learning model 515 receives the text prompt 500, the first attribute value 505, and the second attribute value 510 to generate multiple composite images (e.g., the first composite image 520, the intermediate composite image 525, and the final composite image 530). For example, the text prompt 500 states "Photograph of an eagle flying in the woods." In some embodiments, the first attribute value 505 and the second attribute value 510 are part of the same attribute integrated into a single user control. For example, the first attribute value 505 includes first information of the attribute (e.g., wing pose). The second attribute value 510 includes second information of the same attribute. For example, the first attribute value 505 and the second attribute value 510 represent the shape / position of the wing pose of an owl (e.g., the element described by the text prompt 500). For example, the first attribute value 505 represents a wing pose in a downward direction, and the second attribute value 510 represents a wing pose in an upward direction.

[0065] Machine learning model 515 generates a first composite image 520 and a final composite image 530 based on first attribute value 505 and second attribute value 510, respectively. Furthermore, machine learning model 515 generates an intermediate composite image 525 by interpolating wing pose information based on first attribute value 505 and second attribute value 510. For example, machine learning model 515 may generate a plurality of intermediate attribute values based on first attribute value 505 and second attribute value 510, wherein intermediate composite images 525 are generated based on the plurality of intermediate attribute values. In one aspect, each of the plurality of composite images (e.g., first composite image 520, intermediate composite image 525, and final composite image 530) depicts the same owl (e.g., the element described by text prompt 500), but with varying wing poses. In one aspect, the visual changes in wing poses are continuous and dynamic.

[0066] Text prompt 500 is a reference Figure 3 、 Figure 4 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 8 、 Figure 9 、 Figure 12 and Figure 13 The machine learning model 515 is a reference to the corresponding elements of the description. Figure 3 、 Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The first composite image 520, the intermediate composite image 525 and the final composite image 530 are reference images. Figure 3 、 Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Examples of composite images described or including reference Figure 3 、 Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Describe aspects of the composite image.

[0067] Figure 6 An example of a method 600 for generating a composite image based on a text prompt according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed in conjunction with other operations.

[0068] At operation 605, the system obtains a textual hint describing the element and attribute values of the continuous attributes of the element. In some cases, the operation of this step involves Figure 3 、 Figure 5 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The machine learning model described can be used as reference Figure 3 、 Figure 5 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13In some cases, a user provides a text prompt and an attribute value to the machine learning model of the image generation system. For example, the text prompt describes a dog and the attribute value includes attribute information of the dog, such as orientation.

[0069] Continuous properties, such as the orientation of an object or the apparent camera view of a scene, may be difficult to describe accurately using text. For example, they may include one or more numerical parameters, such as distances and angles (e.g., the distance between an object and a viewpoint, or an angle describing the relationship between an object and a light source). Therefore, these parameters may be provided separately from the text. For example, a user may move one or more sliders or other UI elements to adjust the object orientation, view, pose, or lighting position. Properties may be described using one or more continuous variables, such as 3D position coordinates, Euler angles, or orientation angles (such as yaw, pitch, and roll).

[0070] In some aspects, for example, a text prompt can be short, long, or complex. For example, a text prompt can describe one or more elements or objects. In some cases, an element includes an object (e.g., a chair, a table, or a book), a feature (e.g., a shadow, lighting, or color), a category (e.g., a photo, an image, or a sketch), etc. In some cases, an attribute value can include information that can be understood by a computing device. For example, an attribute value can include a value, a natural language, a shape, a coordinate, a data point, etc. In some cases, a continuous attribute includes a 3D property of an element. For example, a continuous attribute can include a 3D orientation, a lighting direction, a non-rigid shape transformation, an object posture, a scaling effect, etc. In some cases, a continuous attribute can include a 2D property of an element, such as an edge, a contour, a color intensity, etc. In one aspect, a continuous attribute includes a variable 3D property of the element described by the text prompt. For example, a variable 3D property includes a range of values or a value that can be changed.

[0071] At operation 610, the system embeds the text hint to obtain a text embedding in the text embedding space. In some cases, the operation of this step involves as described in reference Figure 7 and Figure 8 The text embedding model described can also be used as reference Figure 7 and Figure 8 In some cases, the text cue is divided into a plurality of tokens, wherein the text embedding is based on the plurality of tokens. In some cases, the text cue includes random number tokens corresponding to continuous attributes. In some cases, the text cue includes words corresponding to the continuous attributes. In some cases, the text embedding can be represented in the form of a table, wherein each cell of the text embedding represents a word token for the text cue.

[0072] According to some aspects, a text embedding model generates text embeddings based on textual cues. In one aspect, an embedding (such as a text embedding, an image embedding, or a guided embedding) refers to a numerical representation of a word, sentence, document, or image in a vector space. The embedding is used to encode the semantic meaning, relationships, and context of a word, sentence, document, or image, wherein the encoding can be processed by a machine learning model.

[0073] In one aspect, an embedding space refers to a space formed by vectors (e.g., embeddings) representing data points (e.g., text prompts). Vector space provides a framework for representing and manipulating data (in the form of vectors), calculating distances between vectors, and transforming input data to implement complex relationships. The dimensionality of a vector space is determined by the number of features in the feature vector. For example, if each data point has three features (e.g., length, width, and height), the vector space is three-dimensional. In some cases, a joint vector space includes a high-dimensional vector space and a low-dimensional vector space. In some cases, image embeddings are in a high-dimensional vector space, and text embeddings are in a low-dimensional vector space.

[0074] In one aspect, a text token or token refers to a meaningful unit of natural language. Tokenization is the process of breaking down a sequence of text into individual tokens. In some cases, a token can be a word, a subword, or a character. For example, a word token represents each individual word in the text. Subword tokens represent a further breakdown of a word. For example, if the word is "individual," subword tokens could be "indi" and "vidual." Character tokenization is the breakdown of a word in the text into its individual characters. For example, the character tokens for the word "token" are "t," "o," "k," "e," and "n." Tokenization allows machine learning models to understand, process, analyze, or classify data containing text.

[0075] In one aspect, a nonce token refers to a placeholder token that can be added to text or a text prompt. For example, a nonce token can be represented by a symbol, a shape, or a letter. The nonce token can be placed in a specific location in a text prompt. The value of the nonce token can be a variable rather than a specific value.

[0076] At operation 615, the system embeds the attribute values using the continuous control model to obtain attribute embeddings in the text embedding space. In some cases, the operation of this step involves referring to Figure 7 、 Figure 8 and Figure 13 The continuous control model described can be obtained by referring to Figure 7 、 Figure 8 and Figure 13The continuous control model described herein is executed. For example, the continuous control model includes a multilayer perceptron (MLP), wherein the MLP is capable of receiving continuous input (e.g., attribute values) and generating continuous output (e.g., attribute embeddings). In some cases, the attribute embeddings of the attribute values are combined with the text embeddings of the text cues as input to the image generation model. For example, the attribute embeddings are added to the regions of the sequence of text embeddings.

[0077] In some examples, attribute embeddings can be used as tokens and combined with tokens from text in the same embedding space. Although attributes may be difficult to describe using words, the text embedding space may have sufficient parameters to accurately represent them. In some cases, these tokens are further combined. For example, transformers can be used to encode contextual information within individual tokens, or to generate individual embeddings that represent the combined text and attribute embeddings. The combined text and attribute embeddings can be used as input to an image generation model.

[0078] At operation 620, the system generates a composite image based on the text embedding and the attribute embedding using the image generation model, wherein the composite image depicts the continuous attributes of the element based on the attribute values. In some cases, the operation of this step involves as described in reference Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The image generation model described can also be used as reference Figure 4 、 Figure 7 、 Figure 8 、 Figure 12 and Figure 13 The image generation model described herein performs. For example, the image generation model receives text embeddings (including attribute embeddings) and noise inputs (e.g., noise maps) to generate synthetic images. In some cases, the image generation model includes a diffusion model. The diffusion model is a reference Figure 9 Examples of corresponding elements described or including reference Figure 9 Aspects of the corresponding elements are described.

[0079] System Architecture

[0080] exist Figure 1 、 Figure 7-9 and Figure 14In the present invention, an apparatus and system for image processing include: at least one processor; at least one memory storing instructions executable by the at least one processor; a continuous control model including parameters stored in the at least one memory and trained to embed attribute values of continuous attributes to obtain attribute embeddings in a text embedding space; and an image generation model including parameters stored in the at least one memory and trained to generate a composite image based on text embeddings and attribute embeddings of text prompts, wherein the composite image depicts the continuous attribute based on the attribute values.

[0081] Some examples of the apparatus and system further include a text encoder comprising parameters stored in at least one memory and configured to encode text embeddings and attribute embeddings to obtain guidance information for the image generation model. In some aspects, the continuous control model comprises a multilayer perceptron (MLP). In some aspects, the image generation model comprises a diffusion model.

[0082] Figure 7 An example of an image processing apparatus 700 according to aspects of the present disclosure is shown. The example shown includes the image processing apparatus 700, a processor unit 705, an I / O module 710, a memory unit 715, a data preparation component 745, and a training component 755. In one aspect, the memory unit 715 includes a machine learning model 720, a text embedding model 725, a continuous control model 730, a text encoder 735, and an image generation model 740. In one aspect, the data preparation component 745 includes a trained image generation model 750.

[0083] According to some embodiments of the present disclosure, the image processing device 700 includes a computer-implemented artificial neural network (ANN). An ANN is a hardware or software component that includes a plurality of connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (such as a physical synapse in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. In some examples, the nodes may determine the output using other mathematical algorithms (e.g., selecting the maximum value from the input as the output) or any other suitable algorithm for activating nodes. Each node and edge is associated with one or more node weights that determine how to process and send signals. The image processing device 700 is a reference Figure 1 Examples of corresponding elements described or including reference Figure 1 Aspects of the corresponding elements are described.

[0084] The processor unit 705 is an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor unit 705 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor unit 705 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor unit 705 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor unit 705 is a reference Figure 14 Examples of processors described herein include reference Figure 14 Describes aspects of the processor.

[0085] The I / O module 710 (e.g., input / output interface) may include an I / O controller. The I / O controller may manage input and output signals for the device. The I / O controller may also manage peripheral devices that are not integrated into the device. In some cases, the I / O controller may represent a physical connection or port to an external peripheral device. In some cases, the I / O controller may utilize an operating system, such as or another known operating system. In other cases, an I / O controller may represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller may be implemented as part of a processor. In some cases, a user may interact with a device via the I / O controller or via hardware components controlled by the I / O controller.

[0086] In some examples, the I / O module 710 includes a user interface. The user interface can enable a user to interact with the device. In some embodiments, the user interface can include an audio device, such as an external speaker system, an external display device such as a display screen, or an input device (e.g., a remote control device that interfaces with the user interface directly or through an I / O controller module). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel and can also record and process communications. A communication interface is provided herein to enable a processing system to be coupled to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to send (or transmit) and receive signals for the communication device via an antenna. The I / O module 710 is a reference Figure 14 Examples of I / O interfaces described or included with reference Figure 14 Describes aspects of the I / O interface. The user interface is referenced Figure 1 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 14 Examples of corresponding elements described or including reference Figure 1 、 Figure 3 、 Figure 4 、 Figure 5 and Figure 14 Aspects of the corresponding elements are described.

[0087] Examples of memory unit 715 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory unit 715 include solid-state memory and a hard disk drive. In some examples, memory unit 715 is used to store computer-readable, computer-executable software including instructions that, when executed, cause the processor to perform the various functions described herein.

[0088] In some cases, memory unit 715 includes, among other things, a basic input / output system (BIOS), which controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory units. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within memory unit 715 store information in the form of logical states.

[0089] In one aspect, the memory unit 715 includes a machine learning model 720, a text embedding model 725, a continuous control model 730, a text encoder 735, and an image generation model 740. The memory unit 715 is a reference Figure 14 Examples of memory subsystems described or including reference Figure 14 Describes aspects of the memory subsystem.

[0090] According to some aspects, the machine learning model 720 includes a text embedding model 725, a continuous control model 730, a text encoder 735, and an image generation model 740. In some cases, the machine learning model 720 is a computing algorithm, model, or system that is designed to identify patterns, make predictions, or perform specific tasks (e.g., image processing) without being explicitly programmed. According to some aspects, the machine learning model 720 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof.

[0091] According to some embodiments of the present disclosure, the machine learning model 720 includes an ANN, which is a hardware or software component that includes multiple connected nodes (e.g., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another node (such as a physical synapse in the brain). When a node receives a signal, the node processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated by a function of the sum of its inputs. In some examples, the nodes can use other mathematical algorithms (e.g., selecting the maximum value from the input as the output) or any other suitable algorithm for activating the node to determine the output. Each node and edge is associated with one or more node weights that determine how to process and send signals.

[0092] During the training process, one or more node weights are adjusted to increase the accuracy of the results (e.g., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). The weights of the edges increase or decrease the strength of the signal sent between the nodes. In some cases, the nodes have a threshold below which the signal is not sent at all. In some examples, the nodes are grouped into layers. Different layers perform different transformations on their corresponding inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0093] According to some embodiments, the machine learning model 720 includes a computer-implemented convolutional neural network (CNN). CNN is a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNN can enable the processing of digital images with minimal preprocessing. CNN can be characterized by the use of convolutional (or cross-correlation) hidden layers. These layers apply a convolution operation to the input before sending the result to the next layer with a signal. Each convolution node can process data of a limited input field (e.g., a receptive field). During the forward pass of the CNN, the filter at each layer can be convolved across the input volume, calculating the dot product between the filter and the input. During the training process, the filter can be modified so that the filter is activated when a specific feature within the input is detected.

[0094] In one aspect, the machine learning model 720 includes machine learning parameters. Machine learning parameters (also known as model parameters or weights) are variables that provide the behavior and characteristics of the machine learning model 720. Machine learning parameters can be learned or estimated from training data and used to make predictions or perform tasks based on the patterns and relationships learned in the data.

[0095] During the training process, machine learning parameters are adjusted to minimize a loss function or maximize a performance metric. The goal of the training process is to find optimal values for the parameters that allow the machine learning model 720 to make accurate predictions or perform well on a given task.

[0096] For example, during the training process, the algorithm adjusts machine learning parameters to minimize the error or loss between the predicted output and the actual target based on optimization techniques such as gradient descent, stochastic gradient descent, or other optimization algorithms. Once the machine learning parameters are learned from the training data, the machine learning parameters are used to make predictions on new, unseen data.

[0097] According to some embodiments, the machine learning model 720 includes a computer-implemented recurrent neural network (RNN). RNN is a class of ANNs in which the connections between nodes form a directed graph along an ordered (e.g., time) sequence. This enables RNNs to model temporal dynamic behavior, such as predicting what element should appear next in a sequence. Therefore, RNNs are suitable for tasks involving ordered sequences, such as text identification (wherein words are ordered in a sentence). In some cases, RNNs include one or more finite impulse recurrent networks (characterized in that nodes form a directed acyclic graph), one or more infinite impulse recurrent networks (characterized in that nodes form a directed cyclic graph), or a combination thereof.

[0098] According to some embodiments, the machine learning model 720 includes a transformer (or transformer model or transformer network), where the transformer is a neural network model for natural language processing tasks. The transformer network uses an encoder and a decoder to transform one sequence into another sequence. The encoder and decoder include modules that can be stacked multiple times on top of each other. These modules include multi-head attention and feedforward layers. The input and output (target sentence) are first embedded in an n-dimensional space. The position encoding of the different words (for example, giving a relative position to each word / part in the sequence, because the sequence depends on the order of its elements) is added to the embedding representation (n-dimensional vector) of each word. In some examples, the transformer network includes an attention mechanism, where the attention looks at the input sequence and decides which other parts of the sequence are important at each step. The attention mechanism involves a query, key, and value, represented by Q, K, and V, respectively. Q is a matrix containing the query (a vector representation of a word in the sequence), K is the key (a vector representation of the word in the sequence), and V is the value, which is again a vector representation of the word in the sequence. For the encoder and decoder multi-head attention modules, V consists of the same word sequence as Q. However, for the attention module that considers encoder and decoder sequences, V is different from the sequence represented by Q. In some cases, the values in V are multiplied by some attention weights a and added.

[0099] In the field of machine learning, an attention mechanism (e.g., implemented in one or more ANNs) is a method of assigning different levels of importance to different elements of the input. Computing attention may involve three basic steps. First, the similarity between the query obtained from the input and the key vector is calculated to generate attention weights. Similarity functions used for this process can include dot products, concatenation, detectors, etc. Next, the attention weights are normalized using a softmax function. Finally, the attention weights are weighted together with the corresponding values. In the context of attention networks, keys and values are vectors or matrices used to represent the input data. Keys are used to determine which parts of the input the attention mechanism should focus on, while values are used to represent the actual data being processed.

[0100] Attention mechanisms are a key component in some ANN architectures, particularly those employed in natural language processing (NLP) and sequence-to-sequence tasks, that allow the ANN to focus on different parts of the input sequence when making predictions or generating outputs. Some sequence models, such as RNNs, process the input sequence sequentially, maintaining an internal hidden state that captures information from previous steps. However, in some cases, this sequential processing makes it difficult to capture long-range dependencies or focus on specific parts of the input sequence.

[0101] The attention mechanism addresses these difficulties by enabling the ANN to selectively focus on different parts of the input sequence, assigning different degrees of importance or attention to each part. The attention mechanism achieves selective attention by considering the relevance of each input element to the current state of the ANN.

[0102] The term "self-attention" refers to a machine learning model in which representations of the input interact with each other to determine the attention weights of the input. Self-attention can be distinguished from other attention models because the attention weights are at least partially determined by the input itself.

[0103] According to some aspects, the machine learning model 720 obtains a textual hint describing an element and an attribute value of a continuous attribute of the element. In some aspects, the continuous attribute includes a 3-dimensional characteristic of the element. In some aspects, the textual hint includes a random number token corresponding to the attribute value. In some aspects, the textual hint includes a word corresponding to the continuous attribute.

[0104] In some examples, the machine learning model 720 identifies a negative cue based on an object from a training image set, wherein the composite image is generated based on the negative cue. In some examples, the machine learning model 720 obtains additional attribute values corresponding to an additional continuous attribute, wherein the composite image is generated to depict the additional attribute values. In some examples, the machine learning model 720 obtains a set of attribute values for the continuous attribute. The machine learning model 720 is a reference Figure 3 、 Figure 4、 Figure 5 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 8 、 Figure 12 and Figure 13 Aspects of the corresponding elements are described.

[0105] According to some aspects, the text embedding model 725 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some aspects, the text embedding model 725 embeds the text prompt to obtain a text embedding in a text embedding space. In some examples, the text embedding model 725 divides the text prompt into a token set. In some examples, the text embedding model 725 embeds each token in the token set using the text embedding model 725. The text embedding model 725 is a reference Figure 8 Examples of corresponding elements described or including reference Figure 8 Aspects of the corresponding elements are described.

[0106] According to some aspects, continuous control model 730 is implemented as software stored in memory unit 715 and executable by processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some aspects, continuous control model 730 embeds attribute values using continuous control model 730 to obtain attribute embeddings in a text embedding space.

[0107] According to some aspects, the continuous control model 730 includes parameters stored in at least one memory and is trained to embed attribute values of continuous attributes to obtain attribute embeddings in a text embedding space. In some aspects, the continuous control model 730 includes a multilayer perceptron (MLP). The continuous control model 730 is a reference Figure 8 and Figure 13 Examples of corresponding elements described or including reference Figure 8 and Figure 13 Aspects of the corresponding elements are described.

[0108] According to some aspects, text encoder 735 is implemented as software stored in memory unit 715 and executable by processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some aspects, text encoder 735 encodes text embeddings and attribute embeddings to obtain guidance information for image generation model 740, wherein a synthetic image is generated based on the guidance information.

[0109] According to some aspects, the text encoder 735 includes parameters stored in at least one memory and is configured to encode text embeddings and attribute embeddings to obtain guidance information for the image generation model 740. The text encoder 735 is a reference Figure 8 and Figure 9 Examples of corresponding elements described or including reference Figure 8 and Figure 9 Aspects of the corresponding elements are described.

[0110] According to some aspects, the image generation model 740 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some aspects, the image generation model 740 generates a composite image based on the text embedding and the attribute embedding, wherein the composite image depicts a continuous attribute of the element based on the attribute value. In some examples, the image generation model 740 performs a diffusion process on the noise input to obtain the composite image.

[0111] In some aspects, image generation model 740 is trained using a training set comprising a set of training images, each of which depicts an object having a set of continuous attribute values. In some examples, image generation model 740 generates a set of synthetic images based on the same random input and the set of attribute values. In some aspects, image generation model 740 is trained independently in a first phase. In some aspects, image generation model 740 is trained together with continuous control model 730 in a second phase.

[0112] According to some aspects, the image generation model 740 includes parameters stored in at least one memory and is trained to generate a composite image based on a text embedding and an attribute embedding of a text prompt, wherein the composite image depicts a continuous attribute based on an attribute value. In some aspects, the image generation model 740 includes a diffusion model. The image generation model 740 is a reference Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 8 、 Figure 12 and Figure 13 Aspects of the corresponding elements are described.

[0113] According to some aspects, the data preparation component 745 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some embodiments, the data preparation component 745 is implemented as software stored in the memory unit and executable by a processor in the processor unit of a separate computing device, as firmware in a separate computing device, as one or more hardware circuits in a separate computing device, or as a combination thereof. In some examples, the data preparation component 745 is part of another device other than the image processing device 700 and communicates with the image processing device 700. In some examples, the data preparation component 745 is part of the image processing device 700.

[0114] According to some aspects, the data preparation component 745 includes a training image generation model 750. In one aspect, the data preparation component 745 obtains a training set including a set of training images, each of which depicts an object having a set of values for a continuous attribute. In some examples, the data preparation component 745 renders the training image set based on a 3D model of the object.

[0115] According to some aspects, the training image generation model 750 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some embodiments, the training image generation model 750 is implemented as software stored in the memory unit and executable by a processor in the processor unit of a separate computing device, as firmware in a separate computing device, as one or more hardware circuits of a separate computing device, or as a combination thereof. In some examples, the training image generation model 750 is part of another device other than the image processing device 700 and communicates with the image processing device 700. In some examples, the training image generation model 750 is part of the image processing device 700.

[0116] According to some aspects, the training image generation model 750 generates training images based on the 3D model of the object. Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 12 and Figure 13 In some embodiments, the training image generation model 750 includes 3D rendering. In some embodiments, the training image generation model 750 includes ControlNet.

[0117] According to some aspects, the training component 755 is implemented as software stored in the memory unit 715 and executable by the processor unit 705, as firmware, as one or more hardware circuits, or as a combination thereof. According to some embodiments, the training component 755 is implemented as software stored in the memory unit and executable by a processor in the processor unit of a separate computing device, as firmware in a separate computing device, as one or more hardware circuits of a separate computing device, or as a combination thereof. In some examples, the training component 755 is part of another device other than the image processing device 700 and communicates with the image processing device 700. In some examples, the training component 755 is part of the image processing device 700.

[0118] According to some aspects, training component 755 initializes machine learning model 720. In some examples, training component 755 uses the training set to train image generation model 740 to generate synthetic images having the value set of the continuous attribute. In some examples, training component 755 uses the training set to train continuous control model 730 to generate inputs to image generation model 740 corresponding to the continuous attribute.

[0119] In some examples, the training component 755 calculates a reconstruction loss based on the training set. In some examples, the training component 755 updates parameters of the image generation model 740 and parameters of the continuous control model 730 based on the reconstruction loss.

[0120] Figure 8 An example of a machine learning model 800 according to aspects of the present disclosure is shown. The example shown includes the machine learning model 800, a text prompt 805, a text embedding model 810, a text embedding 815, attributes 820, a continuous control model 825, an attribute embedding 830, a text encoder 835, guide features 840, a noise input 845, an image generation model 850, a synthesized image 855, and a negative prompt 860.

[0121] refer to Figure 8, machine learning model 800 generates a composite image 855 based on text prompts 805 and attributes 820. In some cases, composite image 855 includes the elements described by text prompt 805 and 3-dimensional features from attributes 820. In some cases, for example, text prompt 805 states "a view of a chair in the woods." In some cases, text prompt 805 includes a random number token in a region of the sequence of text prompts 805. For example, the random number token is represented as<V*> For example, text prompt 805 states "The chair in the woods<V*> The text embedding model 810 receives the text prompt 805 to generate a text embedding 815. In some embodiments, the text embedding model 810 divides the text prompt 805 into a plurality of word tokens. In some aspects, the text embedding 815 comprises a table, where each cell of the table comprises the word tokens of the text prompt 805 in sequence.

[0122] According to some embodiments, the continuous control model 825 receives the attribute 820 to generate the attribute embedding 830. In some aspects, the continuous control model 825 is trained using a continuous function, wherein the continuous control model 825 is capable of interpolating between two training data. For example, the continuous control model 825 may receive a first value of the attribute 820 and a second value of the attribute 820, and the continuous control model 825 is trained to generate an intermediate value between the first value and the second value. Thus, the continuous control model 825 can generate a continuous output.

[0123] In some cases, attribute 820 includes a 3-dimensional property of the element described by the text hint. For example, attribute 820 includes multiple values for the 3-dimensional orientation of a chair. In some cases, attribute 820 is integrated into a user control, where the value of attribute 820 can be easily modified using the user control. In some cases, for example, the attribute embedding includes an encoding of the semantic meaning of the 3-dimensional property of attribute 820, where the encoding can be processed by the machine learning model 800. In some embodiments, attribute embedding 830 is combined with text embedding 815 as an input embedding to the text encoder 835 of the image generation model 850. For example, attribute embedding 830 is added to a region of a sequence of text embeddings 815.

[0124] In some embodiments, the text encoder 835 receives the text embeddings 815 (including the attribute embeddings 830) to generate guiding features 840 for the image generation model 850. For example, the guiding features 840 are used to guide the diffusion process in the image generation model 850. In some cases, the guiding features 840 are text embeddings of the text prompt 805 and the attributes 820. In some embodiments, the noise input 845 and the guiding features 840 are provided to the image generation model 850 to generate a composite image 855. In some cases, the noise input 845 is a noise map. In some cases, the noise input 845 includes a noise image obtained from the noise map and the training image. The image generation model 850 performs a diffusion process on the noise input 845 to obtain the composite image 855.

[0125] In some embodiments, the image generation model 850 also receives a negative hint 860 to generate the composite image 855. For example, the negative hint 860 is used to guide the image generation model 850 away from generating the element described by the negative hint 860. For example, the negative hint 860 includes the element depicted in the training image. In one embodiment, the negative hint 860 is provided to the text encoder 835 to generate a negative hint embedding, wherein the guidance features 840 include the negative hint embedding.

[0126] Machine Learning Model 800 is a reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 12 and Figure 13 The text prompt 805 is a reference to the corresponding element. Figure 3-5 、 Figure 9 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3-5 、 Figure 9 、 Figure 12 and Figure 13 The text embedding model 810 is a reference to the corresponding elements of Figure 7 Examples of corresponding elements described or including reference Figure 7 Aspects of the corresponding elements are described.

[0127] Text embedding 815 is a reference Figure 13 Examples of corresponding elements described or including reference Figure 13 Attribute 820 is a reference to the corresponding element. Figure 3 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 12 and Figure 13 The continuous control model 825 is a reference to the corresponding elements of the Figure 7 and Figure 13 Examples of corresponding elements described or including reference Figure 7 and Figure 13 Aspects of the corresponding elements are described.

[0128] Text encoder 835 is a reference Figure 7 and Figure 9 Examples of corresponding elements described or including reference Figure 7 and Figure 9 The guide feature 840 is a reference to the corresponding element of the Figure 9 Examples of corresponding elements described or including reference Figure 9 The image generation model 850 is a reference to the corresponding elements of the Figure 4 、 Figure 7 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 7 、 Figure 12 and Figure 13 The composite image 855 is a reference to the corresponding elements of the Figure 3 、 Figure 4 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 12 and Figure 13 Aspects of the corresponding elements are described.

[0129] Figure 9 An example of a diffusion model 900 according to aspects of the present disclosure is shown. The example shown includes the diffusion model 900, an original image 905, a pixel space 910, an image encoder 915, original image features 920, a latent space 925, a forward diffusion process 930, noise features 935, an inverse diffusion process 940, denoised image features 945, an image decoder 950, an output image 955, a textual hint 960, a textual encoder 965, guided features 970, and a guided space 975.

[0130] Diffusion models are a class of generative neural networks that can be trained to generate new data with features similar to those found in the training data. Specifically, diffusion models can be used to generate novel images. Diffusion models can be used for a variety of image generation tasks, including image super-resolution, image generation with perceptual metrics, conditional generation (e.g., text-guided, color-guided, style-guided, and image-guided generation), image inpainting, and image manipulation.

[0131] Types of diffusion models include denoising diffusion probabilistic models (DDPMs) and denoising diffusion implicit models (DDIMs). In DDPMs, the generative process involves inverting a randomized Markov diffusion process. DDIMs, on the other hand, use a deterministic process so that the same input produces the same output. Diffusion models can also be characterized by whether the noise is added to the image itself or to image features generated by the encoder (e.g., latent diffusion).

[0132] The diffusion model works by iteratively adding noise to the data in a forward process and then learning to recover the data by denoising the data during the reverse process. For example, during training, the diffusion model 900 may take as input an original image 905 in pixel space 910 and apply an image encoder 915 to convert the original image 905 into original image features 920 in a latent space 925. Then, a forward diffusion process 930 gradually adds noise to the original image features 920 to obtain noise features 935 (also in the latent space 925) at various noise levels.

[0133] Next, an inverse diffusion process 940 (e.g., a U-Net ANN) gradually removes noise from the noise features 935 at various noise levels to obtain denoised image features 945 in the latent space 925. In some examples, the denoised image features 945 are compared with the original image features 920 at each of the various noise levels, and the parameters of the inverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, the image decoder 950 decodes the denoised image features 945 to obtain an output image 955 in the pixel space 910. In some cases, the output image 955 is created at each of the various noise levels. The output image 955 can be compared with the original image 905 to train the inverse diffusion process 940. In some cases, the output image 955 refers to a synthesized image (e.g., a reference image). Figure 3 、 Figure 4 、 Figure 5 、 Figure 8 、 Figure 12 and Figure 13 description).

[0134] In some cases, the image encoder 915 and the image decoder 950 are pre-trained before training the inverse diffusion process 940. In some examples, the image encoder 915 and the image decoder 950 are jointly trained or fine-tuned with the inverse diffusion process 940.

[0135] The inverse diffusion process 940 may also be guided based on the textual cue 960 or another guiding cue such as an image, layout, style, color, segmentation map, etc. The textual cue 960 may be encoded using a text encoder 965 (e.g., a multimodal encoder) to obtain guiding features 970 in a guiding space 975. The guiding features 970 may be combined with the noise features 935 at one or more layers of the inverse diffusion process 940 to ensure that the output image 955 includes the content described by the textual cue 960. For example, the guiding features 970 may be combined with the noise features 935 using a cross-attention block within the inverse diffusion process 940. In some cases, the textual cue 960 refers to a reference image. Figure 3 、 Figure 4 、 Figure 5 、 Figure 8 、 Figure 12 and Figure 13 The corresponding elements are described.

[0136] Cross-attention, also known as multi-head attention, is an extension of the attention mechanism used in some ANNs, for example for NLP tasks. In some cases, cross-attention attends to multiple parts of the input sequence simultaneously, thereby capturing interactions and dependencies between different elements. In cross-attention, there are two input sequences: a query sequence and a key-value sequence. The query sequence represents the elements that require attention, while the key-value sequence contains the elements to be attended to. In some cases, to compute cross-attention, the cross-attention block transforms each element in the query sequence (e.g., using linear projection) into a "query" representation, and transforms the elements in the key-value sequence into "key" and "value" representations.

[0137] The cross-attention block calculates an attention score by measuring the similarity between each query representation and the key representation, where a higher similarity indicates more attention is paid to the key element. The attention score indicates the importance or relevance of each key element to the corresponding query element.

[0138] The crisscross attention block then normalizes the attention scores to obtain attention weights (e.g., using a softmax function), where the attention weights determine how much information from each value element is incorporated into the final attended representation. By simultaneously focusing on different parts of the key-value sequence, the crisscross attention block captures relationships and dependencies across the input sequence, allowing the machine learning model to understand the context and generate more accurate and contextually relevant outputs.

[0139] In some examples, the diffusion model is based on a neural network architecture called U-Net. U-Net takes input features with an initial resolution and an initial number of channels and processes the input features using an initial neural network layer (e.g., a convolutional network layer) to generate intermediate features. The intermediate features are then downsampled using a downsampling layer so that the downsampled features have a resolution less than the initial resolution and a number of channels greater than the initial number of channels.

[0140] This process is repeated multiple times, and then the process is reversed. For example, the downsampled features are upsampled using the upsampling process to obtain upsampled features. The upsampled features can be combined with intermediate features of the same resolution and number of channels via skip connections. These inputs are processed using the final neural network layer to produce output features. In some cases, the output features have the same resolution as the initial resolution and the same number of channels as the initial number of channels.

[0141] In some cases, U-Net uses additional input features to produce conditionally generated outputs. For example, the additional input features can include vector representations of input prompts. The additional input features can be combined with intermediate features at one or more layers within the neural network. For example, a crisscross attention module can be used to combine the additional input features with the intermediate features.

[0142] The diffusion process can also be modified based on conditional guidance. In some cases, the user provides a text prompt (e.g., text prompt 960) that describes the content to be included in the generated image. For example, the user can provide the prompt "view of a chair in the woods." In some examples, guidance can be provided in a form other than text, such as via an image, sketch, color, style, or layout. The system converts the text prompt 960 (or other guidance) into a conditional guidance vector or other multidimensional representation. For example, a transformer model or a multimodal encoder can be used to convert the text into a vector or a series of vectors. In some cases, the encoder for conditional guidance is trained independently of the diffusion model.

[0143] Initialize a noise map containing random noise. The noise map can be in pixel space or latent space. By initializing an image with random noise, different variations of the image can be generated that include content described by the conditional guidance. Then, the diffusion model 900 generates an image based on the noise map and the conditional guidance vector.

[0144] The diffusion process may include both a forward diffusion process 930 for adding noise to an image (e.g., original image 905) or a feature (e.g., original image feature 920) in the latent space 925 and a reverse diffusion process 940 for denoising the image (or feature) to obtain a denoised image (e.g., output image 955). The forward diffusion process 930 may be represented as q(xt |x t-1 ), and the reverse diffusion process 940 can be expressed as p(x t-1 |x t In some cases, a forward diffusion process 930 is used during training to generate images with successively greater noise, and the neural network is trained to perform a reverse diffusion process 940 (eg, to successively remove noise).

[0145] In an example forward diffusion process 930 for a latent diffusion model (e.g., diffusion model 900), diffusion model 900 uses a Markov chain to map observation variables x0 (in pixel space 910 or latent space 925) to intermediate variables x1, ..., x T When the latent variable is passed through a neural network such as U-Net, the Markov chain gradually adds Gaussian noise to the data to obtain an approximate posterior q(x 1：T |x0), where x1, ..., x T has the same dimensions as x0.

[0146] The neural network can be trained to perform the inverse diffusion process 940. During the inverse diffusion process 940, the diffusion model 900 is trained from noisy data x, such as a noisy image. T Start by denoising the data to obtain p(x t-1 |x t ). At each step t-1, the reverse diffusion process 940 converts x t (such as the first intermediate image) and t as input. Here, t represents a step in the sequence of transitions associated with different noise levels, and the reverse diffusion process 940 iterates to output x t-1 (such as the second intermediate image), until x T Restore back to x0, the original image 905. The reverse diffusion process 940 can be expressed as:

[0147]

[0148] The joint probability of a sequence of samples in a Markov chain can be written as the product of the conditional and marginal probabilities:

[0149]

[0150] where p(x T )=N(x T 0, I) is a pure noise distribution, because the reverse diffusion process 940 takes the result of the forward diffusion process 930 (ie, pure noise samples) as input, and Represents the sequence of Gaussian transformations corresponding to the sequence in which Gaussian noise is added to the samples.

[0151] At the interference time, the observation data x0 in the pixel space can be mapped into the latent space 925 as input, and the generated data The latent space 925 is mapped back to the pixel space 910 as the output. In some examples, x0 represents the original input image with low image quality, and the latent variables x1, ..., x T represents a noisy image, and Indicates a generated image with high image quality.

[0152] Diffusion model 900 can be trained using both forward diffusion process 930 and reverse diffusion process 940. In one example, a user initializes an untrained model. Initialization can include defining the architecture of the model and establishing initial values for the model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0153] The system then adds noise to the training images using a forward diffusion process 930 in N stages. In some cases, the forward diffusion process 930 is a fixed process in which Gaussian noise is continuously added to the image. In a latent diffusion model, Gaussian noise can be continuously added to features in the latent space 925 (e.g., original image features 920).

[0154] At each stage n, starting from stage N, the inverse diffusion process 940 is used to predict the image or image features at stage n-1. For example, the inverse diffusion process 940 can predict the noise added by the forward diffusion process 930 and the predicted noise can be removed from the image to obtain a predicted image. In some cases, the original image 905 is predicted at each stage of the training process.

[0155] Training components (e.g., reference Figure 7 The training component described herein compares the predicted image (or image features) at stage n-1 with the actual image (or image features) (such as the image at stage n-1 or the original input image). For example, given observation data x, the diffusion model 900 can be trained to minimize the negative log-likelihood of the training data - log p θ The training component then updates the parameters of the diffusion model 900 based on the comparison. For example, gradient descent can be used to update the parameters of the U-Net. The time-dependent parameters of the Gaussian transition can also be learned.

[0156] Text prompt 960 is a reference Figure 3-5 、 Figure 8 、 Figure 12 and Figure 13 Examples of corresponding elements described or including reference Figure 3-5 、 Figure 8 、 Figure 12 and Figure 13 The text encoder 965 is a reference to the corresponding elements of the description. Figure 7 and Figure 8 Examples of corresponding elements described or including reference Figure 7 and Figure 8 The guide feature 970 is a reference to the corresponding element of the Figure 8 Examples of corresponding elements described or including reference Figure 8 Aspects of the corresponding elements are described.

[0157] Figure 10 An example of a method 1000 for generating a composite image based on embedding according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed in conjunction with other operations.

[0158] At operation 1005, the system divides the text prompt into token sets. In some cases, the operation of this step involves referring to Figure 7 and Figure 8 The text embedding model described can also be used as reference Figure 7 and Figure 8 The described text embedding model performs. In some cases, the text embedding model divides the text prompt into multiple word tokens.

[0159] At operation 1010, the system embeds each token in the token set to obtain a text embedding. In some cases, the operation of this step involves as described in reference Figure 7 and Figure 8 The text embedding model described can also be used as reference Figure 7 and Figure 8 The described text embedding model performs. In some cases, the text embedding includes a lookup table where each cell of the table includes word tokens of a text prompt in sequence.

[0160] At operation 1015, the system encodes the text embedding and the attribute embedding of the continuous attribute to obtain guidance information for the image generation model, wherein the synthetic image is generated based on the guidance information. In some cases, the operation of this step involves as described in reference Figure 7-9 The text encoder described can also be used as reference Figure 7-9The described text encoder performs. For example, the guidance information is used to guide the diffusion process in the image generation model. In some cases, the guidance information is a text embedding of textual cues and attributes. In some embodiments, the image generation model is provided with the noise input and the guidance information to generate a synthetic image.

[0161] Training and evaluation

[0162] exist Figure 11-13 In the present invention, a method, apparatus, non-transitory computer-readable medium, and system for image processing include initializing a machine learning model, obtaining a training set including a plurality of training images, the plurality of training images respectively depicting an object having a plurality of values of a continuous attribute, training an image generation model using the training set to generate a composite image having the plurality of values of the continuous attribute, and training a continuous control model using the training set to generate an input to the image generation model corresponding to the continuous attribute.

[0163] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include rendering a plurality of training images based on the 3D model of the object. Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include generating the training images based on the 3D model of the object using the training image generation model.

[0164] In some aspects, the image generative model is trained separately in the first phase. In some aspects, the image generative model is trained together with the continuous control model in the second phase. Some examples of the methods, apparatuses, non-transitory computer-readable media, and systems further include calculating a reconstruction loss based on the training set. Some examples further include updating parameters of the image generative model and parameters of the continuous control model based on the reconstruction loss.

[0165] Figure 11 An example of a method 1100 for training a machine learning model according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a set of codes to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0166] At operation 1105, the system initializes the machine learning model. In some cases, the operation of this step involves referring to Figure 7 The training components described may be provided by reference to Figure 7In some cases, initialization can include defining the architecture of the machine learning model and establishing initial values for the model parameters. In some cases, initialization can include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0167] At operation 1110, the system obtains a training set comprising a set of training images, each of which depicts an object having a set of values for a continuous attribute. In some cases, the operation of this step involves referring to Figure 7 The data preparation components described may be provided by reference to Figure 7 In some cases, obtaining the training set includes creating the training set using the data preparation component. For example, the data preparation component generates the training set according to a number of attributes I = f(a1, a2, a3, ..., a n ) obtain an image I including an object O from category C, where a i Belongs to the image attribute set Shape, material reflectivity, rotation / translation, camera intrinsic / extrinsic, shape deformation, etc. In some embodiments, attribute a is controlled by using a rendering engine to generate training images with attribute value a=x. In addition, token T x Assigning to identified images having the same attribute value. In some aspects, attribute a is continuous and has multiple values, wherein an image generation model is trained using tokens and corresponding attribute values to provide fine-grained control over the attribute. In some cases, the trained image generation model includes a 3D renderer that generates training images based on 3D data of the object and the multiple continuous attributes.

[0168] In some embodiments, the training set is augmented to prevent the fine-tuning process from overfitting to a simple white background and predefined object textures. For example, a training image generation model is used to enhance the background and texture of the training images during the rendering process (e.g., generating training images). In some embodiments, ControlNet is used to generate the augmented training images. In some cases, when the attribute is reflected in shape changes (e.g., wing pose), the training image generation model uses the ground truth depth map as a conditioning for the ControlNet-generated augmented training images. In some cases, when the attribute cannot be reflected from the depth map (e.g., lighting), the training image generation model generates preliminary training images without texture and uses a line art extractor to obtain sketches of the preliminary training images. The sketches of the preliminary training images capture features such as shading and shadows in pixel space, which can be used as conditioning for the ControlNet-generated augmented training images.

[0169] In some embodiments, ControlNet is provided with additional cues describing the appearance and background of objects to generate augmented training images. In some cases, the additional cues are simple and brief. In some embodiments, the training set includes both training images and augmented training images. For example, the training set includes a subset of augmented training images. In some cases, the additional cues are used to guide the image generation model in the second stage of training.

[0170] At operation 1115, the system trains the image generation model using the training set to generate a synthetic image having the value set of the continuous attribute. In some cases, the operation of this step involves as described in reference Figure 7 The training components described may be provided by reference to Figure 7 The training component described performs. For example, an image generation model is trained to generate a synthetic image depicting an element described by a text prompt and 3-dimensional features from an attribute input (e.g., a continuous attribute). Figure 12 and Figure 13 Describes more details about training image generation models.

[0171] At operation 1120, the system trains the continuous control model using the training set to generate inputs to the image generation model corresponding to the continuous attributes. In some cases, the operation of this step involves Figure 7 The training components described may be provided by reference to Figure 7 The training component described performs the following. For example, the continuous control model is trained to generate attribute embeddings based on attribute inputs, where the attribute embeddings are added to the text embeddings of the text prompts as input to the image generation model. Figure 12 and Figure 13 More details about training continuous control models are described.

[0172] Figure 12 An example of a first phase of training according to aspects of the present disclosure is shown. The example shown includes a machine learning model 1200, training data 1205, attributes 1210, a trained image generation model 1215, training images 1220, noisy images 1225, text prompts 1230, an image generation model 1235, a synthesized image 1240, and a loss 1245.

[0173] refer to Figure 12, the machine learning model 1200 is fine-tuned using loss 1245 during the first phase of training. For example, the machine learning model 1200 obtains a training set including training data 1205 and attributes 1210. In one aspect, the training data 1205 includes 3D data points (or mesh) of an object (e.g., a dog). In one aspect, the attributes 1210 include 3-dimensional characteristics of the object, such as, for example, 3-dimensional orientation, lighting direction, wing pose, etc. Using the training data 1205 and the attributes 1210, a training image generation model 1215 is used to generate a training image 1220 depicting a dog based on the training data 1205 and the 3-dimensional characteristics from the attributes 1210. In one aspect, the training image generation model 1215 includes a 3D renderer that generates an image (e.g., training image 1220) based on the mesh (e.g., training data 1205).

[0174] According to some embodiments, the image generation model 1235 is fine-tuned using a loss 1245. For example, the machine learning model 1200 applies a noise map to the training image 1220 to obtain a noisy image 1225. The image generation model 1235 receives the noisy image 1225 and the text prompt 1230 to generate a composite image 1240. For example, the text prompt 1230 states "[obj] photo of a dog." In one aspect, [obj] represents the identity of the dog from the training data 1205. By training the image generation model 1235 using the identifier [obj], the image generation model 1235 is trained to retain and learn the identity of the dog to be generated in the composite image 1240. In some embodiments, the loss 1245 is calculated based on the composite image 1240 and the training image 1220. For example, the loss 1245 includes a reconstruction loss. In some aspects, the training loss (e.g., loss 1245) is represented as:

[0175]

[0176] Among them I a represents a training image 1220 depicting attribute a, represents a noisy image 1225 with noise ∈, and P(g Φ (a)) indicates the hint of attribute a.

[0177] Machine Learning Model 1200 is a reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 and Figure 13The training data 1205 is a reference to the corresponding elements. Figure 13 Examples of corresponding elements described or including reference Figure 13 Attribute 1210 is a reference to the corresponding element. Figure 3 、 Figure 8 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 8 and Figure 13 Aspects of the corresponding elements are described.

[0178] The training image generation model 1215 is a reference Figure 7 and Figure 13 Examples of corresponding elements described or including reference Figure 7 and Figure 13 The training image 1220 is a reference image. Figure 13 Examples of corresponding elements described or including reference Figure 13 The noise image 1225 is a reference to the corresponding element. Figure 13 Examples of corresponding elements described or including reference Figure 13 Aspects of the corresponding elements are described.

[0179] Text prompt 1230 is a reference Figure 3-5 、 Figure 8 、 Figure 9 and Figure 13 Examples of corresponding elements described or including reference Figure 3-5 、 Figure 8 、 Figure 9 and Figure 13 The image generation model 1235 is a reference to the corresponding elements of the Figure 4 、 Figure 7 、 Figure 8 and Figure 13 Examples of corresponding elements described or including reference Figure 4 、 Figure 7 、 Figure 8 and Figure 13 The composite image 1240 is a reference to the corresponding elements of the description. Figure 3 、 Figure 4 、 Figure 8 and Figure 13 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 8 and Figure 13 The corresponding elements of the description are shown in FIG. 1245. Figure 13 Examples of corresponding elements described or including reference Figure 13 Aspects of the corresponding elements are described.

[0180] Figure 13 An example of second-stage training according to aspects of the present disclosure is shown. The example shown includes a machine learning model 1300, training data 1305, attributes 1310, a trained image generation model 1315, training images 1320, noisy images 1325, a continuous control model 1330, textual prompts 1335, text embeddings 1340, an image generation model 1345, a synthesized image 1350, and a loss 1355.

[0181] refer to Figure 13 , using loss 1355 to fine-tune machine learning model 1300 during the second phase of training. For example, machine learning model 1300 obtains a training set including training data 1305 and attributes 1310. In one aspect, training data 1305 includes 3D data points (or a mesh) of an object (e.g., a dog). In one aspect, attributes 1310 include 3D characteristics of the object, such as, for example, 3D orientation, lighting direction, wing pose, etc. Using training data 1305 and attributes 1310, training image generation model 1315 is used to generate training images 1320 depicting a dog based on training data 1305 and the 3D characteristics from attributes 1310. In one aspect, training image generation model 1315 includes a 3D renderer that generates an image (e.g., training image 1320) based on the mesh (e.g., training data 1305). In one aspect, training image generation model 1315 includes a ControlNet that generates training image 1320 based on training data 1305 and attributes 1310.

[0182] According to some embodiments, the continuous control model 1330 generates attribute embeddings based on the attributes 1310. In one aspect, the machine learning model 1300 encodes the text prompt 1335 to obtain the text embedding 1340. In some embodiments, the attribute embedding is combined with the text embedding 1340 as input to the image generation model 1345. For example, the image generation model 1345 receives the noisy image 1325 (e.g., obtained from the training image 1320) and the text embedding 1340 (e.g., obtained from the attributes 1310 and the text prompt 1335) to generate a composite image 1350. In some cases, the image generation model 1345 performs a diffusion process (e.g., referring to the image generation model 1345) on the noisy image 1325. Figure 9 ) to generate the composite image 1350.

[0183] In some embodiments, the machine learning model 1300 (including the image generation model 1345 and the continuous control model 1330) is based on a continuous function To train, this continuous function maps a set of attributes from the continuous domain D to the token embedding domain In some embodiments, the machine learning model 1300 uses positional encoding to project each attribute a∈a into a high-frequency space before providing it to a continuous function. For example, an attribute (e.g., attribute 1310) is provided to a continuous control model 1330 to generate an attribute embedding, which includes a 2-layer multi-layer perceptron (MLP). By transforming the attributes into a high-frequency space, the machine learning model 1300 enables the user to easily control continuous attributes from a text prompt 1335 enhanced by a token embedding (e.g., an attribute embedding).

[0184] In some embodiments, the image generation model 1345 is fine-tuned using a loss 1355 calculated based on the synthesized image 1350 and the training image 1320. For example, the loss 1355 includes a reconstruction loss. In some aspects, the training loss (e.g., loss 1355) is expressed as:

[0185]

[0186] Among them, T O Indicates the adjustment of the text prompt 1335 describing the object O. According to some aspects, for I O For each image with different attribute a, the machine learning model 1300 adjusts P(T O ) are associated with the same object O and use the prompt condition P(T O , g Φ (a) Training the image generation model 1345 and the continuous control model g Φ θ (e.g., text embedding 1340 includes the text embedding of text prompt 1335 and the attribute embedding of attribute 1310). Thus, machine learning model 1300 can be trained to generate a composite image 1350 depicting the element described by text prompt 1335 and attribute 1310.

[0187] Machine learning model 1300 is a reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 and Figure 12 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 5 、 Figure 7 、 Figure 8 and Figure 12 The training data 1305 is a reference to the corresponding elements. Figure 12 Examples of corresponding elements described or including reference Figure 12 Attribute 1310 is a reference to the corresponding element. Figure 3 、 Figure 8 and Figure 12 Examples of corresponding elements described or including reference Figure 3 、 Figure 8 and Figure 12 Aspects of the corresponding elements are described.

[0188] The training image generation model 1315 is referenced Figure 7 and Figure 12 Examples of corresponding elements described or including reference Figure 7 and Figure 12 The training image 1320 is a reference image. Figure 12 Examples of corresponding elements described or including reference Figure 12 The noise image 1325 is a reference to the corresponding element. Figure 12 Examples of corresponding elements described or including reference Figure 12 Aspects of the corresponding elements are described.

[0189] Continuous control model 1330 is a reference Figure 7 and Figure 8 Examples of corresponding elements described or including reference Figure 7 and Figure 8 The text prompt 1335 is a reference to the corresponding element. Figure 3-5 、 Figure 8 、 Figure 9 and Figure 12 Examples of corresponding elements described or including reference Figure 3-5 、 Figure 8 、 Figure 9 and Figure 12 The text embedding 1340 is a reference to the corresponding element. Figure 8 Examples of corresponding elements described or including reference Figure 8 Aspects of the corresponding elements are described.

[0190] Image generation model 1345 is a reference Figure 4 、 Figure 7 、 Figure 8 and Figure 12 Examples of corresponding elements described or including reference Figure 4 、 Figure 7 、 Figure 8 and Figure 12 The composite image 1350 is a reference to the corresponding elements of the Figure 3 、 Figure 4 、 Figure 8 and Figure 12 Examples of corresponding elements described or including reference Figure 3 、 Figure 4 、 Figure 8 and Figure 12 The corresponding elements of the description are shown in FIG. 1355. Figure 12 Examples of corresponding elements described or including reference Figure 12 Aspects of the corresponding elements are described.

[0191] computing devices

[0192] Figure 14 An example of a computing device 1400 according to aspects of the present disclosure is shown. The example shown includes computing device 1400, processor 1405, memory subsystem 1410, communication interface 1415, I / O interface 1420, user interface component 1425, and channel 1430.

[0193] In some embodiments, computing device 1400 is a reference Figure 1 and Figure 7 Examples of the image processing apparatus described herein or including reference Figure 1 and Figure 7 In some embodiments, the computing device 1400 includes a processor 1405 that can execute instructions stored in the memory subsystem 1410 to obtain a text hint describing an element and attribute values describing continuous attributes of the element, embed the text hint to obtain a text embedding in a text embedding space, embed the attribute values to obtain an attribute embedding in the text embedding space, and generate a composite image based on the text embedding and the attribute embedding, wherein the composite image depicts the continuous attributes of the element based on the attribute values.

[0194] According to some embodiments, the processor 1405 includes one or more processors. In some cases, the processor 1405 is an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof. In some cases, the processor 1405 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into the processor 1405. In some cases, the processor 1405 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor 1405 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing. The processor 1405 is a reference Figure 7 Examples of processor units described or including reference Figure 7 Aspects of a processor unit are described.

[0195] According to some embodiments, the memory subsystem 1410 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable, computer-executable software, including instructions that, when executed, cause the processor to perform the various functions described herein. In some cases, the memory includes, among other things, a basic input / output system (BIOS), which controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory cells. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within the memory store information in the form of logical states. The memory subsystem 1410 is a reference to Figure 7 Examples of memory cells described or including reference Figure 7 Aspects of a memory cell are described.

[0196] According to some embodiments, communication interface 1415 operates at the boundary between communication entities (such as computing device 1400, one or more user devices, the cloud, and one or more databases) and channel 1430 and can record and process communications. In some cases, communication interface 1415 is provided to enable the processing system to couple to a transceiver (e.g., a transmitter and / or receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna. In some cases, a bus is used in communication interface 1415.

[0197] According to some embodiments, I / O interface 1420 is controlled by an I / O controller to manage input and output signals for computing device 1400. In some cases, I / O interface 1420 manages peripheral devices that are not integrated into computing device 1400. In some cases, I / O interface 1420 represents a physical connection or port to an external peripheral device. In some cases, an I / O controller uses an operating system, such as or other known operating systems. In some cases, an I / O controller represents or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, an I / O controller is implemented as a component of a processor. In some cases, a user interacts with a device via I / O interface 1420 or a hardware component controlled by an I / O controller. I / O interface 1420 is a reference to Figure 7 Examples of I / O modules described or included with reference Figure 7 Describes aspects of the I / O module.

[0198] According to some embodiments, user interface component 1425 enables a user to interact with computing device 1400. In some cases, user interface component 1425 includes an audio device, such as an external speaker system, an external display device, such as a display screen, an input device (e.g., a remote control device that interfaces with the user interface, either directly or through an I / O controller), or a combination thereof.

[0199] The performance of the apparatus, system, and method of the present disclosure has been evaluated, and the results indicate that embodiments of the present disclosure have achieved increased performance over prior art techniques (e.g., conventional image generation models). Example experiments show that the image processing apparatus based on the present disclosure outperforms conventional image generation models. Reference Figure 3 、 Figure 4 and Figure 5 Details of example use cases based on embodiments of the present disclosure are described.

[0200] The descriptions and drawings described herein represent example configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in block diagram form to illustrate the relationships between components and avoid obscuring the concepts being described. Similar components or features may have the same name but different reference numerals corresponding to different figures.

[0201] For those skilled in the art, some modifications to the present disclosure may be obvious, and the principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should be given the widest scope consistent with the principles and novel features disclosed herein.

[0202] The methods described may be implemented or performed by a device comprising a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration). Therefore, the functions described herein may be implemented in hardware or software and may be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored on a computer-readable medium in the form of instructions or code.

[0203] Computer-readable media include both non-transient computer storage media and communication media, and communication media include any medium that facilitates delivery of code or data. Non-transient storage media can be any available medium that can be accessed by a computer. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc (CD) or other optical disc storage, magnetic disk storage or any other non-transient medium for carrying or storing data or code.

[0204] Additionally, a connecting component may also be appropriately termed a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals) is used to transmit code or data from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0205] In this disclosure and the claims that follow, the word "or" indicates an inclusive list, such that, for example, a list of X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. Furthermore, the phrase "based on" is not intended to indicate a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted to mean "based, at least in part, on." Furthermore, "a" or "an" indicates "at least one."

Claims

1. A method comprising: Get the text hint describing the element and the attribute values of the continuous attributes of the element; embedding the text hint to obtain a text embedding in a text embedding space; embedding the attribute value using a continuous control model to obtain an attribute embedding in the text embedding space; as well as A composite image is generated based on the text embedding and the attribute embedding using an image generation model, wherein the composite image depicts the continuous attribute of the element based on the attribute value.

2. The method according to claim 1, wherein: The continuous attributes include 3-dimensional characteristics of the element.

3. The method of claim 1 , wherein embedding the text prompt comprises: dividing the text prompt into a plurality of tokens; as well as Each token in the plurality of tokens is embedded using a text embedding model.

4. The method according to claim 1, wherein: The text prompt includes a random number token corresponding to the attribute value.

5. The method according to claim 1, wherein: The text prompt includes words corresponding to the continuous attributes.

6. The method according to claim 1, further comprising: The text embedding and the attribute embedding are encoded to obtain guidance information for the image generation model, wherein the synthetic image is generated based on the guidance information.

7. The method of claim 1 , wherein generating the composite image comprises: A diffusion process is performed on the noise input to obtain the composite image.

8. The method according to claim 1, wherein: The image generation model is trained using a training set comprising a plurality of training images, the plurality of training images respectively depicting an object having a plurality of values for the continuous attribute.

9. The method according to claim 8, further comprising: A negative cue is identified based on the object from the plurality of training images, wherein the composite image is generated based on the negative cue.

10. The method according to claim 1, further comprising: An additional attribute value corresponding to an additional continuous attribute is obtained, wherein the composite image is generated to depict the additional attribute value.

11. The method according to claim 1 , further comprising: Obtaining multiple attribute values of the continuous attribute; as well as A plurality of synthetic images are generated using the image generation model based on the same random input and the plurality of attribute values, respectively.

12. A method comprising: Initialize the machine learning model; obtaining a training set comprising a plurality of training images, the plurality of training images respectively depicting an object having a plurality of values for a continuous attribute; training an image generation model using the training set to generate a composite image having the plurality of values of the continuous attribute; as well as A continuous control model is trained using the training set to generate inputs to the image generation model corresponding to the continuous attributes.

13. The method according to claim 12, wherein obtaining the training set comprises: The plurality of training images are rendered based on the 3D model of the object.

14. The method according to claim 12, wherein obtaining the training set comprises: A training image is generated based on the 3D model of the object using a training image generation model.

15. The method of claim 12, wherein: The image generation model is trained separately in the first stage, and The image generation model is trained together with the continuous control model in the second stage.

16. The method of claim 12, wherein training the image generation model comprises: Calculating a reconstruction loss based on the training set; as well as Parameters of the image generation model and parameters of the continuous control model are updated based on the reconstruction loss.

17. An apparatus comprising: at least one processor; at least one memory storing instructions executable by the at least one processor; a continuous control model comprising parameters stored in the at least one memory and trained to embed attribute values of continuous attributes to obtain attribute embeddings in a text embedding space; as well as An image generation model comprising parameters stored in the at least one memory and trained to generate a composite image based on a text embedding of a text prompt and the attribute embedding, wherein the composite image depicts the continuous attribute based on the attribute value.

18. The apparatus according to claim 17, further comprising: A text encoder includes parameters stored in the at least one memory and is configured to encode the text embedding and the attribute embedding to obtain guidance information for the image generation model.

19. The apparatus of claim 17, wherein: The continuous control model includes a multi-layer perceptron (MLP).

20. The apparatus of claim 17, wherein: The image generation model includes a diffusion model.

Images