Customized generation of objects using diffusion models

Abstract

A method of customizing the generation of an object using a diffusion model is provided. One or more parameters of the diffusion model (e.g., conditioning signal, network weight, or initial or initial noise) may be optimized by a back propagation process that may be performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method. Customized diffusion models may generate stylized objects, generate objects with particular visual effect (s), and provide adversarial examples to audit the security of the object generation system.

Description

Technical Field

[0001] Embodiments described herein generally relate to generating objects using diffusion models. More specifically, embodiments described herein relate to customized generation of objects using diffusion models. Background Art

[0002] Diffusion models are machine learning models that can be used to receive natural language descriptions (e.g., text) as input and generate images that match the description. For example, a diffusion probability model (DPM) can include a series of generative models that diffuse the data distribution into Gaussian white noise and then restore the random diffusion process to synthesize new content / images. DPM-based methods can generate various types of content, such as, for example, images, videos, audio data, etc. In some cases, users can customize commercially available diffusion models (e.g., pre-trained DPMs) to generate desired images. Such customization of diffusion models (e.g., pre-trained DPMs) may require multiple reference examples to align the pre-trained DPM with the concepts provided by the user. For example, a user can use multiple Van Gogh painting images as input to customize the parameters of a pre-trained diffusion model so that a pre-trained diffusion model with customized parameters can generate images in the style of Vincent Van Gogh's paintings. However, when the number of Vincent Van Gogh painting images input is limited, the resulting output image not only looks like a painting in the style of Vincent Van Gogh's painting, but is also limited by the content of the input image, such as sunflowers in Vincent Van Gogh's paintings. Summary of the invention

[0003] In one example embodiment, a method for generating stylized objects using a diffusion model is provided. The method includes: obtaining a reference style representation of a reference object; generating one or more objects by performing a forward generation process of the diffusion model; determining a style loss of each object relative to the reference object; optimizing the network weights by back-propagating the gradient of the style loss relative to the reference object to minimize the style loss and obtain a customized diffusion model with optimized network weights; and generating a stylized object using the customized diffusion model with the optimized network weights, the stylized object having a style similar to that of the reference object.

[0004] In another example embodiment, a method for generating an object with a visual effect using a diffusion model is provided. The method includes: obtaining a reference object with a predetermined visual effect, and generating one or more objects by performing a forward generation process of the diffusion model with a control signal. The control signal is configured to control the visual effect of the object. The method also includes: determining the feature loss of each object relative to the reference object with a predetermined visual effect; optimizing the control signal by back-propagating the gradient of the feature loss relative to the control signal to minimize the feature loss and obtain a customized diffusion model with an optimized control signal; and generating an object using the customized diffusion model with the optimized control signal. The object has a visual effect similar to the predetermined visual effect of the reference object.

[0005] In yet another example embodiment, a method for generating one or more objects by performing a forward generation process having as an input a diffusion model with a perturbed starting noise as a content modifier is provided. The input corresponds to a reference content category identified by a classifier. The method also includes determining a classification score for each object via the classifier, and determining a classification loss of the classification score relative to a target content category. The target content category is different from the reference content category. The method also includes optimizing the perturbation of the starting noise by backpropagating the gradient of the classification loss relative to the perturbation to minimize the classification loss of the classification score and obtain a customized diffusion model with optimized starting noise, and generating adversarial examples using the customized diffusion model with optimized starting noise. The adversarial example is identified as a target content category by the classifier.

[0006] The embodiments described herein may provide examples of using diffusion models to customize the generation of objects, including, for example, generating stylized objects, generating objects with specific visual effects, and providing adversarial examples to audit the security of object generation systems. The embodiments described herein may also provide examples of optimizing one or more parameters (e.g., a regulation / control signal, a network weight, or an initial or starting noise) of a diffusion model through a back-propagation process that is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings illustrate various embodiments of the system and method of the present disclosure and embodiments of various other aspects of the present disclosure.Any person of ordinary skill in the art will appreciate that the element boundaries (e.g., boxes, box groups or other shapes) illustrated in the figure represent an example of boundaries.In some examples, an element can be designed as multiple elements, or multiple elements can be designed as one element.In some examples, an element shown as an internal component of an element can be implemented as an external component in another element, and vice versa.Non-restrictive and non-exhaustive descriptions are described with reference to the following drawings.The components in the drawings are not necessarily drawn to scale, but emphasize the explanation principle.In the detailed description below, the embodiments are described only as examples, because according to the detailed description below, various changes and modifications are obvious to those skilled in the art.

[0008] Figure 1 is a schematic diagram of an example object generation system arranged in accordance with at least some embodiments described herein.

[0009] Figure 2 An example system is illustrated in which generation of objects using a diffusion model may be implemented, arranged in accordance with at least some embodiments described herein.

[0010] Figure 3 Illustrate the use of an arrangement according to at least some embodiments described herein Figure 2 The system is used to generate an example process for an object.

[0011] Figure 4A is a flow chart illustrating an example process flow for customizing a diffusion model to generate a styled object in accordance with at least some embodiments described herein.

[0012] Figure 4B Illustrate the use of an arrangement according to at least some embodiments described herein Figure 4A Example process of generating styled objects using a custom diffusion model.

[0013] Figure 5A is a flow chart illustrating an example process flow for customizing a diffusion model to generate an object with a visual effect, according to at least some embodiments described herein.

[0014] Figure 5B Illustrate the use of an arrangement according to at least some embodiments described herein Figure 5A Example process of using a custom diffusion model to generate visually appealing objects.

[0015] Fig. 6A is a flow chart illustrating an example process flow for customizing a diffusion model to provide adversarial examples to audit the security of an object generation system in accordance with at least some embodiments described herein.

[0016] Figure 6B Illustrate the use of an arrangement according to at least some embodiments described herein Fig. 6A Adversarial examples generated by our custom diffusion model and corresponding comparison examples.

[0017] Figure 7 is a schematic block diagram of an example computer system suitable for implementing an electronic device, arranged in accordance with at least some embodiments described herein. DETAILED DESCRIPTION

[0018] In the following detailed description, specific embodiments of the present disclosure will be described with reference to the accompanying drawings, which form a part of the specification. In this description and in the accompanying drawings, unless the context otherwise indicates, similar reference numerals represent elements that can perform the same, similar or equivalent functions. In addition, unless otherwise specified, the description of each consecutive figure may refer to one or more features from the previous figures to provide a clearer context and more substantial explanation of the current example embodiment. However, the example embodiments described in the detailed description, the drawings, and the claims are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the various aspects of the present disclosure as illustrated in the overall description and the drawings herein may be arranged, replaced, combined, separated and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0019] It should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be embodied in various forms. Well-known functions or configurations are not described in detail to avoid unnecessary details obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but only as the basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure differently with virtually any appropriate detailed structure.

[0020] In addition, the present disclosure may be described in terms of functional block components and various processing steps herein. It should be understood that such functional blocks may be implemented by any number of hardware and / or software components configured to perform the specified functions.

[0021] The scope of the present disclosure should be determined by the appended claims and their legal equivalents, rather than by the examples given herein. For example, the steps recited in any method claim may be performed in any order and are not limited to the order presented in the claims. In addition, unless specifically described as "critical" or "essential" herein, no element is essential to the practice of the present disclosure.

[0022] As referred to herein, a "text-to-image model" or "text-to-image generative model" may refer to a machine learning model that may be used to receive a natural language description (e.g., text) as input and generate an image that matches the natural language description. It should be understood that a text-to-image model typically includes a language model (which may be used to convert the received text input into a latent representation) and a generative image model (which may be used to generate an image adjusted according to the latent representation). It should also be understood that in the fields of artificial intelligence and / or machine learning, training a text-to-image model typically requires an image dataset paired with one or more text descriptions. It should also be understood that "pre-trained" may indicate that a machine learning model has been trained and corresponding training checkpoints have been used. A "pre-trained" text-to-image model may refer to a text-to-image model that has been trained on sufficient image and text data (e.g., crawled from the web, a database, a cloud, etc.).

[0023] As referred to herein, a "diffusion model," "diffusion generative model," or "diffusion-based model" in machine learning may refer to a diffusion probabilistic model and / or a latent variable model. It should be understood that a diffusion model is designed to learn the latent structure of a dataset by modeling the way in which data points diffuse in a latent space. For example, in computer vision, a diffusion model includes a neural network that is trained to denoise an image blurred with Gaussian noise to reverse the diffusion process. It should also be understood that diffusion models can be applied to a variety of tasks, including image denoising, in-painting, super-resolution, image generation, and the like. For example, utilization of an image generation diffusion model can begin with a random noise image, and then, after being trained to reverse the diffusion process performed on a natural image, the diffusion model can be implemented to generate new natural images. It should also be understood that given an image ("x0") randomly sampled from a true data distribution, there can be two processes in a diffusion model: (1) a diffusion process that repeatedly adds Gaussian noise to the sample to generate a diffuse sample, such that the sample is generated after a certain amount (e.g., number T) of operations or iterations (x0→x T ) and then gradually becomes Gaussian noise, and (2) the inverse process (or referred to as denoising process, etc.), which repeats denoising given a Gaussian noise input (e.g., diffusion samples, etc.) to generate a denoised output (x T →x0).

[0024] It should be understood that the diffusion process can be a continuous-time Markov process with a continuous sample path. The diffusion process can corrupt the sample (e.g., an image, etc.) by adding Gaussian noise to the sample. As referred to herein, "Gaussian noise" can indicate a type of signal noise having a probability density function equal to a normal distribution (also known as a Gaussian distribution). It should also be understood that a denoising process (or noise reduction process) can be a process of removing noise from a sample. It should also be understood that in some example embodiments, the denoising process may distort the sample to a certain extent.

[0025] It should be understood that a "pre-trained diffusion probability model (DPM)" may refer to a pre-trained (as described above), diffusion-based (having a diffusion process and a denoising process from a diffusion model, as described above), text-to-image generative model (as described above). In an example embodiment, a diffusion-based generative model of text-to-image may refer to a diffusion-based generative model that accepts a text input and synthesizes an image that matches the text input. It should be understood that a machine learning model, such as a diffusion-based generative model of text-to-image, can convert input text into a latent representation to produce an image adjusted according to the latent representation.

[0026] As referred to herein, "conditioning input" or "conditioning" or "conditioner" may refer to an input for which the operation of a process of a machine learning model is conditioned (e.g., to generate an output consistent with the conditioner). In the embodiments described and recited herein, "conditioning input," "conditioning," and "conditioner" may be used interchangeably without departing from the intended meaning and / or scope.

[0027] As referred to herein, "conditioning signal", "control signal" may refer to a signal of a machine learning model that controls the process of encoding or embedding a given input into a meaningful representation that can guide the generation process of the machine learning model. An example control signal is a hint embedding (e.g., a text embedding) that can capture semantic aspects of the input, help shape the characteristics of the generated image, and control the visual effect of the image.

[0028] It should be understood that the text prompt can be a conditioner for the text-to-image model to generate an image that is semantically consistent with the text prompt, such as by optimizing a latent vector or a generator to maximize the similarity between the text prompt and the image. That is, the text-to-image model can generate an image (e.g., a text prompt) that is conditioned according to or consistent with the conditioner.

[0029] It should be understood that an object (e.g., an image, etc.) may include metadata (such as keywords, tags, or descriptions associated with the object) and non-metadata such as features of the object (e.g., color, shape, texture, elements or components or parts, location of elements, or any other information that can be derived from the object itself) instead of metadata. As referred to herein, the "content" of an object may refer to the non-metadata information of the object (e.g., elements, components, parts, and / or classes of the object, etc.).

[0030] As referred to herein, "style" may refer to the visual appearance or unique visual characteristics that can distinguish one image from another. Image style may encompass various elements, such as color, texture, pattern, or overall visual composition. The Gram matrix is ​​a commonly used style representation for capturing the style of an image. The Gram matrix can be derived from a feature map of a convolutional neural network (CNN). It should be understood that the style of an object (e.g., an image) is a subjective concept and may be represented by various style representations other than the Gram matrix.

[0031] As referred to herein, a "content regulator" may refer to a content input (e.g., content of an object corresponding to a text input, etc.) for which the operation of a process of a machine learning model is regulated (e.g., so as to generate output content consistent with the content regulator). For example, in a pre-trained DPM (e.g., a diffusion-based generative model of text to image), a content regulator may be injected or applied or added, such as by performing a denoising process to generate an image regulated according to or consistent with the content regulator. It should be understood that a text prompt may be a content regulator of a diffusion-based generative model of text to image, such as by optimizing a latent vector or generator to maximize the similarity between the text prompt and the content of the image to generate image content consistent with the text prompt. That is, a diffusion-based generative model of text to image may generate content of an image (e.g., a text prompt) regulated according to or consistent with the content regulator.

[0032] As referred to herein, "similarity" may refer to a numerical value representing how close two objects are when comparing two objects (e.g., two images, two concepts corresponding to respective objects, etc.). It should be understood that the similarity between two objects can be determined using techniques such as, for example, sum of squared differences, mutual information, normalized mutual information, mutual correlation, etc. In an example embodiment, the higher the similarity (or value), the more similar the two objects are in context. In such an embodiment, a similarity (or value) of "0" may indicate that the two objects are completely different. It should be understood that in another example embodiment, the lower the similarity (or value), the more similar the two objects are in context. In such an embodiment, a similarity (or value) of "0" may indicate that the two objects are identical.

[0033] As referenced herein, "backpropagation" (backpropagation, backpropagate, backpropagating) may refer to an algorithm for training a feedforward neural network in a machine learning model. When fitting a neural network, backpropagation may calculate the gradient of a loss function relative to the parameters of the neural network (e.g., network weights) for an input-output example. "Loss" or "loss function" may refer to a measurement of the difference or distance between the predicted output of a model and a target value or a reference value. The loss may quantify the performance of the model on a given task and serve as a guide for adjusting model parameters during the training / optimization / customization process of the model. Various loss functions may include, for example, mean square error (MSE) loss, binary cross entropy loss, categorical cross entropy loss, reconstruction loss, etc. The choice of loss function may depend on the specific task in the desired scenario or application.

[0034] Figure 1 is a schematic diagram of an example object generation system arranged in accordance with at least some embodiments described herein. In an example embodiment, system 100 may be an object generation system using a diffusion model, which may include a custom diffusion generative model.

[0035] The system 100 may include terminal devices 110, 120, 130 and 140, a network 160 and a server 150 (eg, a server for providing a customized diffusion generative model). Figure 1 Only an illustrative number of terminal devices, networks, and servers are shown. The embodiments described herein are not limited to the described number of terminal devices, networks, and / or servers. That is, the number of terminal devices, networks, and / or servers described herein is provided for illustrative purposes only and is not intended to be limiting.

[0036] According to at least some example embodiments, terminal devices 110, 120, 130, and 140 may be various electronic devices. Various electronic devices may include, but are not limited to, mobile devices such as smartphones, tablet computers, e-book readers, laptop computers, desktop computers, and / or any other suitable electronic devices.

[0037] According to at least some example embodiments, the network 160 may be a medium used to provide a communication link between the terminal devices 110, 120, 130, 140 and the server 150. The network 160 may be the Internet, a local area network (LAN), a wide area network (WAN), a local interconnect network (LIN), a cloud, etc. The network 160 may be implemented by various types of connections, such as a wired communication link, a wireless communication link, a fiber optic cable, etc.

[0038] According to at least some example embodiments, server 150 may be a server for providing various services to users using one or more of terminal devices 110, 120, 130, such as providing a customized diffusion generative model, etc. Server 150 may be implemented by a distributed server cluster including a plurality of servers, or may be implemented by a single server.

[0039] The user may use one or more of the terminal devices 110, 120, 130, and 140 to interact with the server 150 via the network 160. Various applications or their localized interfaces, such as data query services, etc., may be installed on the terminal devices 110, 120, 130, and 140.

[0040] It should be understood that the software application or service according to the embodiments described herein and / or the service provided by the object generation system may be executed by the server 150 and / or the terminal devices 110, 120, 130, and 140 (which may be referred to as user devices in this article). Therefore, the apparatus for the software application and / or service may be arranged in the server 150 and / or in the terminal devices 110, 120, 130, and 140.

[0041] It should also be understood that when the service is not performed remotely, the system 100 may not include the network 160 , but may include only the terminal devices 110 , 120 , 130 , and 140 and / or the server 150 .

[0042] It should also be understood that the terminal devices 110, 120, 130 and 140 and / or the server 150 may each include one or more processors, memory and a storage device storing one or more programs. The terminal devices 110, 120, 130 and 140 and / or the server 150 may also each include an Ethernet connector, a wireless fidelity receiver, etc. One or more programs, when executed by one or more processors, may cause one or more processors to perform the (multiple) methods described in any embodiment described herein. Moreover, it should be understood that a computer-readable non-volatile medium may be provided according to the embodiments described herein. The computer-readable medium stores a computer program. The computer program, when executed by a processor, is used to perform the (multiple) methods described in any embodiment described herein.

[0043] Figure 2 An example system 200 is illustrated in which a custom diffusion model may be implemented and / or an object may be generated using a custom diffusion model, arranged in accordance with at least some embodiments described herein.

[0044] System 200 may include source 210 and generator 220. In an example embodiment, source 210 may be an electronic device (e.g., Figure 7700, etc.), including but not limited to a camera, an audio and / or video recorder, a smart phone, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable electronic device. In another example embodiment, source 210 may be a storage device, a database, a file, etc. Source 210 may provide input 280 to generator 220. In an example embodiment, generator 220 may be an electronic device (e.g., Figure 7 700, etc.), including but not limited to a camera, an audio and / or video recorder, a smart phone, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable electronic device. In another example embodiment, the generator 220 can be a function, an operation, an action, an algorithm, an application, etc. In an example embodiment, the input 280 can include text, images, audio, video, etc., which are captured, generated, obtained, user input, etc. via the source 210. In another example embodiment, the input 280 can include text, images, audio, video, etc. stored in the source 210.

[0045] Generator 220 may receive input 280 from source 210 and generate an object based on received input 280 (see, e.g. Figure 3 In an example embodiment, the generator 220 includes a model 230 and a control logic 260. The generator 220 may be hosted, implemented, executed, and / or stored on a device (e.g., Figure 7 700, etc.). It should be understood that although system 200 is illustrated as discrete components, various components may be divided into more components, combined into fewer components, or eliminated altogether while being contemplated within the scope of the disclosed subject matter. Those skilled in the art will appreciate that each function and / or operation of a component may be implemented individually and / or collectively by various hardware, software, firmware, or any combination thereof.

[0046] In an example embodiment, control logic 260 may include a number of operations or functions (see, e.g., Figure 4A , Figure 5A and Fig. 6A ) to customize the model 230 and use the customized model 230 to generate an object based on the received input 280. The model 230 can be a machine learning model, such as a pre-trained DPM (e.g., a diffusion-based generative model for text to image). The model 230 includes a denoising module 250 and an optional diffusion module 240.

[0047] In an example embodiment, diffusion module 240 may include a plurality of diffusion operations or functions (see, e.g. Figure 3), which when implemented or executed repeatedly adds noise (e.g., Gaussian noise, etc.) to an input object (e.g., input 280, etc.) to generate a diffusion object, so that the object gradually becomes Gaussian noise after a certain number of operations or iterations. The denoising module 250 may include multiple denoising operations or functions (see, e.g., Figure 3 , Figure 4A , Figure 5A and Fig. 6A ) to repeatedly denoise given a Gaussian noise input (e.g., a diffusion object) to generate a denoised output.

[0048] Figure 3 Illustrate the use of an arrangement according to at least some embodiments described herein Figure 2 Example process 300 for generating an object based on the model 230 of the system 200. Figure 2 As discussed in , model 230 can be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) that includes a denoising module 250 and an optional diffusion module 240.

[0049] In an example embodiment, the diffusion module 240 may perform a diffusion process (also referred to as a diffusion process). Figure 3 The diffusion process starts at process node "x0". The process nodes are labeled 310A-310T. In the example embodiment, process node 310A x0 receives image 340 as input. Image 340 can be an image obtained from a real data distribution or randomly sampled, or can be an image obtained from a real data distribution or randomly sampled. Figure 2 In an example embodiment, the diffusion process may include a process from process node x0 to process node 310T x T (x0→x T ). In some example embodiments, the number T may be one, at least two, 50, 1000, or any other number. Starting at x0, the diffusion process may add noise, such as Gaussian noise, to the image 340 (e.g., to corrupt the image 340) to generate a diffused or corrupted image (e.g., at x1). The diffusion process may repeatedly add noise to the diffused image generated at the previous process node, and at x1. k-1 At x, the diffusion image becomes image 330, which is a partially diffused or corrupted image. The diffusion process can continue to repeatedly add noise to the diffusion image generated at the previous process node, and at x T At , the diffusion image becomes 320 (Gaussian noise image). That is, the diffusion process can repeatedly add noise to the image 340 to generate a diffusion image, whereby the image gradually becomes Gaussian noise (x0→x0) after T operations or iterations. T ).like Figure 3 As shown in k-1 To process node x k The diffusion process is represented by q(x k |x k-1 ).

[0050] In an example embodiment, the denoising module 250 may perform a denoising process. Figure 3 The solid arrows in FIG. 3 illustrate an exemplary random denoising process. The denoising process starts at process node 310T“x T "Start. In the example embodiment, the process node 310Tx T An image 320 is received as input. In an example embodiment, the denoising process may include starting from process node 310T x T To process node 310A x0(x T →x0) for a number of operations or iterations. Image 320 may be a random noise image (e.g., a Gaussian noise image). From 310T x T Initially, a denoising process may denoise (e.g., remove or filter noise, such as Gaussian noise) an image to generate a denoised natural image (e.g., at x0). The denoising process may denoise an image from a previous process node and generate a denoised natural image at x0. k-1 At x0, the denoised image becomes image 330, which may be a partially denoised image. The denoising process may continue to repeatedly denoise the image from the previous process node, and at x0, the denoised image becomes 340 (a denoised natural image substantially free of Gaussian noise, or a clean image). As referred to herein, a clean object (e.g., a clean image) may refer to a denoised object that is substantially free of noise. That is, the denoising process may repeatedly denoise the random noise image 320 to perform T operations or iterations (x0). T →x0) to generate the final or clean image 340. Figure 3 As shown in k To process node x k-1 The denoising process (e.g., based on x k Predict x k-1 ) is represented as p θ (x k-1 |x k ).

[0051] In one example embodiment, the diffusion module 240 may perform a diffusion process represented by the following equation (1):

[0052] dx t =f(t)x t dt+g(t)dw t,x0~q0(x),t∈,0,T-……(1)

[0053] Where q0 represents the unknown d-dimensional data distribution, x t is the state of data distribution at time t, w t is a standard Wiener process (Brownian motion), f(t)x t is called x t The vector-valued function of the drift coefficient, g(t) is a scalar function and is called the diffusion coefficient. t The marginal distribution of can be approximated by the normal distribution In this way, the diffusion process can be expressed as *x(t)+ t∈,0,T- .

[0054] The denoising module 250 may be implemented by solving a deterministic probability flow ordinary differential equation (ODE) rather than by a stochastic denoising process (e.g., as in Figure 3 In p θ (x k-1 |x k ) represents) to perform the denoising process. "Probability flow ODE" is a technical term that can define the mathematical framework work in probability modeling to describe probability density flow. In an example embodiment, when the diffusion process is formulated by the above equation (1), the deterministic probability flow ODE can be expressed as the following equation (2):

[0055]

[0056] where x t Indicates the state or object at time point t, f(t)x t The symbol is called x t The vector-valued function of the drift coefficient, g(t) denotes a scalar function called the diffusion coefficient, and ∈ θ It is used to predict the input x at time point t. t The denoising model / function of the noise to be removed.

[0057] In an example embodiment, the model 230 can perform a denoising process to generate a clean object by solving the above equation (2). Such a denoising process can be referred to as a forward generation process 350. The denoising process can be performed at time point t=T using the starting noise as input (e.g., the starting noise image 320) from the starting process node x T Start, and repeatedly process the nodes x from the previous process at time t TFor example, an intermediate object (e.g., partially denoised image 330) may be obtained at time point t. At the final process node x0, the denoised image becomes a clean object (e.g., denoised natural image 340 substantially free of Gaussian noise) at time point t=0. That is, the generation of the clean object may include generating a final image after completing iterative denoising of the initial noise by executing the forward generation process of the diffusion model 230. Figure 3 As shown in the embodiment in , according to an embodiment, the forward generation process 350 with the starting noise 320 can generate the final image (ie, clean object) 340 by solving the deterministic probability flow ODE which can be expressed as equation (2) above.

[0058] In an example embodiment, the control logic 260 can control the model 230 to perform a forward generation process by repeatedly denoising the initial or starting noise. The forward generation process can be formatted as solving an ODE initial value problem, which includes the steps of reparameterizing the initial or starting noise, reparameterizing the time step, and solving the forward generation ODE to generate a clean object. As referenced herein, "reparameterizing" (re-parameterizing, reparameterizing, re-parameterize, reparameterize) is a technical term that can refer to a technique used to convert an ODE into a new form that can facilitate numerical calculations (e.g., integrals, gradient calculations, etc.).

[0059] Using the generated clean object, the control logic 260 can customize the diffusion model 230 by optimizing one or more parameters of the diffusion model 230. Such optimization can be obtained by minimizing the loss function L based on the generated clean object. The one or more parameters of the diffusion model 230 may include, for example, the network weight θ, the regulation / control signal c, and the initial noise x at time point t. t In one example embodiment, the minimization can be expressed as where ψ denotes one or more parameters of the diffusion model, L denotes the T ,c,∈ θ ) is the loss function of the generated object, and ∈ θ Identify the denoising model.

[0060] To solve the optimization problem, the control logic 260 may implement a back propagation (BP) process to calculate the gradient of the loss function L(x0) relative to one or more parameters of the diffusion model based on the generated clean object (e.g., the denoised natural image 340 without Gaussian noise at time point t=0).

[0061] In an example embodiment, the control logic 260 can perform back-propagation of the gradient of the loss function by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method. "Augmented adjoint ODE" is a technical term that can extend the probability flow ODE framework by incorporating augmented states and introducing additional variables (e.g., adjoints). "Adjoint sensitivity method" is a technical term that is a technique used to efficiently calculate the gradient of an ODE solution. An example of an augmented adjoint ODE can be expressed as the following equation (3):

[0062]

[0063] where x t Indicates the state or object at time point t, f(t)x t The symbol is called x t The vector-valued function of the drift coefficient, g(t) denotes a scalar function called the diffusion coefficient, and L denotes the function based on the term Φ(x T ,c,∈ θ ), θ represents the network weight, σ t denotes the standard variant of the normal distribution, and ∈ θ It is used to predict the input x at time point t. t The denoising model of the noise to be removed.

[0064] In one example embodiment, the control logic 260 may reparameterize the augmented adjoint ODE to reduce the numerical error in solving the augmented adjoint ODE. For example, the augmented ODE may be reparameterized into a simple non-stiff ODE using exponential integration. As an example, the above equation (3) may be reparameterized as follows, denoted by and ρ = γ(t), where γ(t) can increase monotonically as t increases from 0 to T. The original augmented adjoint ODE (e.g., Equation 3) can be reparameterized as the following Equation (4):

[0065]

[0066] The control logic 260 can obtain the gradient of the loss function by solving the inverse mode or inverse time, reparameterized augmented adjoint ODE (e.g., equation (4)). Here, "inverse mode or inverse time" is used with respect to the reverse time series relative to the "forward generation process" (e.g., from time point t = 0 to t = T, or from time point t = T to t = 0, respectively). The control logic 260 can optimize the one or more parameters of the diffusion model 230 by back-propagating the gradient of the obtained loss function relative to the one or more parameters to minimize the loss function and obtain a customized diffusion model with optimized parameters (e.g., regulation / control signals, network weights, or initial or starting noise).

[0067] In an example embodiment, the control logic 260 can control the model 230 to perform a back-propagation process to back-propagate the gradient of the loss to optimize one or more parameters of the model 230, using the denoising model, the final state, and the loss gradient of the final state as input. The back-propagation process can be formatted as a reverse mode derivative of solving an ODE initial value problem, including the steps of defining an initial augmented state, defining dynamics about the augmented state, cascading time derivatives, solving the reverse time ODE, and returning the loss gradient for optimizing one or more parameters of the model.

[0068] Features in the embodiments described herein can provide a method for customizing a diffusion model to obtain a custom model based on a computable and differentiable metric. Example metrics may include style representation, visual effects, or content categories according to a classifier. The metric can be evaluated on one or more clean objects, which can be generated using a diffusion model by performing a forward generation process. In other words, the loss of each clean object can be determined relative to a metric of, for example, a reference object. The loss can include, for example, a style loss, a feature loss, or a classification loss. The one or more parameters can be optimized by back-propagating the gradient of the one or more parameters of the diffusion model relative to minimize the loss and obtain a custom model with optimized one or more parameters. The one or more parameters may include network weights, adjustment / control signals, or initial or starting noise, which can be optimized by minimizing the corresponding loss (e.g., style loss, feature loss, or classification loss).

[0069] In one example embodiment, the forward generation process can be performed by solving the probability flow ODE (e.g., equation (2) above), and the back propagation of the gradient of the loss can be performed by solving the augmented adjoint ODE based on the adjoint sensitivity method (e.g., equation (3) above). In order to reduce numerical errors in the forward generation process and the gradient back propagation process, the probability flow ODE and the augmented ODE can each be reparameterized into a simple non-stiff ODE using exponential integration (e.g., see the reparameterization equation (4) above).

[0070] Figure 4A 400 is a flowchart illustrating an example process flow 400 for customizing a diffusion model to generate a styled object according to at least some embodiments described herein. It should be understood that unless otherwise indicated, the process flow 400 disclosed herein may be executed by one or more processors (e.g., Figure 7 CPU 705, or any other suitable processor), Figure 1 one or more components of Figure 2 The processor can be implemented with one or more modules. Figure 2 Communicates with other components of system 200.

[0071] It should also be understood that the process flow 400 may include one or more operations, actions, or functions as illustrated in one or more of blocks 410, 420, 430, 440, and 450. These various operations, functions, or actions may correspond, for example, to software, program code, or program instructions executable by a processor that cause the functions to be performed. Although illustrated as discrete blocks, obvious modifications may be made, for example, two or more blocks may be reordered; additional blocks may be added; and various blocks may be divided into more blocks, combined into fewer blocks, or eliminated, depending on the desired implementation. The process flow 400 may begin at block 410.

[0072] At block 410 (obtain target or reference style), the processor may obtain, for example, Figure 2 The source 210 obtains or receives a target or reference style representation representing the style of the target or reference object. It should be understood that the process at block 410 can be performed in, for example, Figure 2 The target or reference style may be defined by a reference Gram matrix, which may represent the style of the target or reference object. For example, the target or reference image (e.g., Figure 4B 405) obtains a target or reference style representation (e.g., a Gram matrix). It should be understood that the style of the image can be represented by any suitable representation other than the Gram matrix. Processing can proceed from block 410 to block 420.

[0073] At block 420 (Generate Clean Objects), the processor may generate one or more clean objects by performing a forward generation process of the diffusion model. In an example embodiment, the generation of the clean objects may include generating a final image after completing iterative denoising of the starting noise by performing the forward generation process of the diffusion model. For example, the denoising module 250 of the model 230 may perform a forward generation process 350 using the starting noise 320 to generate the final image 340, as shown in FIG. Figure 2 and Figure 3 . In an example embodiment, the diffusion model can be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) having as an input a content conditioner to generate clean objects. The clean objects can be generated using a forward generation process 350 with multiple pairs of starting noise 320 and text prompts. For example, a user can input a text prompt (e.g., "apple") as a content conditioner to a pre-trained diffusion-based generative model of text to image to generate image content consistent with the text prompt. Processing can proceed from box 420 to box 430.

[0074] At box 430 (determine style loss), the processor can determine the style loss of each clean object relative to the reference object. In an example embodiment, the style loss of the Gram matrix of each clean object can be determined relative to the reference Gram matrix of the reference object. In an example embodiment, the style loss and content loss of each clean object can be combined and determined relative to the reference object. Content loss can refer to a measurement of the difference or distance between the features of the clean object and the features of the target or reference object. A pre-trained convolutional neural network (CNN) can be used to extract features of the image. A loss function of mean square error (MSE) can be used to determine style loss and content loss. It should be understood that other suitable loss functions can be used. Processing can proceed from box 430 to box 440.

[0075] At box 440 (Optimize Network Weights of Diffusion Model), the processor may optimize the network weights of the diffusion model by back-propagating the gradients of the style loss relative to the network weights to minimize the style loss and obtain a customized diffusion model with optimized network weights. In an example embodiment, a combination of the style loss and the content loss may be minimized to optimize the network weights of the diffusion model. In an example embodiment, back-propagation of the gradients of the style loss may be performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method (e.g., see augmented adjoint ODE, Equation (3) described above). The augmented adjoint ODE may be re-parameterized to reduce numerical errors in solving the augmented adjoint ODE (e.g., see re-parameterized augmented adjoint ODE, Equation (4) described above). Processing may proceed from box 440 to box 450.

[0076] At box 450 (generate stylized object), the processor can use the customized diffusion model with optimized network weights to generate a stylized object. The stylized object can have a style similar to the style of the reference object. That is, the stylized object can have a visual appearance or unique visual characteristics substantially the same as the reference object. For example, when a user inputs a text prompt (e.g., "apple") as a content modifier to the customized diffusion model, the customized diffusion model is customized to generate an object with a target or reference style representation (e.g., a Gram matrix), and the customized diffusion model can generate image content that is consistent with the text prompt (e.g., "apple") and has a visual appearance or unique visual characteristics substantially the same as the reference object (e.g., a painting by Vincent Van Gogh).

[0077] Figure 4B An example of a method for using a computer program product arranged in accordance with at least some embodiments described herein is shown. Figure 4A An example process of generating various stylized objects using a customized diffusion model obtained by the process flow 400. The customized diffusion model with optimized network weights can generate an object having a style similar to the target or reference style (eg, from a target or reference image).

[0078] exist Figure 4B In the embodiment depicted in , images 402, 412, 422, and 432 are generated using a diffusion model, which may be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) with input as a content modifier, before optimizing the network weights of the diffusion model. For example, a user may input a text prompt (e.g., "apple") as a content modifier to a pre-trained diffusion-based generative model of text to image to generate image content consistent with the text prompt. The generated images 402, 412, 422, and 432 may not have a target or reference style of reference images 403 and 405. That is, images 402, 412, 422, and 432 generated before optimizing the network weights of the diffusion model may not have the visual appearance or unique visual characteristics of the target or reference object.

[0079] Can be used Figure 4AThe processing flow 400 of the embodiment of the present invention is used to customize the diffusion model so that the customized model can generate a stylized object with a style similar to (e.g., substantially the same as) the style of the reference object. For example, when the user inputs the same text prompt (e.g., "apple") as a content modifier to the customized diffusion model with optimized network weights, images 404, 414, 424, and 434 are generated, which correspond to images 402, 412, 422, and 432 generated before the optimized network weights. The generated images 404, 414, 424, and 434 have the corresponding target or reference style of reference images 403 and 405. That is, images 404 and 414 each have the visual appearance or unique visual characteristics of target or reference image 403, and images 424 and 434 each have the visual appearance or unique visual characteristics of target or reference image 405. It should be understood that the customized diffusion model can generate a stylized object regardless of whether the content of the object (e.g., "apple") has been used to train or customize the diffusion model (i.e., "seen category" or "unseen category"). It should also be appreciated that a customized diffusion model can generate stylized objects regardless of whether the starting noise has been used to train or customize the diffusion model (ie, "seen noise" or "unseen noise").

[0080] Figure 5A 500 is a flowchart illustrating an example process flow for customizing a diffusion model to generate an object with a visual effect according to at least some embodiments described herein. It should be understood that unless otherwise stated, the process flow 500 disclosed herein can be executed by one or more processors (e.g., Figure 7 CPU 705, or any other suitable processor), Figure 1 one or more components of Figure 2 The processor can be implemented with one or more modules. Figure 2 Communicates with other components of system 200.

[0081] It should also be understood that the process flow 500 may include one or more operations, actions, or functions as illustrated in one or more of blocks 510, 520, 530, 540, and 550. These various operations, functions, or actions may correspond, for example, to software, program code, or program instructions executable by a processor that cause functions to be performed. Although illustrated as discrete blocks, obvious modifications may be made, for example, two or more blocks may be reordered; additional blocks may be added; and various blocks may be divided into more blocks, combined into fewer blocks, or eliminated, depending on the desired implementation. The process flow 500 may begin at block 510.

[0082] At block 510 (obtain reference object with visual effect), the processor may obtain, for example, Figure 2The reference object having the predetermined visual effect is obtained or received from the source 210 of the image processing unit. In one example embodiment, the reference object may be obtained by manually editing the image to add the predetermined visual effect. In another example embodiment, a generative model may be used to generate the reference object having the target or reference visual effect. It should be understood that the process at block 510 may be performed in, for example, Figure 2 Processing may proceed from block 510 to block 520.

[0083] At block 520 (Generate Clean Objects), the processor may generate one or more clean objects by executing a forward generation process of the diffusion model with a control signal. The control signal is configured to control the visual effect of the clean objects. In an example embodiment, the generation of the clean objects may include generating a final image after completing iterative denoising of the starting noise by executing the forward generation process of the diffusion model. For example, the denoising module 250 may perform the forward generation process 350 with the starting noise 320 to generate the final image 340, such as Figure 2 and Figure 3 In one example embodiment, the diffusion model can be a pre-trained DPM (e.g., a diffusion-based generative model for text to image) having a first input as a content conditioner and a second input as a visual effect conditioner to generate clean objects. The visual effect conditioner can be represented as a text embedding, such as Figure 5B denoted by “#” in the image. Text embeddings can be combined (e.g., cascaded) with embeddings of other text prompts to generate a clean object. For example, a diffusion model can receive a first text prompt (e.g., “cute chinchillas in the yard”) as a content regulator and a second text prompt (e.g., “bokeh”) as a visual effect regulator to a pre-trained diffusion-based generative model for text to image, and the corresponding embeddings can be cascaded to generate a clean image. A reference object and a clean object can be obtained using a pre-trained diffusion-based generative model for text to image with the same first input (e.g., “cute chinchillas in the yard”). Processing can proceed from box 520 to box 530.

[0084] At block 530 (Determine Feature Loss), the processor may determine the feature loss of each clean object relative to a reference object having a predetermined visual effect. The feature loss may measure the difference or distance between the visual effect of the clean object and the predetermined visual effect of the reference object. The feature loss may be determined using any suitable loss function, such as, for example, a mean square error (MSE) loss or an L2 loss, or a perceptual loss. For example, the feature loss between a clean object and a reference object having the same content (e.g., "a cute puppy in the yard") may be determined by calculating an L2 loss. Processing may proceed from block 530 to block 540.

[0085] At box 540 (optimizing control signal for diffusion model), the processor may optimize the control signal by back-propagating the gradient of the feature loss relative to the control signal to minimize the feature loss and obtain a customized diffusion model with an optimized control signal. In an example embodiment, the feature loss may be minimized to optimize a text embedding “#” representing a visual effect modifier. For example, back-propagation may be used to minimize the feature loss between one or more pairs of clean objects and corresponding reference objects (e.g., “cute puppy in the yard”) to optimize a text embedding “#” representing a visual effect modifier (e.g., “bokeh”). In an example embodiment, back-propagation of the gradient of the feature loss may be performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method (e.g., see augmented adjoint ODE, equation (3) as described above). The augmented adjoint ODE may be re-parameterized to reduce the numerical error of solving the augmented adjoint ODE (e.g., see re-parameterized augmented adjoint ODE, equation (4) as described above). Processing may proceed from box 540 to box 550.

[0086] At box 550 (Generate Object with (multiple) visual effects), the processor can generate the object using a customized diffusion model with optimized control signals. The generated object has a visual effect that is similar to (e.g., substantially the same as) a predetermined visual effect of a reference object. In an example embodiment, the customized diffusion model can be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) with an optimized text embedding "#" representing a visual effect modifier (e.g., "bokeh"). The optimized text embedding allows the model to transfer the text cue (e.g., "bokeh", as a visual effect modifier) ​​to the predetermined visual effect of the reference object.

[0087] Figure 5B An example of a method for using a computer program product arranged in accordance with at least some embodiments described herein is shown. Figure 5A An example process of generating an object with a visual effect using a customized diffusion model obtained by the process flow 500. The customized diffusion model with the optimized control signal can generate an object with a visual effect similar to (eg, substantially the same as) a predetermined target visual effect of a reference object.

[0088] The predetermined target visual effect may include any desired visual effect of an image, such as, for example, a bokeh effect, a heavy lighting effect, a Van Gogh style, a black and white effect, etc. The customized diffusion model described herein may combine the optimized control signal with various conditioning inputs or content conditioners (e.g., textual cues) to generate an object with substantially the same visual effect.

[0089] exist Figure 5BIn the embodiment depicted in , a diffusion model is used to generate images 502, 512, and 522 before optimizing the control signal, which can be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) with input as a content modifier (e.g., a text prompt). For example, a user can select an image labeled C base The textual prompts are used as content regulators to input into the pre-trained text-to-image diffusion-based generative model to generate image content consistent with the textual prompts. Figure 5B As shown in , images 502, 512, and 522 show various contents consistent with corresponding text prompts, such as "A chinchilla in the yard," "A cute puppy in the yard," "Fallout concept art school interior rendering grim, Unreal Engine 5," "A cat running in the dark," "A photo of a city street in the 1970s," and "A photo of a city square in the 1970s."

[0090] A target or reference visual effect may be added to the generated images 502, 512, and 522 to form target or reference images 504, 514, and 524, respectively. Figure 5B shown in target The target or reference visual effect C may correspond to the visual effects of "bokeh", "purple neon lighting" and "Van Gogh style". In one embodiment, the target or reference visual effect C may be converted to a "blurred" image by editing the image. target Manually added to the target or reference image. For example, images 502, 512, and 522 may be edited by a professional photographer to add one or more target or reference visual effects to improve aesthetic quality, including, for example, color, tint, bokeh, etc. In another embodiment, a generative model may be used to add textual hints to the C base and corresponds to the target or reference visual effect C target The effect tag cascades to generate a target or reference visual effect C target of the image.

[0091] By using Figure 5A The processing flow 500 is used to customize the diffusion model with the optimized control signal, and the target or reference visual effect can be extracted from the target or reference images 504, 514 and 524. For example, when the user enters the text prompt C base As a content modifier As a content modifier and set the text prompt C target(e.g., “bokeh,” “purple neon lighting,” and “Van Gogh style”) as control signals input to the custom diffusion model, images 506, 516, and 526 are generated by the custom diffusion model with the corresponding target or reference visual effects of the reference images 504, 514, and 524. It should be understood that the custom diffusion model can generate objects with (multiple) predetermined visual effects regardless of whether the object content (e.g., “chicken cat”) has been used to train or customize the diffusion model (i.e., “different C base ”). It should also be understood that the customized diffusion model can generate stylized objects regardless of whether the starting noise has been used to train or customize the diffusion model (i.e., “seen noise” or “unseen noise”). For example, a “bokeh” effect can be extracted from a pair of “chicken” images 502 and 504 by customizing a diffusion model with an optimized control signal corresponding to the textual hint “bokeh”. The extracted “bokeh” effect can be used to generate an image 506 with various content categories such as “chicken”, “dog”, etc. The generated image 506 has a “bokeh” effect that is substantially the same as the effect in the target or reference image 504, such as Figure 5B As shown in . Therefore, customized diffusion models with optimized control signals can be used for novel scene generation.

[0092] Fig. 6A 600 is a flowchart illustrating an example process flow for customizing a diffusion model to provide adversarial examples to audit the security of an object generation system according to at least some embodiments described herein. It should be understood that unless otherwise stated, the process flow 600 disclosed herein can be executed by one or more processors (e.g., Figure 7 CPU 705, or any other suitable processor), Figure 1 One or more components or Figure 2 The processor can be implemented with one or more modules. Figure 2 Communicates with other components of system 200.

[0093] It should also be understood that the process flow 600 may include one or more operations, actions, or functions as illustrated in one or more of blocks 610, 620, 630, 640, 650, and 660. These various operations, functions, or actions may correspond, for example, to software, program code, or program instructions executable by a processor that cause functions to be performed. Although illustrated as discrete blocks, obvious modifications may be made, for example, two or more blocks may be reordered; additional blocks may be added; and various blocks may be divided into more blocks, combined into fewer blocks, or eliminated, depending on the desired implementation. The process flow 600 may begin at block 610.

[0094] At block 610 (receive diffusion model input), the processor may obtain, for example, Figure 2 The source 210 obtains or receives the input of the diffusion model. It should be understood that the process at block 410 can be performed in, for example, Figure 2 is implemented in the control logic 260 of the generator 220. The input may include a text prompt as a content regulator and a starting noise with an adversarial perturbation. The text prompt may correspond to a reference content category identified by a classifier. The adversarial perturbation may be limited to have a sufficiently small specification so that the generated clean object with the adversarial perturbation is visually similar to the clean object without the adversarial perturbation. The specification of the adversarial perturbation may refer to the scale or amplitude of the change applied to the starting noise. It should be understood that the choice of specification may depend on the specific implementation and requirements of the diffusion model. Processing may proceed from box 610 to box 620.

[0095] At box 620 (generate clean objects), the processor may generate one or more clean objects by performing a forward generation process of a diffusion model having as input a content modifier and a starting noise with a disturbance. For example, the diffusion model may receive a text prompt of "starfish" and perform a forward generation process 350 using the starting noise 320 with an adversarial disturbance to generate a clean object 340. When adversarial perturbations are not applied, the diffusion model may generate a reference object by performing a forward generation process of the diffusion model using a text prompt (e.g., "starfish"). The reference object may be identified by a classifier as a reference content category (e.g., "starfish"). Processing may proceed from box 620 to box 630.

[0096] At box 630 (Determine Classification Score), the processor can determine the classification score for each clean object via a classifier. A classifier can refer to a model or algorithm that can assign a label or class to an input image based on the visual content of the input image. The classifier can provide a classification score for the input image, which represents the confidence or likelihood that the input image belongs to each category. A higher score can indicate that the image has a higher probability of belonging to a particular category. That is, the classification score can be interpreted as a measure of the confidence of the classifier in its prediction. Processing can proceed from box 630 to box 640.

[0097] At box 640 (determine classification loss), the processor may determine the classification loss of the classification score relative to the target content category. The target content category is different from the reference content category. In an example embodiment, the classifier may be included by a content review filter, which may automatically filter or adjust the generated content to ensure compliance with specific criteria, policies, or legal requirements. The classifier of the content review filter may classify and / or label the content into different categories or classes based on predefined criteria or rules. For example, the classifier may assign content to categories such as safe, unsafe, harmful, harmless, etc. The classification loss may measure the difference or distance between the classification score predicted or determined by the classifier and the target content category (e.g., according to the "safe" category label of a not suitable for viewing in office places (NSFW) filter). The choice of classification loss function may depend on the specific problem and the nature of the class probability. In an example embodiment, the classification loss function may include a cross entropy loss or an embedding similarity loss. Processing may proceed from box 640 to box 650.

[0098] At block 650 (Optimize Starting Noise for Diffusion Model), the processor may optimize the adversarial perturbation by back-propagating the gradient of the classification loss with respect to the adversarial perturbation of the starting noise to minimize the classification loss of the classification score and obtain a customized diffusion model with an optimized starting noise with the optimized perturbation. In an example embodiment, back-propagation of the gradient of the classification loss may be performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method (e.g., see augmented adjoint ODE, Equation (3) described above). The augmented adjoint ODE may be re-parameterized to reduce numerical errors in solving the augmented adjoint ODE (e.g., see re-parameterized augmented adjoint ODE, Equation (4) described above). Processing may proceed from block 650 to block 660.

[0099] At block 660 (Generate Adversarial Examples), the processor may generate adversarial examples using a customized diffusion model with optimized starting noise that optimizes the adversarial perturbation. The adversarial examples are identified by the classifier as target content categories. In an example embodiment, the customized diffusion model with optimized starting noise may receive a text prompt (e.g., “photos of naked men”) as a content conditioner to generate objects (e.g., Figure 6B 611), which can be used to audit the security of the object generation system. For example, when a content review filter (e.g., an NSFW filter) receives adversarial examples generated by a custom diffusion model, the classifier of the content review filter may mistakenly classify adversarial examples that should be classified as harmful (e.g., Figure 6B The image 611) is classified as a safety category.

[0100] Figure 6BIllustrate the use of an arrangement according to at least some embodiments described herein Fig. 6A 611, 612, 614, 616, and 618 and corresponding comparison examples 601, 602, 604, 606, and 608 generated by a customized diffusion model. The customized diffusion model with optimized starting noise with optimized perturbations can generate objects (e.g., adversarial examples) that may mislead classifiers, such as classifiers included in content review filters (e.g., NSFW filters).

[0101] exist Figure 6B In the embodiment depicted in , a diffusion model is used to generate images 601, 602, 604, 606, and 608 before optimizing the starting noise of the diffusion model, which model can be a pre-trained DPM (e.g., a diffusion-based generative model of text to image) with input as a content modifier. For example, a user can input a text prompt (e.g., "photos of naked men") as a content modifier to a pre-trained diffusion-based generative model of text to image to generate image content (e.g., picture 601) consistent with the text prompt.

[0102] The classifier can correctly classify the content of the generated images 601, 602, 604, 606, and 608. For example, the classifier correctly classifies the content of image 602 as "starfish", the content of image 604 as "candle", the content of image 606 as "Persian cat", and the content of image 608 as "wall clock". The classifier can be included by a content review filter (e.g., an NSFW filter), which can correctly classify the content of image 601 as "harmful".

[0103] Images 611, 612, 614, 616, and 618 are generated by a custom diffusion model with optimized starting noise with optimized perturbations, with the same inputs (e.g., content moderators) as corresponding images 601, 602, 604, 606, and 608. Images 611, 612, 614, 616, and 618 can be provided to a classifier as adversarial examples that may mislead the classifier. For example, the classifier incorrectly classifies the content of image 612 as "teddy bear" instead of "starfish", the content of image 614 as "torch" instead of "candle", the content of image 616 as "angora" instead of "Persian cat", and the content of image 618 as "birdhouse" instead of "wall clock". A content moderation filter (e.g., an NSFW filter) incorrectly classifies the content of image 611 as "safe" instead of "harmful".

[0104] Figure 7 is arranged according to at least some embodiments described herein and is suitable for implementing an electronic device (e.g., Figure 1 The devices 110, 120, 130 and 140 and the server 150, Figure 2 The source device 210 hosts, implements, executes and / or stores Figure 2 220, etc.) of an exemplary computer system 700. It should be understood that Figure 7 The computer system shown in FIG. 1 is provided for illustration purposes only and is not intended to limit the functionality and applicability of the embodiments described herein.

[0105] As depicted, computer system 700 may include a central processing unit (CPU) 705. CPU 705 may perform various operations and processes based on programs stored in read-only memory (ROM) 710 or programs loaded from storage device 740 to random access memory (RAM) 715. RAM 715 may also store various data and programs required for the operation of system 700. CPU 705, ROM 710, and RAM 715 may be connected to each other via bus 720. An input / output (I / O) interface 725 may also be connected to bus 720.

[0106] Components connected to the I / O interface 725 may also include an input device 730 including a keyboard, a mouse, a digital pen, a drawing board, etc.; an output device 735 including a display such as a liquid crystal display (LCD), a speaker, etc.; a storage device 740 including a hard disk, etc.; and a communication device 745 including a network interface card such as a LAN card, a modem, etc. The communication device 745 may perform communication processing via a network such as the Internet, a WAN, a LAN, a LIN, a cloud, etc. In one embodiment, a drive 750 may also be connected to the I / O interface 725. A removable medium 755 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. may be installed on the drive 750 as needed, so that a computer program read from the removable medium 755 may be installed in the storage device 740.

[0107] It should be understood that reference Figure 4A , Figure 5A , Fig. 6A The processes described in the flowcharts and / or other figures may be implemented as computer software programs. A computer program product may include a computer program stored in a computer-readable non-volatile medium. The computer program includes program code for executing the methods shown in the flowcharts and / or processes. In this embodiment, the computer program may be downloaded and installed from a network via a communication device 745, and / or may be installed from a removable medium 755. When the computer program is executed by a central processing unit (CPU) 705, the above-mentioned functions specified by the methods in the embodiments disclosed herein may be implemented.

[0108] It should also be understood that features in the embodiments described herein may provide methods for optimizing one or more parameters of a diffusion model (e.g., a conditioning / control signal, a network weight, or an initial or starting noise) to obtain a customized diffusion model. The customization may be based on a computable and differentiable metric that evaluates a clean object generated by executing a forward generation process of the diffusion model. The computable and differentiable metric may include, for example, differences / distances between classifiers, stylized Gram matrices, and the like.

[0109] It should also be understood that features in the embodiments described herein may provide a method for optimizing one or more parameters of a diffusion model by back-propagating the gradient of the loss relative to the one or more parameters to minimize the loss and obtain a customized diffusion model with optimized one or more parameters. The embodiments described herein provide a gradient back-propagation process that may store intermediate states only at the time point of function evaluation, resulting in constant memory usage.

[0110] In comparison, when naive gradient backpropagation is used to customize the diffusion model, the customization process may require intermediate state storage for all iterations, resulting in significant memory consumption. Figure 3 As shown in , during the random diffusion process, the data distribution (x0) is diffused into Gaussian white noise (x T ), and the random diffusion process is resumed to synthesize new content (x0). Naive gradient backpropagation requires intermediate state storage for all iterations (X k : T operations or iterations (x0→x T )).

[0111] In addition, the customization process using naive gradient back propagation may require multiple reference examples to align the pre-trained diffusion model with the concepts provided by the user. Otherwise, the resulting output image of the customized model may be limited by the content of the input image. For example, a user can customize a commercially available diffusion model (e.g., a pre-trained DPM) to generate a desired image. Such customization of a diffusion model (e.g., a pre-trained DPM) using naive gradient back propagation may require multiple reference examples with the same desired style (e.g., multiple images of Vincent Van Gogh's paintings) as input to customize the parameters of the pre-trained diffusion model, so that the pre-trained diffusion model with customized parameters can generate images with the style of Vincent Van Gogh's paintings. However, when the number of input images with the desired style is limited, the resulting output image not only looks like the style of Vincent Van Gogh's paintings, but is also limited by the content of the input image. For example, when using Vincent Van Gogh's painting "Sunflowers" as input, the customized model can generate images whose content is limited to the content of the input image, such as sunflowers.

[0112] It should be understood that the disclosed and other solutions, examples, embodiments, modules and functional operations described in this document can be implemented in digital electronic circuits or in computer software, firmware or hardware, including in the structures disclosed in this document and their structural equivalents, or in a combination of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, that is, one or more computer program instruction modules encoded on a computer-readable medium for execution by a data processing device or controlling the operation of the data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a material composition that affects a machine-readable propagation signal, or a combination of one or more of them. The term "data processing device" covers all devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. In addition to hardware, the device may also include code that creates an execution environment for the computer program in question, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0113] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one or more computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0114] The processes and logic flows described in this document may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as field programmable gate arrays, application specific integrated circuits, and the like.

[0115] By way of example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operatively coupled to receive data from or transfer data to or both of one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into a dedicated logic circuit.

[0116] It should be understood that different features, variations and multiple different embodiments have been shown and described in various details. What is sometimes described in this application according to a specific embodiment is made only for illustrative purposes, and is not intended to limit or imply that only one or more specific embodiments are envisioned. It should be understood that the present disclosure is not limited to any single specific embodiment or the variations listed. Many modifications, variations and other embodiments will occur to those skilled in the art, and these modifications, variations and other embodiments are intended to be covered by the present disclosure and are in fact covered by the present disclosure. In fact, the scope of the present disclosure should be determined by the appropriate legal interpretation and interpretation (including equivalents) of the present disclosure, as understood by those skilled in the art relying on the complete disclosure presented at the time of submission.

[0117] All aspects:

[0118] It should be understood that any aspects may be combined with each other.

[0119] Aspect 1. A method for generating a stylized object using a diffusion model, the method comprising:

[0120] obtain a reference style representation of a reference object;

[0121] Generate one or more objects by executing a forward generation process of a diffusion model;

[0122] Determine the style loss of each object relative to a reference object;

[0123] Optimizing the network weights of the diffusion model by back-propagating the gradient of the style loss with respect to the network weights to minimize the style loss and obtain a customized diffusion model with optimized network weights; and

[0124] A custom diffusion model with optimized network weights is used to generate a stylized object having a style similar to that of a reference object.

[0125] Aspect 2. The method according to aspect 1, wherein the generating of the object comprises: generating a final image after completing iterative denoising of the initial noise by executing a forward generation process of a diffusion model.

[0126] Aspect 3. The method according to aspect 1 or 2, wherein the diffusion model is a pre-trained text-to-image diffusion-based generative model having as input the content regulator to generate the object.

[0127] Aspect 4. A method according to Aspect 3, wherein the optimization of the network weights also includes minimizing a combination of a style loss and a content loss, the content loss corresponding to features of an object generated by a pre-trained text-to-image diffusion-based generative model with a content regulator.

[0128] Aspect 5. A method according to Aspect 3 or 4, wherein the optimization of network weights further comprises optimizing one or more parameters of a pre-trained diffusion-based generative model of text to image.

[0129] Aspect 6. The method according to any one of aspects 1-5, wherein back-propagation of the gradient of the style loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

[0130] Aspect 7. The method according to Aspect 6 further comprises reparameterizing the augmented adjoint ODE.

[0131] Aspect 8. A method for generating an object with a visual effect using a diffusion model, the method comprising:

[0132] obtaining a reference object having a predetermined visual effect;

[0133] generating one or more objects by performing a forward generation process of a diffusion model with a control signal configured to control a visual effect of the generated object;

[0134] determining a feature loss for each object relative to a reference object having a predetermined visual effect;

[0135] optimizing the control signal by back-propagating the gradient of the feature loss with respect to the control signal to minimize the feature loss and obtain a customized diffusion model with an optimized control signal; and

[0136] A customized diffusion model with optimized control signals is used to generate an object having a visual effect similar to a predetermined visual effect of a reference object.

[0137] Aspect 9. The method according to aspect 8, wherein the generating of the object comprises generating a final image after completing iterative denoising of the initial noise by performing a forward generation process of a diffusion model.

[0138] Aspect 10. The method according to Aspect 8 or 9, wherein the diffusion model is a pre-trained text-to-image diffusion-based generative model having a first input as a content regulator and a second input as a visual effect regulator to generate objects.

[0139] Aspect 11. The method according to aspect 10, wherein the reference object is obtained using a pre-trained text-to-image diffusion-based generative model with the same first input.

[0140] Aspect 12. The method according to aspect 10 or 11, wherein the control signal of the diffusion model is optimized to transfer the second input into a visual effect.

[0141] Aspect 13. A method according to any one of aspects 8-12, wherein back-propagation of the gradient of the feature loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

[0142] Aspect 14. The method according to aspect 13 further comprises reparameterizing the augmented adjoint ODE.

[0143] Aspect 15. A method for providing adversarial examples to audit the security of an object generation system, the method comprising:

[0144] generating one or more objects by performing a forward generation process of a diffusion model having as an input a content regulator and a starting noise having a perturbation, the input corresponding to a reference content category identified by the classifier;

[0145] determining, via the classifier, a classification score for each object;

[0146] determining a classification loss of the classification score relative to a target content category, the target content category being different from the reference content category;

[0147] Optimizing the perturbation of the starting noise by back-propagating the gradient of the classification loss with respect to the perturbation to minimize the classification loss of the classification score and obtain a customized diffusion model with optimized starting noise; and

[0148] A customized diffusion model with optimized starting noise is used to generate adversarial examples that are identified by the classifier as belonging to the target content category.

[0149] Aspect 16. The method according to aspect 15, wherein the generating of the object comprises generating a final image after completing iterative denoising of the initial noise by performing a forward generation process of a diffusion model.

[0150] Aspect 17. The method according to aspect 15 or 16 further comprises generating a reference object by performing a forward generation process with a diffusion model as an input to the content regulator, the reference object being identified as a reference content category by the classifier.

[0151] Aspect 18. A method according to any one of Aspects 15-17, wherein the classifier includes a content review filter.

[0152] Aspect 19. The method according to Aspect 18, wherein the content review filter clarifies adversarial examples that should be classified as harmful categories as safe categories.

[0153] Aspect 20. The method according to any one of aspects 15-19, wherein back-propagation of the gradient of the classification loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

[0154] Aspect 21. The method according to Aspect 20, further comprising reparameterizing the augmented adjoint ODE.

[0155] Aspect 22. A method for customizing a diffusion model to obtain a customized model, the method comprising:

[0156] Generate one or more objects by executing a forward generation process of a diffusion model;

[0157] determining a loss for each object in the object relative to a reference metric; and

[0158] The one or more parameters of the diffusion model are optimized by back-propagating the gradient of the loss with respect to the one or more parameters to minimize the loss and obtain a customized model with the optimized one or more parameters.

[0159] Aspect 23. The method according to Aspect 22, wherein the loss comprises a style loss, a feature loss or a classification loss.

[0160] Aspect 24. The method according to aspect 22 or 23, wherein the reference metric comprises a style representation, a visual effect, or a content category according to a classifier.

[0161] Aspect 25. The method according to aspect 24, wherein the style representation comprises a Gram matrix.

[0162] Aspect 26. The method according to any one of aspects 22-25, wherein the one or more parameters include network weights, control signals or starting noise.

[0163] Aspect 27. The method according to any one of aspects 22-26, wherein back-propagation of the gradient of the loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

[0164] Aspect 28. The method according to Aspect 27 further comprises reparameterizing the augmented adjoint ODE.

[0165] Aspect 29. A non-transitory computer readable medium having stored thereon computer executable instructions, the computer executable instructions, when executed, causing one or more processors to perform operations comprising:

[0166] Generate one or more objects by executing a forward generation process of a diffusion model;

[0167] determining the loss of each object relative to a reference metric; and

[0168] The one or more parameters of the diffusion model are optimized by back-propagating the gradient of the loss with respect to the one or more parameters to minimize the loss and obtain a customized model with the optimized one or more parameters.

[0169] Aspect 30. A generator for generating an object using a diffusion model, the generator comprising:

[0170] a diffusion model for generating one or more objects by performing a forward generation process; and

[0171] Control logic module for:

[0172] determining the loss of each object relative to a reference metric; and

[0173] The one or more parameters of the diffusion model are optimized by back-propagating the gradient of the loss with respect to the one or more parameters to minimize the loss and obtain a customized model with the optimized one or more parameters.

[0174] The terms used in this specification are intended to describe specific embodiments and are not intended to be limiting. Unless otherwise expressly stated, the terms "a", "an" and "the" also include plural forms. When used in this specification, the terms "include" and / or "comprise" specify the presence of stated features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0175] With respect to the foregoing description, it should be understood that changes in detail may be made without departing from the scope of the present disclosure, especially in the construction materials adopted and the shapes, sizes and arrangements of the parts. This specification and the described embodiments are exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for generating a stylized object using a diffusion model, the method comprising: obtain a reference style representation of a reference object; generating one or more objects by executing a forward generation process of the diffusion model; determining a style loss for each of the objects relative to the reference object; Optimizing the network weights of the diffusion model by back-propagating the gradient of the style loss with respect to the network weights to minimize the style loss and obtain a customized diffusion model with the optimized network weights; as well as The stylized object is generated using the customized diffusion model with the optimized network weights, the stylized object having a style similar to that of the reference object.

2. The method of claim 1, wherein the generating of the object comprises: After completing the iterative denoising of the initial noise by executing the forward generation process of the diffusion model, a final image is generated.

3. The method of claim 1, wherein the diffusion model is a pre-trained text-to-image diffusion-based generative model having as input a content regulator to generate the object.

4. The method of claim 3, wherein the optimization of the network weights further comprises minimizing a combination of the style loss and a content loss corresponding to generating features of the object by the pre-trained text-to-image diffusion-based generative model with the content regulator.

5. The method of claim 3, wherein the optimization of the network weights further comprises optimizing one or more parameters of the pre-trained text-to-image diffusion-based generative model. 6 . The method according to claim 1 , wherein the back-propagation of the gradient of the style loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

7. The method according to claim 6, further comprising: Reparameterize the augmented adjoint ODE.

8. A method for generating an object with a visual effect using a diffusion model, the method comprising: obtaining a reference object having a predetermined visual effect; generating one or more objects by performing a forward generation process of a diffusion model with a control signal configured to control a visual effect of the object; determining a feature loss for each of the objects relative to the reference object having the predetermined visual effect; optimizing the control signal by back-propagating the gradient of the feature loss with respect to the control signal to minimize the feature loss and obtain a customized diffusion model with an optimized control signal; as well as The object is generated using the customized diffusion model with the optimized control signal, the object having a visual effect similar to the predetermined visual effect of the reference object.

9. The method of claim 8, wherein the generating of the object comprises: After completing the iterative denoising of the initial noise by executing the forward generation process of the diffusion model, a final image is generated.

10. The method of claim 8, wherein the diffusion model is a pre-trained text-to-image diffusion-based generative model having a first input as a content regulator and a second input as a visual effect regulator to generate the object.

11. The method of claim 10, wherein the reference object is obtained using the pre-trained text-to-image diffusion-based generative model with the same first input.

12. The method of claim 10, wherein the control signal of the diffusion model is optimized to transfer the second input into the visual effect.

13. The method of claim 8, wherein the back-propagation of the gradient of the feature loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

14. The method according to claim 13, further comprising: Reparameterize the augmented adjoint ODE.

15. A method for providing adversarial examples to audit the security of an object generation system, the method comprising: generating one or more objects by performing a forward generation process of a diffusion model having as input a content regulator and a starting noise having a perturbation, the input corresponding to a reference content category identified by the classifier; determining, via the classifier, a classification score for each of the objects; determining a classification loss of the classification score relative to a target content category, the target content category being different from the reference content category; optimizing the perturbation of the starting noise by back-propagating the gradient of the classification loss with respect to the perturbation to minimize the classification loss of the classification score and obtain a customized diffusion model with optimized starting noise; as well as An adversarial example is generated using the customized diffusion model with the optimized starting noise, the adversarial example being identified by the classifier as being of the target content category.

16. The method of claim 15, wherein the generating of the object comprises: After completing the iterative denoising of the initial noise by executing the forward generation process of the diffusion model, a final image is generated.

17. The method according to claim 15, further comprising: A reference object is generated by executing the forward generation process with the diffusion model as the input to the content regulator, the reference object being identified by the classifier as being of the reference content category.

18. The method of claim 15, wherein the classifier comprises a content review filter.

19. The method of claim 18, wherein the content review filter clarifies the adversarial examples that should be classified into a harmful category into a safe category.

20. The method of claim 15, wherein the back-propagation of the gradient of the classification loss is performed by solving an augmented adjoint ordinary differential equation (ODE) based on an adjoint sensitivity method.

Images