Method and system for fairness in text-to-image diffusion model

Abstract

The invention relates to a method and system for fairness in a text-to-image diffusion model. Methods and systems are provided for reducing biases in text-to-image diffusion models, particularly biases related to gender, race, and cross-sections thereof in occupational cues. The invention introduces a fairness framework based on distribution alignment, and comprises two core technical schemes: (1) distribution alignment loss, which adjusts the output of a model towards the target distribution defined by a user; and (2) performing an adjusted direct fine tuning (adjusted DFT) on the sampling process of the model using the adjusted gradient to optimize the loss based on the generated image. These techniques reduce prejudice while supporting diverse viewpoints of fairness, such as age-controlled deprejudice across multiple concepts. The scalability of the method allows a plurality of prompts to be debiased at the same time, thereby improving the inclusiveness of diffusion model output in different demographic data.

Description

Technical Field

[0001] The present invention generally relates to a method and system for text-to-image generation, and more particularly to a method and system for mitigating bias in text-to-image diffusion models via distribution alignment loss and adjusted direct fine-tuning. Background Art

[0002] We have seen an acceleration in the adoption of text-to-image (T2I) diffusion models by businesses and individuals alike. The scale of images generated by these models is staggering.

[0003] However, this influx of content from diffusion models to society has highlighted the urgent need to address their biases. The rapid adoption of T2I diffusion models has brought challenges related to biases such as racial, gender, and occupational bias. As a result, these models often fail to fairly represent minority groups. In order to address potential issues with the fairness of the generated images, there is a clear need to improve image generation methods in diffusion models.

[0004] Furthermore, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure. Summary of the invention

[0005] According to one aspect of the present invention, a method for generating images from text using a diffusion model to reduce bias is provided, comprising: receiving a text input; and generating an image corresponding to the text input using a diffusion model; wherein the diffusion model is optimized by: aligning the generated image toward a target distribution using a distribution alignment loss; and adjusting the diffusion model by directly fine-tuning the sampling process by adjusting the gradient to minimize the loss function of the generated image.

[0006] In some embodiments, the step of aligning the generated image further comprises identifying attributes in the generated image using a pre-trained classifier; and aligning the identified attributes of the generated image towards a target attribute distribution using a distribution alignment loss.

[0007] In some embodiments, the step of aligning the generated image further includes applying a pre-trained classifier to estimate the class probabilities of the generated image, including the probabilities of specific classes; calculating the transmission distance between the estimated class probabilities of the generated image and the target distribution; and dynamically generating a class distribution of the generated image that matches the target distribution by minimizing the transmission distance.

[0008] In some embodiments, the distribution alignment loss is iteratively applied until the generated image features are aligned with the target distribution.

[0009] In some embodiments, the distribution alignment loss further includes an alignment loss that measures the difference between the generated image and the target distribution, an image semantics preservation loss that measures the semantic consistency of the generated image with the input text, or a face realism preservation loss that penalizes the dissimilarity between the generated face and the closest face in a set of external real faces.

[0010] In some embodiments, the distribution alignment loss is a weighted sum of an alignment loss, an image semantics preservation loss, and / or a facial realism preservation loss.

[0011] In some embodiments, the target distribution is a user-defined target distribution and includes one or more attributes.

[0012] In some embodiments, the target distribution is a non-uniform distribution.

[0013] In some embodiments, the non-uniform distribution is a non-uniform distribution with respect to age, gender, race, or an intersection thereof.

[0014] In some embodiments, the step of directly fine-tuning the sampling process further includes preparing a gradient coefficient of the diffusion model; calculating a gradient value based on the gradient coefficient; adjusting the gradient value to optimize the diffusion model by minimizing a loss function; and back-propagating the adjusted gradient value through the diffusion model to update the model parameters.

[0015] In some embodiments, the gradient values ​​are calculated based on partial derivatives of the loss function with respect to the model parameters.

[0016] In some embodiments, the method further includes de-biasing multiple concepts at once by including different inputs in the fine-tuning data.

[0017] In some embodiments, text input is processed using a natural language processing model to understand semantic features before being used by a diffusion model for image generation.

[0018] In some embodiments, the fine-tuning process uses five soft tokens to adjust the parameters of the diffusion model.

[0019] In some embodiments, the diffusion model is a text-to-image model.

[0020] According to another aspect of the present invention, a system for generating images from text using a diffusion model to reduce bias is provided, comprising: a processor; a memory in electronic communication with the processor; and instructions stored in the memory and executable by the processor to cause the system to perform the following operations: receiving text input; and generating an image corresponding to the text input using a diffusion model; wherein the diffusion model is optimized by: aligning the generated image toward a target distribution using a distribution alignment loss; and adjusting the diffusion model by directly fine-tuning the sampling process by adjusting the gradient to minimize the loss function of the generated image.

[0021] In some embodiments, the system further includes applying a pre-trained classifier to estimate the class probabilities of the generated images, including the probabilities of specific classes; calculating the transmission distance between the estimated class probabilities of the generated images and the target distribution; and dynamically generating a class distribution of the generated images that matches the target distribution by minimizing the transmission distance.

[0022] In some embodiments, the distribution alignment loss further includes an alignment loss that measures the difference between the generated image and the target distribution, an image semantics preservation loss that measures the semantic consistency of the generated image with the input text, or a face realism preservation loss that penalizes the dissimilarity between the generated face and the closest face in a set of external real faces.

[0023] According to another aspect of the present invention, there is provided a device for generating images from text using a diffusion model to reduce bias, comprising: a device for receiving text input; a device for generating images corresponding to the text input using a diffusion model; a device for calculating a gradient matrix for each image in a training image set; wherein the diffusion model is optimized by aligning the generated images toward a target distribution using a distribution alignment loss; and the diffusion model is adjusted by directly fine-tuning the sampling process by adjusting the gradient to minimize the loss on the generated image.

[0024] According to another aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when executed in a computer, the computer program causes the computer to perform any one of the above methods.

[0025] It should be understood that the embodiments described herein are not exhaustive and that additional features and variations of the invention may be incorporated. Various other advantages and novel features of the invention will become apparent from the following detailed description when considered in conjunction with the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present disclosure will be better understood with reference to the detailed description when considered in conjunction with the non-limiting examples and accompanying drawings, in which:

[0027] Figure 1 is an architectural diagram of a text-to-image generation system according to an embodiment of the present invention.

[0028] Figure 2 is a flowchart of a text-to-image generation system according to an embodiment of the present invention.

[0029] Figure 3 is a block diagram of an electronic device for text-to-image generation according to an embodiment of the present invention.

[0030] Figure 4 is a flowchart of a method for generating an unbiased image through a text-to-image generation system according to an embodiment of the present invention.

[0031] Figure 5 is a flow chart of an adjusted direct fine-tuning method for text-to-image generation according to an embodiment of the present invention.

[0032] Figure 6a is a graph of training loss during direct fine-tuning with three different gradients according to an embodiment of the present invention.

[0033] Figure 6b is a gradient scale diagram at different time steps according to an embodiment of the present invention.

[0034] Figure 7 A comparison of unadjusted and adjusted direct fine-tuning of a diffusion model of a sampling process according to an embodiment of the present invention is shown.

[0035] Figure 8 Images generated from the original stable diffusion model (8a) and the joint debiased stable diffusion model (8b) for gender and ethnicity are shown according to an embodiment of the present invention.

[0036] Fig. 9 is a graph of the frequency of age=old representation in generated images of different occupations for the de-biased diffusion model and the original stable diffusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In the following detailed description, reference is made to the accompanying drawings which form a part hereof. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. Unless otherwise indicated, the terms "comprises," "comprising," and grammatical variations thereof as used herein are intended to represent "open" or "inclusive" language such that it includes the described elements, but also permits the inclusion of additional unreferenced elements.

[0038] The various embodiments are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Embodiments described in the context of one of a system or method are similarly valid for the other system or method. Similarly, embodiments described in the context of a method are similarly valid for a system, apparatus, or computer program, and vice versa.

[0039] Features described in the context of one embodiment may be applicable to other embodiments accordingly, even if not explicitly described in these other embodiments. In addition, additions and / or combinations and / or substitutions described for features in the context of one embodiment may be applicable to the same or similar features in other embodiments accordingly.

[0040] Figure 1 is an architecture diagram of a text-to-image generation system according to an embodiment of the present invention.

[0041] The system 100 includes a user terminal 110 , a server 120 and a communication network 130 , which is designed to generate an image based on input text provided by a user. In some embodiments, the text-to-image generation system 100 may include multiple user terminals 110 and multiple servers 120 .

[0042] The user terminal 110 may be a browser, an application (APP) or a network (web) application, or a lightweight application (also called an applet, lightweight application) or a cloud application, etc. The user terminal 110 may be deployed in an electronic device and needs to be run according to the device or an APP in the device, etc. The electronic device may have a display screen and support information browsing, etc. For example, it may be a terminal side device, such as a personal mobile terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0043] The user terminal 110 provides a user interface in which the user can enter text. This text is the basis of the image generation process. Once the user submits the text, the user terminal 110 sends the input to the server 120 via the network 130.

[0044] The server 120 receives text data from the user terminal 110 and processes it to generate an image corresponding to the user input. The server 120 utilizes a text-to-image generation model based on a pre-trained diffusion or generation model. The model converts the text input into visual features and constructs an image. The server can process the text to extract semantic meaning, context, and any specific attributes mentioned in the text to guide the image generation process. Once the image is generated, the server 120 sends the image back to the user terminal 110.

[0045] The server 120 may be responsible for providing a wide range of services, such as facilitating communication between multiple users, supporting background training of models deployed on user terminal devices, and processing data received from users. It is important to note that the server 120 can be implemented in various configurations. It can be operated as a distributed server cluster including multiple servers, or as a single server. The server may also be a server of a distributed system. In addition, the server 120 may also be a cloud-based server (cloud-side device), thereby providing basic cloud computing services. These services may include cloud storage, cloud databases, cloud functions, cloud networks, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data processing, artificial intelligence (AI) platforms, or intelligent cloud computing driven by AI technology. The architecture of the server can be expanded according to system requirements to ensure flexibility and powerful performance.

[0046] The communication between the user terminal 110 and the server 120 is established through the network 130, which may include the Internet, a cellular network, or other wireless communication technologies. The data sent by the user terminal 110 may be encoded, compressed, or encrypted before transmission to ensure security and efficiency. After the server 120 generates the image, it sends it back to the user terminal 110 using the same network connection 130.

[0047] After receiving the generated image from the server 120, the user terminal 110 decodes the image and presents the image on the device. The user can then view, download or share the image through the terminal's interface.

[0048] The system 100 facilitates a seamless text-to-image generation workflow, where user input is converted into a visual representation using an advanced AI model hosted on the server 120. The entire process is automated and designed for real-time interaction between the user terminal 110 and the server 120.

[0049] It should be noted that the text-to-image generation system provided in the embodiments of the present disclosure is generally executed by a server, but in other embodiments of the present disclosure, the user terminal device may also have functions similar to those of the server to execute the text-to-image generation method provided in the embodiments of the present disclosure. In other embodiments, the text-to-image generation method provided in the embodiments of the present disclosure may be executed by the user terminal and the server together.

[0050] Figure 2 is a flowchart of a text-to-image generation method according to an embodiment of the present invention. This method effectively balances creative image generation with fairness considerations, ensuring inclusiveness and representation across dimensions such as race, gender, and other demographic characteristics.

[0051] Method 200 begins by receiving input text from a user. The input is first processed by a text encoder 210, which parses and understands the text input, extracting key semantic and contextual features. This understanding helps identify specific features that the user wants in the generated image, such as objects, settings, or specific traits such as gender or race.

[0052] In some embodiments, the method includes processing input text using natural language processing (NLP) techniques to generate corresponding images in a text-to-image generation system. The method enables the system to understand the semantic meaning of the text input and convert it into a visual representation through a model-driven process.

[0053] The method first receives text input from a user terminal. The method applies NLP techniques to process the input and extract meaningful features such as key entities, actions, and descriptive attributes. This includes breaking down the text into relevant components to understand its structure, context, and semantic role.

[0054] After processing the text, the method encodes the semantic meaning into a format suitable for input into a text-to-image generation model. The generative model then uses the encoded text to create an image that aligns with the features and context provided by the input text.

[0055] Next, the processed text is fed into a fine-tuned diffusion model 220, which is specifically trained to address and mitigate bias. The model employs bias reduction mechanisms such as distribution alignment or fine-tuning to ensure that the generated images are free of bias. The diffusion model has been trained on a wide variety of data, ensuring that it produces fair and balanced representations across gender, race, and / or other factors.

[0056] The final image output 230 is a fair image with reduced bias generated from the input text. The method generates an image based on the user's input while ensuring that the representation complies with the principle of fairness, thereby producing an output without biased portrayals or disproportionate emphasis on certain groups.

[0057] In some embodiments, the generated image is a facial image, which may include a single face captured from various angles or with different expressions. In some embodiments, the generated image may consist of multiple faces within a single image, allowing several people to be recognized or represented at the same time.

[0058] The resulting image is generated based on the input text and sent back to the user terminal. This process allows for efficient and context-aware generation of images from text, providing users with visually accurate and unbiased output.

[0059] Figure 33 is a block diagram of an electronic device for text-to-image generation according to an embodiment of the present invention. The electronic device 300 may include a data diffusion module 310, an input module 305, an NLP module 306, a display module 307, one or more memories 308, and one or more processors 309.

[0060] The data diffusion module 310 further includes a data training module 301, an image generation module 302, an alignment module 303, and a fine-tuning module 304. The diffusion model 310 takes an input training image and processes it through a series of diffusion steps to generate a final output image.

[0061] The data training module 301 is responsible for collecting and preprocessing input training images. These images are used as basic data used by the diffusion model 310. The input training images are prepared and collected to the diffusion model 310 for processing.

[0062] In some embodiments, during a preprocessing step for image generation, input images are prepared to ensure consistency and quality, which is critical to the effectiveness of the diffusion model. This process typically involves normalizing the images to a uniform scale, resizing the images to match the model's input requirements, and applying data augmentation techniques such as flipping, rotation, or color adjustments to enhance the model's ability to generalize from the data. This thorough preprocessing ensures that the diffusion model starts with well-prepared data, resulting in better, more reliable output images.

[0063] The image generation module 302 is a module that generates images using a diffusion model (original or retrained model). This is the core model responsible for generating output images from the input training images of the diffusion model. The diffusion model can be a latent diffusion model (LDM) or any other similar model designed to generate images.

[0064] The alignment module 303 is used to adjust specific attributes in the generated images towards a user-defined target distribution. This distribution alignment loss adjusts the features of the generated images by guiding them to the user-defined target distribution, thereby ensuring that the model produces images that meet fairness criteria. This allows for flexible definition of fairness based on the user's goals, such as age, gender, and ethnicity distribution.

[0065] The fine-tuning module 304 is used to directly fine-tune the sampling process of the diffusion model, adjusting the gradients to optimize the loss of the generated image. It uses the adjusted gradients to directly optimize the loss defined on the generated image during the sampling process of the diffusion model. By adjusting the gradients during fine-tuning, the model can reduce bias at the output level while maintaining high image quality. This approach is particularly effective when fine-tuning with a limited number of tokens, making it resource-efficient.

[0066] The input module 305 is responsible for capturing user input in the form of text. It acts as an interface where the user can provide queries, commands, or any other text data. This module ensures that the input is formatted correctly and prepared for further processing. It handles various types of text input, from natural language sentences to structured commands, and forwards it to the NLP module for further analysis.

[0067] The NLP module 306 processes the user input text received from the input module 305. It applies advanced NLP techniques to understand the meaning and intent behind the user text. This module extracts key entities, identifies the user's intent, and performs any necessary semantic analysis to understand the context of the input.

[0068] In some embodiments, key functions of the NLP module 306 include: tokenizing and parsing text input; identifying and classifying key entities such as dates, names, or places; determining the intent behind user input; and leveraging machine learning models to interpret and contextualize input for the next stage of processing.

[0069] The display module 307 displays the final output image generated by the diffusion module 310 .

[0070] In some embodiments, the system includes a memory 308. It stores training images and their importance scores for further analysis or future use. The memory 308 may include one or more non-transitory computer-readable storage media that may be non-transitory. The memory 308 may further include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 308 is configured to store at least one program code, which is configured to be executed by the processor 309 to implement the data diffusion and data attribution provided in the method embodiments of the present invention.

[0071] In some embodiments, the user device includes a processor 309. The processor 309 can be implemented using various technologies and architectures designed to implement the described functions. It can be implemented as a general-purpose processor, a content addressable memory, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, a programmable logic device, discrete gate or transistor logic, discrete hardware components, or a combination thereof. In addition, the processor can take the form of a microprocessor, a controller, a microcontroller, or a state machine.

[0072] Furthermore, processor 309 may be implemented by a combination of computing devices, such as combining a digital signal processor with a microprocessor, utilizing multiple microprocessors, integrating one or more microprocessors with a digital signal processor core, or in any other suitable configuration.

[0073] Those skilled in the art will recognize that Figure 3 The structure shown in FIG. 3 is not limited to electronic device 300. Electronic device 300 may include more or fewer components than those shown, may combine some components, or may employ alternative component configurations.

[0074] Figure 4 is a flow chart of a method for generating an unbiased image through a text-to-image generation system according to one embodiment of the present invention. As a preliminary matter, it should be understood that the steps of method 400 are not necessarily limiting and that the steps may be added, omitted, and / or performed simultaneously without departing from the scope of the appended claims. It should be understood that method 400 may include any number of additional or alternative tasks and that method 400 may be incorporated into a more comprehensive program or process having additional functionality not described in detail herein. Furthermore, as long as the intended overall functionality remains unchanged, Figure 4 One or more of the tasks shown in the method 400 may be omitted from an embodiment of the method 400. It should also be understood that the method 400 shown may be stopped at any time. The method 400 is computer-implemented in that the various tasks or steps performed in association with the method 400 may be performed by software, hardware, firmware, or any combination thereof.

[0075] This flowchart describes an efficient process for producing text-generated images while addressing and minimizing bias, providing an improved solution for fair image generation. The process includes the following steps:

[0076] At step 401, it receives text input. The method receives text input provided by a user through a user terminal device. The text is the basis for generating an image. For example, the text input can be "a facial photo of an electrical and electronic maintenance person".

[0077] At step 402, it processes and understands the user text input using NLP algorithms. The received text is processed using NLP techniques to understand the semantic meaning and context. The system analyzes the text to extract key information such as objects, descriptions, or any specific features mentioned by the user.

[0078] At step 403, it aligns the distribution of generated images towards the target distribution. The system utilizes a distribution alignment loss mechanism that adjusts the features of the generated images to conform to the user-defined target distribution. This ensures that specific traits such as race, gender, and other attributes are fairly represented according to the defined target distribution.

[0079] In some embodiments, the user defines a target distribution for one or more attributes, such as gender or race. The diffusion model generates an initial batch of images based on user input prompts. The generated images are compared to the target distribution, and a loss is calculated based on the deviation from the target. The model is then iteratively adjusted to minimize the distribution alignment loss, directing the features of the image to the user-defined target. This loss function dynamically directs image generation to a user-defined attribute distribution, such as gender, race, or age, while maintaining the integrity and semantics of the generated images.

[0080] In some embodiments, the step of aligning the generated images involves the following process: using a pre-trained classifier to detect and identify specific attributes in the generated images, such as age, gender, race, or other relevant features. The identified attributes are then aligned towards a predefined target attribute distribution. This alignment is achieved by applying a distribution alignment loss, which adjusts the generated images to ensure that their attributes match the target distribution. This process helps reduce bias and improves the fairness of image generation. This step ensures that the generated images not only reflect the input prompts, but are consistent with the desired attribute balance, thereby contributing to fair and representative outputs.

[0081] In some embodiments, the step of aligning the generated images involves applying pre-trained classifiers to the generated images. These classifiers estimate the class probabilities of the generated images, determining the likelihood that each image belongs to a particular class (such as gender, race, or age group). The transfer distance between the estimated class probabilities of the generated images and the predefined target distribution is calculated. This metric measures the difference between the actual distribution of the generated images and the desired balanced distribution. The class distribution of the generated images is dynamically adjusted to match the target distribution. This is achieved by minimizing the transfer distance, thereby ensuring that the generated images are better aligned with the target attribute balance. This process refines the alignment of the generated images, ensuring that the output class distribution is consistent with the desired distribution, reducing bias and improving fairness.

[0082] In some embodiments, a target distribution is a user-defined distribution that specifies the desired balance of attributes in the generated images. This distribution can include multiple attributes simultaneously, such as gender, race, age, and other relevant characteristics. By defining this target distribution, the user can guide the image generation process to ensure that the output is aligned with specific demographic or attribute-based goals, resulting in a more customized and balanced representation across dimensions. A target distribution is specified for each attribute. For example, a user can define a uniform distribution, where 50% of the generated images should represent females and 50% males, or 75% should be young people and 25% should be old people.

[0083] In some embodiments, the process iteratively adjusts the model until the generated images are closely aligned with the target demographic distribution. A distribution alignment loss is iteratively applied during the image generation process. With each iteration, the generated image features are progressively adjusted to reduce the difference between the current distribution and the target distribution. This iterative application will continue until the features of the generated images are aligned with the predefined target distribution, ensuring that the final output reflects the desired balance of attributes.

[0084] In some embodiments, in order to ensure that the generated images remain of high quality, additional loss functions are introduced, namely alignment loss, image semantics preservation loss, and face realism preservation loss. For the alignment loss, it measures the difference between the generated image and the target distribution. It ensures that the output image is aligned with a predefined distribution, such as a balanced representation of age, gender, or race. For the image semantics preservation loss, it measures the semantic consistency between the generated image and the input text prompt. It ensures that the image accurately reflects the meaning and attributes described in the text, preserving the intent of the prompt. For the face realism preservation loss, it penalizes the dissimilarity between the generated face and the closest match in a set of real external faces. It ensures that the generated face maintains realistic human features by comparing the generated face with real-world facial data. The final distribution alignment loss - distribution alignment loss includes one of the three components and is calculated as a weighted sum of these components, thereby ensuring that the generated image is semantically accurate, visually realistic, and conforms to the target distribution.

[0085] At step 404, it applies an adjusted direct fine-tuning (DFT) of the diffusion model to minimize the loss function. The diffusion model undergoes an adjusted DFT process through an adjusted gradient-based optimization of the model sampling process. This involves using the adjusted gradients to directly optimize the loss function associated with the generated image. This step allows the model to further improve image quality and ensure that the generation process complies with the fairness principle.

[0086] The adjusted DFT method involves computing an adjusted gradient that directly optimizes for bias reduction (e.g., based on gender or race bias) in the generated images. This adjusted gradient is used to fine-tune the diffusion model, which can effectively reduce bias without requiring a lot of computational resources. Notably, this method can significantly reduce gender bias with only five soft tokens during fine-tuning.

[0087] At step 405, it generates unbiased images. The method significantly reduces gender and racial bias, as well as intersectional bias, especially when using occupational cues. By ensuring balanced and fair representation in the output, the generated images reflect diversity and inclusion of various attributes.

[0088] In some embodiments, the method is designed to address multiple bias factors simultaneously. For example, when debiasing images associated with occupational prompts, the method can simultaneously ensure a balance in gender representation and take into account diverse racial groups. In addition, the method supports fairness considerations beyond strict equality, allowing for distributions that reflect specific demographic compositions. The scalability of the system ensures that users can debias multiple prompts in parallel, providing flexibility and efficiency in mitigating bias. The method is able to reduce multiple biases simultaneously, such as those related to gender, race, and age, by incorporating corresponding prompts into the fine-tuning data. For example, the method can control the distribution of generated images to reflect young and old people while eliminating biases in gender and racial attributes. The scalability of the method ensures that multiple concepts can be debiased in parallel.

[0089] This approach provides a robust and scalable solution for mitigating bias in text-to-image diffusion models. By introducing a distribution alignment loss and adjusted direct fine-tuning, the invention reduces bias while maintaining model flexibility, ensuring a fairer and more representative output. It can balance multiple attributes at once (e.g., age, gender, and race) by including relevant cues in the fine-tuning data. The method is scalable, allowing multiple concepts to be debiased simultaneously by incorporating them into the fine-tuning dataset. This flexibility ensures that the system can handle complex population distributions while maintaining image quality.

[0090] Figure 5 is a flow chart of an adjusted direct fine-tuning method for text-to-image generation according to one embodiment of the present invention.

[0091] At step 501, it prepares the gradient coefficient. In this step, the gradient coefficient is prepared. This coefficient is a key factor in adjusting the gradient of the model, ensuring that the model is guided to reduce certain types of bias during the optimization process.

[0092] At step 502, it adjusts the gradient. The method first calculates the gradient based on the current output of the diffusion model. This involves calculating the gradient value based on the partial derivatives of the loss function with respect to the model parameters.

[0093] Once the gradients are calculated, they are adjusted using the prepared gradient coefficients. This adjustment helps fine-tune the optimization process, allowing the model to focus on reducing specific losses, such as those associated with bias in gender, race, or other attributes. The adjusted gradients ensure that the model produces more fair outputs while improving the overall quality of the generated images.

[0094] In some embodiments, an adapted gradient-based method is applied to directly optimize the sampling process of the model. The method removes the dependency on earlier or later time steps by focusing only on the gradient of the current time step. This allows direct optimization, reduces coupling between different stages of the image generation process, and simplifies fine-tuning.

[0095] At step 503, in this final step, the adjusted gradients are back-propagated through the diffusion model. After the gradients are calculated from the generated image, the system back-propagates through the model, adjusting not only the final layers responsible for image creation, but also the earlier layers related to interpreting the textual cues. In this way, the entire model from text input to final image output is refined to produce a more fair image.

[0096] In some embodiments, the method supports scalable fairness optimization, allowing the model to remove bias from multiple concepts at once. For example, the system can simultaneously reduce gender and racial bias in images by incorporating these concepts into the fine-tuning data.

[0097] In some embodiments, it allows the user to define various fairness parameters, such as maintaining a demographic distribution of 75% young people and 25% old people, while ensuring gender balance in the generated content.

[0098] In some embodiments, the fine-tuning process is scalable and can achieve bias reduction with minimal adjustments to model parameters. For example, bias can be reduced by introducing just five soft tokens, allowing fine control over the generated image features.

[0099] Example:

[0100] Background on Diffusion Models

[0101] The diffusion model assumes a forward diffusion process that is based on the variance table β1, ..., β T Gradually inject Gaussian noise into the data distribution q(x0):

[0102]

[0103] Where T is a predefined total number of steps (usually 1000). t} t∈[T] , so that the data distribution q(x0) gradually transforms into an approximate Gaussian distribution The diffusion model then learns to approximate the data distribution by inverting this diffusion process, from a Gaussian distribution start:

[0104]

[0105] where μ θ(xt, t) using the noise prediction network ∈ θ (x t , t) is parameterized, where and {σ t} t∈[T] is the predetermined noise variance. After training, generation from the diffusion model involves the reverse process p θ (x 0：T ), which is done by sampling the noise variable x T ~p(x T ) and then proceed to obtain x0 as follows:

[0106]

[0107] Common diffusion models include the latent diffusion model (LDM), whose forward / backward diffusion process is defined in the latent space. Enc and decoder f Dec , LDM has the potential to represent z0 = f Enc (x0). To generate an image, LDM first trains the latent noise z T Take a sample, run the reverse process to get z0, and use x0 = f Dec (z0) decodes it.

[0108] For the text-to-image diffusion model, the noise prediction network ∈ θ Accept additional text prompts P, that is, ∈ θ (g φ (P), x t , t), where g φ represents a pre-trained text encoder parameterized by φ. Most T2I models, including Stable Diffusion, further adopt LDM and thus use a text-conditional noise prediction model in the latent space, denoted as ∈ θ (g φ (P), z t , t), which is the central focus of our work. Sampling from the T2I diffusion model additionally exploits the classifier-free bootstrapping technique.

[0109] method

[0110] For the present invention, the method consists of (i) a loss design that directs specific properties of the generated images towards the target distribution while preserving image semantics, and (ii) a direct fine-tuning of the diffusion model sampling procedure.

[0111] Loss Design

[0112] In some embodiments, the loss is distributed by aligning the loss and image semantics preservation loss Components. Distributed Alignment Loss (DAL) Suppose the system wants to control the classification properties of the generated images with K classes and move them towards the target distribution Alignment. Each class is represented as a one-hot vector of length K, and is a discrete distribution over these classes. The system first generates a batch of images using the fine-tuned diffusion model and some hint P For each generated image x (i) , the system uses a pre-trained classifier h to produce a class probability vector in Represents x (i) is the estimated probability of coming from class k. Suppose there is another set of vectors {u (i)} i∈[N] represents the target distribution, where each u (i) are all one-hot vectors representing classes, then the system can calculate (i)} i∈[N] to {u (i)} i∈[N] Optimal Transmission (OT):

[0113]

[0114] in represents all permutations of [N], σ = [σ1,…,σ N ] and σ i ∈[N]. Intuitively, σ * Find the most effective modification of the current image in the class probability space to match the target distribution. The system constructs {u (i)} i∈[N] For independent and identically distributed data (IID) samples, calculate the expected value of OT:

[0115]

[0116] q (i) is a probability vector where the kth element is the image x (i) should have a probability of target class k if the generated batch of images indeed follows the target distribution When the number of classes K is small, the expected value of OT can be calculated analytically, or approximated by empirical means as K increases. It is worth noting that one can also construct a fixed set of {u (i)} i∈[N] , such as half male and half female, to represent a balanced gender distribution. However, fixed splits suggest a stronger finite sample alignment goal and ignore the sensitivity of OT.

[0117] Finally, the system generates the target class {y (i)} i∈[N] and the confidence of these targets {c (i)} i∈[N] : The system defines DAL as the cross entropy loss with respect to these dynamically generated targets, with a confidence threshold C,

[0118]

[0119] The system also uses image semantics preservation loss The system keeps a frozen, un-fine-tuned copy of the diffusion model and penalizes image dissimilarity measured by Contrastive Language-Image Pre-training (CLIP) and Label-free Self-Distillation (DINO): CLIP and DINO are two powerful models for image understanding that can both be used to measure image similarity. CLIP was developed to understand images and text in a unified way. It is designed to match image and text embeddings into the same latent space. DINO is a self-supervised learning model that can learn without a labeled dataset. DINO excels at finding similarities in images based on what it has learned, without any labels or guidance.

[0120]

[0121] in is a batch of images generated by the frozen model using the same prompt P. They are called original images. The system requires that for each pair of fine-tuning images x (i) and the original image o (i) Both use the same initial noise generation. The system uses both CLIP and DINO, because CLIP is pre-trained with text supervision, while DINO is pre-trained with image self-supervision. In the implementation, the system uses laion / CLIP-ViT-H-14-laion2B-s32B-b79K and dinov2-vitb14.

[0122] Adaptation to face-centric attributes

[0123] For this invention, the system focuses on face-centric attributes such as gender, race, and age. It was found that the following adaptation from the general case produces the best results. First, the system uses a face detector d face From each generated image x (i) Retrieve the facial region d face (x (i) ). The system applies classifier h and DAL only on the face region Secondly, the system introduces another facial realism preservation loss It penalizes the generated faces face (x (i) ) and a set of external real faces The dissimilarity between the closest faces in

[0124]

[0125] where emb(·) is the face embedding model. Helps maintain the realism of faces, which DAL can edit heavily. In the implementation, the system uses the Celebra and FairFace datasets as external faces. The system uses SFNet-20 (as a face embedding model). The Celebra dataset is a large-scale face dataset that is commonly used for face recognition, attribute prediction, and other computer vision tasks. It is known for its size, diversity, and various labeled facial attributes. The FairFace dataset is a racially balanced facial image dataset. It contains images from 7 different racial groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino.

[0126] Final loss is the weighted sum: It is worth noting that the system uses a dynamic weight λ img If the generated image x (i) The target class y (i) With the original image o (i) Class h(d face (o (i) )) is consistent, the system uses a larger λ img，1 Intuitively, if the original image o (i) The distribution alignment goal has been met, so the system encourages (i) and (i) For the target class y (i) Corresponding to the original image o (i) Class h(d face (O (i) )) Other images that are inconsistent x (i) , the system uses a smaller weight λ for non-face areas img，2 , and use the smallest weight λ for the face area img，3 Intuitively, these images do need editing, especially in the facial areas. Finally, if the image does not contain any faces, the system only applies But do not apply and If the image contains multiple faces, the system will focus on the face that occupies the largest area.

[0127] Direct fine-tuning of the sampling process of the diffusion model

[0128] Consider the T2I diffusion model using the hint P and the initial noise z T Generate image x0 = f Dec (z0). The goal is to fine-tune the diffusion model to minimize the differentiable loss

[0129] In some embodiments, the system calculates the sampling process The exact gradient of , and then perform gradient-based optimization.

[0130] By writing down the gradients explicitly, the system is Analyze gradients. U-Net is a convolutional neural network and an encoder-decoder network based on skip connections, which is usually used to generate images of the same size as the input image. in

[0131] where ∈ (t) represents the U-Net function ∈ evaluated at time step t θ (g φ (P), z t , t). Importantly, the repeated evaluation of the U-Net during back-diffusion results in a factor B that scales exponentially with t. t .

[0132] In some embodiments, the system applies an adjusted DFT using an adjusted setting A t =1 and B t = Gradient of I: It is motivated by the expansion expression of the reverse process:

[0133]

[0134] When the system is set to B t =I, the system essentially converts z t is considered as an external variable and is independent of the U-Net parameter θ, rather than recursively depending on θ. Otherwise, according to the chain rule, it will be in B t All couplings between partial gradients at different time steps are generated in . However, setting B t =I does preserve all uncoupled gradients, i.e., When the system sets A t = 1, it normalizes the relationship between different time steps t in z0 and ∈ θ (gφ (P), z t , t). It is well known that properly weighting different time steps can accelerate diffusion training. Finally, the system implements the adjusted DFT in the following Algorithm A.1.

[0135]

[0136] Figure 6a is a graph of training loss during direct fine-tuning with three different gradients according to one embodiment of the present invention. It shows the unadjusted gradient, the adjusted gradient, and a variant that does not normalize A for the same image semantics preserving loss with the same fixed target image. i After 1000 iterations, there is no significant reduction in the training loss, indicating that the unadjusted gradients applied to the diffusion model are ineffective in optimizing the training process. However, when the adjusted gradients are introduced, the training loss is greatly reduced, indicating that the optimization is improved and the learning is more effective. This highlights the importance of gradient adjustment in improving model performance during the training of the diffusion model. In addition, the normalized A i Further stabilized the optimization process.

[0137] Figure 6b is a gradient ratio diagram at different time steps according to one embodiment of the present invention. The average value and 90% confidence interval CI are calculated by 20 random runs.

[0138] For the gradient values ​​in Equation 9, for values ​​of t close to T=1000, By weight In addition, due to B t Included So this coupling between partial gradients at different time steps introduces The system assumes that is a random Gaussian matrix and in Figure 6b Drawn in and Obviously, as t→1000, surge in both size and variance, but and Neither.

[0139] Figure 7 A comparison of the unadjusted and adjusted direct fine-tuning of the diffusion model for the sampling process according to one embodiment of the present invention is shown. The solid grey line represents the sampling process. The dashed line highlights the gradient calculation with respect to the model parameters (θ). The variable z t and ∈ (t)represents the data and noise prediction at time step t. i and I i represents the direct and indirect gradient paths between adjacent time steps. For example, at t=3, the unadjusted DFT computes the exact gradient (defined in Equation 9), which involves noisy predictions at other time steps (via gradient paths I1I2I3I4I5, I1I2D3I5, and D1I3I5I5). The adjusted DFT uses the adjusted gradients, which eliminates coupling with other time steps and converts A i Normalized to 1 for more effective fine-tuning.

[0140] experiment

[0141] Mitigating gender, race, and intersectional biases

[0142] To reduce storage usage, in all experiments, the system can (i) quantize the diffusion model to float16, (ii) apply gradient checkpointing, and (iii) use DPM-Solver++ as the diffusion scheduler, and T2I generation only takes about 20 steps.

[0143] In some embodiments, the method is applied to runwayml / stable diffusion-v1-5 (SD for short), a T2I diffusion model publicly accessible from Hugging Face, to reduce gender, race, and their intersection biases. The system adopts four racial categories from the FairFace dataset: WMELH = {White, Middle Eastern, Latino Hispanic}, Asian = {East Asian, Southeast Asian}, Black, and Indian. The gender and race classifiers used in DAL are trained on the CelebA or FairFace datasets. The system considers a uniform distribution over gender, race, or their intersection as the target distribution. The system adopts the prompt template "A photo of a face of {profession}, a person" and uses 1000 / 50 professions for training / testing. For the main experiments, unless otherwise stated, the system fine-tunes LoRA with a rank 50 applied on the text encoder.

[0144] In some embodiments of the gender debiasing experiment, the system uses the CelebA dataset to train a gender classifier. The system uses CelebA faces as external faces and maintains a loss for face realism. The system sets λ face =1,λ img，1 =8,λ img，2 =0.2×λ img，1 and img，3 =0.2×λ img，2The system uses a batch size N = 24 and sets the confidence threshold C = 0.8 for the distribution alignment loss. The system is trained for 10k iterations using the AdamW optimizer with a learning rate of 5e-5. The system checks every 200 iterations and reports the best checkpoint. Fine-tuning takes about 48 hours on 8 NVIDIA A100 GPUs.

[0145] In some embodiments for racial debiasing experiments, the system trains a racial classifier using the FairFace dataset. The system uses FairFace faces as external faces, preserving the loss for facial realism. System settings λ face =0.1,λ img，1 =6,λ img，2 =0.6×λ img，1 and img，3 =0.3×λ img，2 The system uses a batch size N = 32 and sets the confidence threshold C = 0.8 for the distribution alignment loss. The system is trained for 12k iterations using the AdamW optimizer with a learning rate of 5e-5. The system checks every 200 iterations and reports the best checkpoint. Fine-tuning takes about 48 hours on 8 NVIDIA A100 GPUs.

[0146] In some embodiments of the experiment to jointly eliminate gender and race bias, the system uses the FairFace dataset to train classifiers for gender and race classification. The system uses the FairFace faces as external faces and maintains a loss for face realism. The system sets λ face =0.1 and W img，1 = 8. For the gender attribute, the system uses λ img，2 =0.2×λ img，1 and λ img，3 =0.2×λ img，2 For the ethnicity attribute, the system uses λ img，2 =0.6×λ img，1 and λ img，3 = 0. The system uses a batch size N = 32 and sets the confidence threshold C = 0.6 for the distribution alignment loss. The system is trained for 14k iterations using the AdamW optimizer with a learning rate of 5e-5. The system checks every 200 iterations and reports the best checkpoint. Fine-tuning takes about 48 hours on 8 NVIDIA A100 GPUs.

[0147] The system trains separate gender and race classifiers for evaluation. The system generates 60, 80, or 160 images for each prompt to evaluate gender, race, or intersectional bias, respectively. For each prompt P, the system computes the following metrics: where freq(i) is the frequency of group i in the generated images. The number of groups K for gender / race / and their intersections is 2 / 4 / 8. Classification of an image into a specific group is based on the face covering the largest area. This bias metric considers a perfectly balanced target distribution. It measures the difference in representation of different groups, averaged across all comparison groups.

[0148] The method consistently achieves the lowest bias in all three cases. Furthermore, the method still maintains strong consistency with the textual prompts.

[0149] Figure 8 The images generated from the original stable diffusion model (8a) and the joint debiased stable diffusion model (8b) for gender and race according to one embodiment of the present invention are shown. It demonstrates the effectiveness of the debiasing process. The models are evaluated using an unseen occupation prompt: "A photo of the face of an electrical and electronics repair worker, a person."

[0150] In the original stable diffusion (SD) model, there was significant bias, with gender bias scores of 0.84, race bias scores of 0.48, and a combined gender-race bias score of 0.24. This version generated images of mostly white men, marginalizing other identities such as women, blacks, Indians, Asians, and intersections of these categories.

[0151] In contrast, the joint debiasing model significantly reduced these biases, achieving a gender bias score of 0.11, a race bias score of 0.10, and a combined gender-race bias score of 0.06. The debiasing model significantly improved representation of minority groups, providing a more balanced and inclusive portrayal of individuals of different genders and races, particularly for the “Electrical and Electronics Repairer” occupation in the test set.

[0152] Ablation of different components was performed for fine tuning.

[0153] The system fine-tunes individual components of stable diffusion to reduce gender bias, and the results are shown in Table 1. First, the method provides high robustness to the number of fine-tuned parameters. By optimizing only five soft tokens as prompt prefixes, gender bias can be significantly mitigated. Second, Table 1 shows that fine-tuning the text encoder and U-Net is the most effective.

[0154] These findings provide clues for deciding which components to fine-tune when debiasing a diffusion model. In some embodiments, it prioritizes fine-tuning language understanding components, including prompts and text encoders. By doing so, it encourages the model to maintain an overall visual representation of gender and racial identity, rather than manipulating low-level pixels to represent gender and race.

[0155]

[0156]

[0157] Table 1: Fine-tuning different SD components. For the hint prefix, five soft tokens are fine-tuned. For the other components, LoRA is fine-tuned with rank 50.

[0158] The distribution consistency of age

[0159] Fig. 9 is a graph of the frequency of age=older representations in the generated images of different occupations for the debiased diffusion model and the original stable diffusion according to one embodiment of the present invention. The x-axis represents different occupations, and the 25% horizontal line represents the target distribution of the "older" representation. For each occupation, it shows the results of the original stable diffusion model and the debiased G.XR.& aligned age results. The debiased G.XR.& aligned age results are shown on the left, while the original stable diffusion model results are shown on the right. The horizontal line near the 25% value represents the distribution of the debiased diffusion model, while the curve shows the distribution of the original stable diffusion model.

[0160] The system can align the age distribution for each occupation prompt with a non-uniform distribution, specifically 75% young and 25% old, while eliminating biases in gender and race. Using the age attribute in the FairFace dataset, young people are defined as 0-39 years old and old people include 39 years old and above. To avoid the pitfall of the model consistently generating images of young white women and old black men, we fine-tune with a stronger DAL to align the age towards the target distribution based on gender and race. The system is evaluated using an independently trained age classifier.

[0161] like Fig. 9 As shown, in the case of the debiased diffusion model, the frequency of “older” representations in most occupations is always close to the target line of 25%, indicating that the model is successful in mitigating age-related bias. In contrast, the original stable diffusion model follows the trend of the curve, deviating from the target, showing a less balanced distribution of “older” representations.

[0162] Eliminate bias from multiple concepts at once

[0163] The system can debias multiple concepts at once by including only these cues in the fine-tuning data, so the approach is scalable. In some embodiments, the system uses a mixture of the following four categories of cues to debias SD: (1) Occupation cues: formulated with the template "a photo of a face of {occupation}, a person". The system uses the same 1000 / 50 occupations for training / testing. (2) Sports cues: formulated with the template "a person playing {sport}". The system uses 250 / 50 sports activities for training / testing, such as "yoga", "taekwondo", and "ninjutsu". (3) Occupation cues with style & context: These are non-templated cues that specify occupations with diverse styles or contexts. The system trains / tests on 150 / 19 such cues obtained from captions in the LAION-AESTETICS dataset. For example, an example is "a magician performing magic on an ancient machine, an aesthetic portrait of conceptual art". Finally, (4) Person descriptors: These cues describe one or more people. The system uses 40 / 10 such cues for training / testing. Examples include "hot personal training instructor" and "oil painting of a person wearing colorful fabric."

[0164] The evaluation is reported in Table 2. Debiased SD reduces gender, race, and intersectional biases for all four concepts without degrading cue-image alignment. Compared to single-concept debiasing, it increases the probability of generating images that blend masculine and feminine features.

[0165]

[0166] Table 2: Eliminating gender, race, and intersectional biases on multiple concepts at once.

[0167] In summary, for the current system, it consists of two main technical contributions. First, it designs a loss function that guides the generated images towards the desired distribution while preserving image semantics. A key component is the distribution alignment loss (DAL). For a batch of generated images, DAL uses a pre-trained classifier to estimate class probabilities (e.g., male and female probabilities) and dynamically generates target categories that match the target distribution and have the minimum transfer distance. In order to maintain image semantics, the system regularizes the CLIP and DINO similarities between images generated by the original model and the fine-tuned model.

[0168] Second, the system applies an adapted direct fine-tuning of the diffusion model. The adapted DFT aims to directly fine-tune the sampling process of the diffusion model to minimize any loss defined on the generated images. It opens the way for more refined and targeted fine-tuning of the diffusion model and can be applied to objectives beyond fairness.

[0169] Empirically, the system significantly reduces gender, race, and their intersection bias for occupational prompts. Debiasing is effective even for prompts that are invisible to style and context. The method is applicable to any component of the diffusion model being fine-tuned. Ablation studies show that fine-tuning the text encoder while keeping the U-Net unchanged can achieve the best results, effectively mitigating bias and reducing the potential negative impact on image quality. Surprisingly, fine-tuning only five soft tokens as prompt prefixes can greatly reduce gender bias, demonstrating the effectiveness of soft prompt adjustment for fairness. These results emphasize the robustness of the method and the effectiveness of debiasing the language understanding component of the T2I diffusion model by fine-tuning it.

[0170] A notable feature of the system is its flexibility, which allows users to specify the desired target distribution. It can effectively adjust the age distribution to achieve a ratio of 75% young people and 25% old people, while removing bias from gender and race. By expanding the set of prompts used for fine-tuning, it can remove bias from multiple concepts at once, such as occupation, sports, and personal descriptors.

[0171] For the present invention, any suitable programming language may be used to implement the routines of a particular embodiment, including C, C++, Java, assembly language, etc. Different programming techniques may be employed, such as procedural or object-oriented programming techniques. These routines may be executed on a single processing device or multiple processors.

[0172] Particular embodiments may be implemented in a computer-readable storage medium (also referred to as a machine-readable storage medium) for use by or in conjunction with an instruction execution system, apparatus, system, or device. Particular embodiments may be implemented in the form of control logic in software or hardware, or a combination of both. When executed by one or more processors, the control logic is operable to perform the operations described in the particular embodiments.

[0173] Although various aspects and embodiments are disclosed herein, it is apparent that after reading the above disclosure, those skilled in the art will appreciate various other modifications and adaptations of the present invention without departing from the spirit and scope of the present invention, and all such modifications and adaptations are within the scope of the appended claims. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to be limiting, and the true scope and spirit of the present invention are indicated by the appended claims.

Claims

1. A method for generating images from text using a diffusion model to reduce bias, comprising: Receive text input; as well as generating an image corresponding to the text input using the diffusion model; The diffusion model is optimized in the following way: Align the generated images towards the target distribution using a distribution alignment loss; as well as The diffusion model is tuned by directly fine-tuning the sampling process by adjusting the gradients to minimize the loss function of the generated images.

2. The method according to claim 1, wherein: The step of aligning said generated image further comprises: identifying attributes in the generated image using a pre-trained classifier; and The identified attributes of the generated images are aligned towards a target attribute distribution using the distribution alignment loss.

3. The method according to any one of claims 1 to 2, wherein: The step of aligning said generated image further comprises: applying a pre-trained classifier to estimate class probabilities for the generated images, including probabilities for specific classes; calculating a transfer distance between the estimated class probabilities of the generated images and the target distribution; and The class distribution of the generated images matching the target distribution is dynamically generated by minimizing the transmission distance.

4. The method according to any one of claims 1 to 3, iteratively applying the distribution alignment loss until the generated image features are aligned with the target distribution.

5. The method according to any one of claims 1 to 4, wherein: The distributed alignment loss further comprises: an alignment loss that measures the difference between the generated image and the target distribution, an image semantics preservation loss, which measures the semantic consistency of the generated image with the input text, or A facial realism preserving loss that penalizes the dissimilarity between the generated face and the closest face in a set of external real faces.

6. The method according to claim 5, wherein: The distribution alignment loss is a weighted sum of the alignment loss, the image semantics preservation loss and / or the facial realism preservation loss.

7. The method according to any one of claims 1 to 6, wherein: The target distribution is a user-defined target distribution and includes one or more attributes.

8. The method according to any one of claims 1 to 7, wherein: The target distribution is a non-uniform distribution with respect to gender, race, or an intersection thereof.

9. The method according to claim 8, wherein: The non-uniform distribution is a non-uniform distribution with respect to age, gender, race or a cross thereof.

10. The method according to any one of claims 1 to 9, wherein: The steps of directly fine-tuning the sampling process further include: preparing a gradient coefficient for the diffusion model; Calculate the gradient value according to the gradient coefficient; adjusting the gradient value to optimize the diffusion model by minimizing the loss function; and The adjusted gradient values ​​are back-propagated through the diffusion model to update the model parameters.

11. The method according to claim 10, wherein: The gradient value is calculated based on the partial derivative of the loss function with respect to the model parameters.

12. The method according to any one of claims 1 to 11, further comprising: Remove bias from multiple concepts at once by including diverse inputs in the fine-tuning data.

13. The method according to any one of claims 1 to 12, wherein: The text input is processed using a natural language processing model to understand semantic features before the text input is used by the diffusion model for the image generation.

14. The method according to any one of claims 1 to 13, wherein: The fine-tuning process uses five soft tokens to adjust the parameters of the diffusion model.

15. The method according to any one of claims 1 to 14, wherein: The diffusion model is a text-to-image model.

16. A system for generating images from text using a diffusion model to reduce bias, comprising: processor; a memory in electronic communication with the processor; as well as Instructions stored in the memory and executable by the processor to cause the system to: receiving text input; and generating an image corresponding to the text input using the diffusion model; The diffusion model is optimized in the following way: Align the generated images towards the target distribution using a distribution alignment loss; as well as The diffusion model is tuned by directly fine-tuning the sampling process by adjusting the gradients to minimize the loss function of the generated images.

17. The system of claim 16, further comprising: applying a pre-trained classifier to estimate class probabilities for the generated images, including probabilities for specific classes; calculating a transfer distance between the estimated class probabilities of the generated images and the target distribution; as well as The class distribution of the generated images matching the target distribution is dynamically generated by minimizing the transmission distance.

18. A system according to any one of claims 16 to 17, wherein: The distributed alignment loss further comprises: an alignment loss that measures the difference between the generated image and the target distribution, an image semantics preservation loss, which measures the semantic consistency of the generated image with the input text, or A facial realism preserving loss that penalizes the dissimilarity between the generated face and the closest face in a set of external real faces.

19. An apparatus for generating images from text using a diffusion model to reduce bias, comprising: means for receiving text input; means for generating an image corresponding to the text input using the diffusion model; as well as means for calculating a gradient matrix for each image in said training image set; The diffusion model is optimized in the following way: Align the generated images towards the target distribution using a distribution alignment loss; as well as The diffusion model is tuned by directly fine-tuning the sampling process by adjusting the gradients to minimize the loss on the generated image.

20. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 15.

Images