Stable diffusion model construction method based on prejudice removing strategy

Abstract

The invention provides a stable diffusion model construction method based on a debias strategy, and aims to eliminate the prejudice in a pre-trained stable diffusion model, the scheme adopts a collaborative distance constraint debias strategy, improves the fairness of the stable diffusion model by optimizing the expression of a CLIP text encoder in an embedding space, and meanwhile, improves the stability of the stable diffusion model. A two-stage text guide sampling strategy is introduced, the strategy switches different CLIP text encoders at key time points in the denoising process so as to eliminate potential prejudice and maintain image quality, compared with the prior art, expensive full-model fine tuning and dependence on an external reference model are avoided, and the method capable of maintaining image generation quality and improving image quality is realized. And attribute prejudice is effectively reduced. According to the scheme, the stable diffusion model prejudice is removed in a lightweight manner.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, specifically the field of computer vision and generative artificial intelligence, and more specifically, to a method for constructing a stable diffusion model based on a debiasing strategy. Background Art

[0002] Stable diffusion models are a key research area in computer vision and generative AI, particularly in text-to-image generation. The model's core mechanism employs denoising techniques to generate high-quality images that match text descriptions from random noise. As a leading example in this field, the Stable Diffusion model combines latent variable models with text-guided techniques to efficiently generate high-resolution images. This approach not only improves image generation efficiency but also ensures high consistency between the generated images and the text descriptions.

[0003] Existing research shows that due to the distribution bias of training data, the CLIP text encoder in the stable diffusion model may form biased representations in the embedding space. This bias is particularly evident in the representation of social attributes such as gender, race, and age. The existence of such bias not only undermines the fairness of the model, but also may bring ethical issues in practical applications. To alleviate this problem, current research has proposed two types of bias removal methods: one is the full-model fine-tuning method, which adjusts the generation distribution by jointly optimizing multiple components of the stable diffusion model. However, this method has high computational overhead and may even lead to image quality degradation; the other is the method of using external reference models or additional constraints to adjust the generation results in the inference stage. This method may increase sampling costs and even introduce new implicit biases, thereby affecting the model's generalization ability and real-time performance.

[0004] In summary, the main reason for bias in the stable diffusion model is that the CLIP text encoder's encoding forms an uneven distribution in the embedding space due to the uneven distribution of training samples. In addition, among existing bias removal schemes for the stable diffusion model, mainstream methods focus on full model fine-tuning and reducing bias with the help of external reference models or adding additional constraints. However, these traditional methods have significant drawbacks. Full model fine-tuning often leads to high computational costs, and while relying on external models or constraints can weaken some bias, it can easily cause a decline in image generation quality and may even introduce new types of bias, making it difficult to achieve a balance between bias removal and image quality assurance. Therefore, there is an urgent need for a low-cost, generalizable stable diffusion model debiasing solution that does not rely on additional auxiliary networks.

[0005] It should be noted that this background information is provided solely to introduce relevant information of the present invention to facilitate understanding of the technical solution of the present invention. It does not necessarily constitute prior art. In the absence of evidence demonstrating that the relevant information was disclosed prior to the filing date of the present invention, the relevant information should not be considered prior art. Summary of the Invention

[0006] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a method for constructing a stable diffusion model based on a debiasing strategy.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a method for constructing a stable diffusion model based on a debiasing strategy is proposed, wherein the stable diffusion model is used to generate a corresponding image based on a given prompt text, and the method comprises: step S1, obtaining a pre-trained stable diffusion model and a CLIP image encoder, wherein the pre-trained stable diffusion model comprises an initial CLIP text encoder and a decoder, wherein the initial CLIP text encoder is used to generate a text encoding from the prompt text, and the decoder is used to generate a corresponding image based on the text encoding; step S2, performing multiple rounds of iterative training on the initial CLIP text encoder to obtain a debiased CLIP text encoder, wherein each round of training comprises: step S21, obtaining a training set, wherein the training set comprises multiple samples, each sample comprises a prompt text and its corresponding image encoding, wherein the prompt text comprises a neutral prompt word and a The prompt word describing the attribute or multiple prompt words describing the attribute that do not belong to the same attribute category, the image encoding is to use the CLIP image encoder to perform encoding operations on the image generated by the pre-trained stable diffusion model based on the prompt text, and perform preprocessing operations according to the preset attention strategy to obtain the image encoding; step S22, calculate the semantic center of each attribute category based on all image encodings in the training set, and configure the preset first loss function and the preset second loss function based on the calculated semantic center; step S23, based on the training set obtained in step S22 and the preset first loss function and the preset second loss function configured in step S22, use the LoRA training technology to train the CLIP text encoder to convergence; S3, the de-biased stable diffusion model is formed by the CLIP text encoder, the original CLIP text encoder and the decoder.

[0009] Preferably, the preset attention strategy is:

[0010]

[0011]

[0012]

[0013] in, represents the activation function of the attention strategy, 、 and represent the query matrix, key matrix, and value matrix of the attention strategy, respectively. represents the dimension of the bond matrix, Represents the CLIP image encoder based on the pre-trained stable diffusion model based on the inclusion property With neutral prompt words The original image encoding is obtained by encoding the image generated by the prompt text. Represents the new image encoding obtained by encoding the original image using the attention strategy, Indicates that the CLIP text encoder contains attributes With neutral prompt words The text encoding of the prompt text.

[0014] Preferably, in step S23, the semantic center of each attribute category is calculated in the following manner: all image codes corresponding to the prompt text containing the same attribute category are obtained from the training set, the average value of all the obtained image codes is calculated, and the calculated average value is used as the semantic center of the attribute category.

[0015] Preferably, the preset first loss function is:

[0016]

[0017]

[0018] in, Represents a set of all attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates a neutral prompt word, Represents the CLIP text encoder for neutral prompt words The text encoding, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

[0019] Preferably, the preset second loss function is:

[0020]

[0021]

[0022] in, Represents a set of all opposing attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates that it does not belong to the set The neutral prompt word of the corresponding attribute category, Indicates that the CLIP text encoder contains attributes With neutral prompt words The text encoding of the prompt text, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

[0023] Preferably, the debiased stable diffusion model is configured in the following manner: when generating an image using the debiased stable diffusion model, the original CLIP text encoder is used to generate a text encoding guide based on the prompt text to construct the basic structure of the image, and at a preset moment, the original CLIP text encoder is switched to the debiased CLIP text encoder, and the debiased CLIP text encoder is used to generate a text encoding guide based on the prompt text to construct the attributes of the image.

[0024] Preferably, the preset time is ,in, The total time required to denoise the image generated by the debiased stable diffusion model.

[0025] According to a second aspect of the present invention, an image generation method is proposed for generating a specified image, the method comprising: step S1, constructing a debiased stable diffusion model using the method described in any one of claims 1 to 8; step S2, generating the specified image based on a prompt text describing the specified image using the constructed debiased stable diffusion model.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed by a processor to implement the steps of any one of the methods described in the first aspect of the present invention.

[0027] According to the fourth aspect of the present invention, an electronic device is proposed, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of any one of the methods described in the first aspect of the present invention by executing the executable instructions.

[0028] Compared with the prior art, the advantages of the present invention are:

[0029] The present invention proposes a method for constructing a stable diffusion model based on a debiasing strategy. This method fundamentally improves the fairness of image generation of the stable diffusion model by eliminating the inherent bias of the CLIP text encoder at the embedding space level. At the same time, a two-stage text-guided sampling strategy is introduced, which switches different CLIP text encoders at key time points in the denoising process to eliminate potential bias and maintain image quality. Compared with the existing technology, the present invention avoids expensive full-model fine-tuning and dependence on external reference models, and realizes a lightweight stable diffusion model debiasing method that can maintain image generation quality and effectively reduce attribute bias. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0031] Figure 1 2. A schematic flow chart of a lightweight bias removal method for a stable diffusion model according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of experimental data of a comparative experiment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0034] As mentioned in the background technology section, in the existing bias elimination technology of stable diffusion models, the mainstream methods focus on fine-tuning the entire model, and reducing bias with the help of external reference models or adding additional constraints. However, these traditional methods have significant disadvantages. Fine-tuning the entire model often leads to high computational costs. Although relying on external models or constraints can weaken some biases, it is easy to cause a decline in image generation quality and may even introduce new types of bias, making it difficult to achieve a balance between bias elimination and image quality assurance.

[0035] It should be noted that: the inventor believes that the uneven distribution of training samples in the stable diffusion model will cause the encoding results of the CLIP text encoder to form an uneven distribution in the embedding space, which will lead to bias in the images generated by the stable diffusion model. In other words, the distance between the position of the text encoding of different attributes in the same attribute category by the text encoder in the embedding space and the position of the semantic center of the attribute category is not equal, resulting in inconsistent probabilities of the stable diffusion model generating images of different attributes in the same attribute category. Therefore, the inventor proposes to transform the generation probability fairness problem into a distance equality constraint to guide the removal of bias in the stable diffusion model. Specifically, in a pre-trained stable diffusion model, assuming is the set of properties of the image generated by the stable diffusion model. For any opposing properties , given a neutral word , if the model satisfies the generation probability fairness, that is: , the fair conversion of the generation probability into the constraint of equal distance can be expressed as:

[0036]

[0037] in, Represents the encoding operation performed by the CLIP text encoder or the CLIP image encoder, The encoding operation performed for the CLIP text encoder, Indicates that it contains description attributes and neutral prompt words The text encoding or image encoding corresponding to the prompt text prompt word concept and encoding, Indicates that it contains description attributes and neutral prompt words The text encoding or image encoding corresponding to the prompt text prompt word concept and encoding, Indicates prompt words The corresponding text encoding. For example, assume is the opposite attribute in the gender attribute category, that is Indicates male, Indicates female, neutral word For a doctor, the above formula indicates that the distance between the text encoding or image encoding corresponding to the prompt text "male doctor" and the text encoding corresponding to the prompt text "doctor" in the stable diffusion model, as well as the distance between the text encoding or image encoding corresponding to the prompt text "male doctor" and the text encoding corresponding to the prompt text "doctor" in the stable diffusion model are equal, that is, the probability of the stable diffusion model generating images of "male doctor" and "female doctor" is equal, that is, it is believed that the stable diffusion model has no bias in gender attributes.

[0038] According to one embodiment of the present invention, based on the reasons for the bias in the stable diffusion model and the inventor's proposal to convert the generation probability fairness into a constraint of equal distance, the present invention proposes a method for constructing a stable diffusion model based on a debiasing strategy. This method eliminates the inherent bias of the CLIP text encoder at the embedding space level, thereby fundamentally improving the fairness of image generation by the stable diffusion model. At the same time, a two-stage text-guided sampling strategy is introduced. In the denoising process of the stable diffusion model, two differentiated CLIP text encoders are used for collaborative guidance, which can not only ensure the high quality of the generated image, but also significantly reduce attribute bias such as gender and race, thereby achieving dual optimization of fairness and image quality.

[0039] Before introducing the present invention, the following explanation is required. In the present invention, the central prompt word refers to a text prompt word that does not contain any specific attribute or tendency information, such as doctor, teacher, lawyer, etc., while the attribute description prompt word refers to a text prompt word that contains specific attribute or feature information, which is used to guide the generation model to create outputs with specific characteristics, such as male, female, black skin, white skin, fat, thin, etc. In addition, an attribute category is a set that contains all opposite attributes belonging to the same type. For example, the gender attribute category is a set that contains two opposite attributes of male and female, and the skin color attribute is a set that contains three opposite attributes of black skin, white skin and yellow skin. It should be understood that the language used in the text is determined by the stable diffusion model, and the present invention does not impose any restrictions on the language used in the text. In addition, the neutral prompt words, attribute description prompt words and attribute categories listed above are all illustrative and non-exhaustive.

[0040] According to one embodiment of the present invention, the method for constructing a stable diffusion model based on a debiasing strategy proposed in the present invention is used to remove bias in a pre-trained stable diffusion model. Figure 1, the figure shows the steps S1, S2 and S3 of the method for constructing a stable diffusion model based on a debiasing strategy, wherein, in step S1, a pre-trained stable diffusion model and a CLIP image encoder are obtained, and the pre-trained stable diffusion model includes an initial CLIP text encoder and a decoder, wherein the initial CLIP text encoder is used to generate a text encoding from a prompt text, and the decoder is used to generate a corresponding image based on the text encoding; step S2, multiple rounds of iterative training are performed on the initial CLIP text encoder to obtain a debiased CLIP text encoder, wherein each round of training includes: step S21, obtaining a training set, the training set includes multiple samples, each sample includes a prompt text and its corresponding image encoding, wherein the prompt text includes a neutral prompt word and a prompt word describing an attribute or multiple prompt words that do not belong to the same attribute The image encoding is obtained by sequentially using the CLIP image encoder to perform encoding operations on the image generated by the pre-trained stable diffusion model based on the prompt text, and performing preprocessing operations on the encoding results of the image encoder according to the preset attention strategy; step S22, based on all image encodings in the training set, the semantic center of each attribute category is calculated, and the preset first loss function and the preset second loss function are configured based on the calculated semantic center; step S23, based on the training set obtained in step S22 and the preset first loss function and the preset second loss function configured in step S22, the CLIP text encoder is trained to convergence using the LoRA training technology; S3, based on the debiased CLIP text encoder, the original CLIP text encoder and the decoder constitute a debiased stable diffusion model. In order to better understand the present invention, each step of the present invention will be described in detail below in conjunction with specific embodiments.

[0041] In order to ensure that the content generated by the stable diffusion model remains balanced and unbiased in multiple attribute categories such as gender and race, calculating the fairness benchmark of the attribute category is crucial to removing model bias. According to one embodiment of the present invention, the present invention obtains the fairness benchmark of different attribute categories by determining the semantic center of different attribute categories in the embedding space. Specifically, it is first necessary to obtain the image encoding corresponding to different attributes in different attribute categories, and then calculate the semantic center of different attribute categories based on the generated image encoding.

[0042] According to one embodiment of the present invention, when the present invention scheme obtains a training set, first, the pre-trained stable diffusion model is allowed to generate a corresponding image based on a given prompt text, and then the generated image is encoded into an image code using the CLIP image encoder. In addition, each image code in the training set may contain only one attribute type or multiple attribute types, but all image codes in the training set need to cover all attributes in each attribute category. For example, when only the gender attribute is selected to construct the image training set, if the central prompt word is configured as doctor, then the training set should contain prompt texts with the same or similar descriptions as "male doctor" and "female doctor", as well as multiple image codes corresponding to "male doctor", and image codes corresponding to "female doctor". When constructing the image training set with multiple attribute categories, assuming that gender and skin color are selected as two attribute categories, then the training set should contain prompt texts with the same or similar descriptions as "male black doctor", "male white doctor", "male yellow doctor", "female black doctor", "female white doctor" and "female yellow doctor", as well as image codes corresponding to each prompt text.

[0043] In order to reduce the influence of background information on semantic center calculation in image coding and enhance attention to foreground objects in image coding, according to one embodiment of the present invention, in step S22 of the present invention, an attention strategy is used to preprocess the image code obtained after encoding by the CLIP image encoder, and then the semantic center is calculated based on the preprocessed image code, wherein the attention strategy is configured as follows:

[0044] ,

[0045] ,

[0046] ,

[0047] in, represents the activation function of the attention strategy, 、 and represent the query matrix, key matrix, and value matrix of the attention strategy, respectively. represents the dimension of the bond matrix, Represents the CLIP image encoder based on the pre-trained stable diffusion model based on the inclusion property With neutral prompt words The original image encoding is obtained by encoding the image generated by the prompt text. Represents the new image encoding obtained by encoding the original image using the attention strategy, Indicates that the CLIP text encoder contains attributes With neutral prompt words It should be understood that the present invention adaptively configures key parameters of the attention strategy (also known as the attention mechanism) to enhance focus on foreground objects in image encoding. In addition, the use of the attention strategy to preprocess image encoding is a well-known technology in the art and is not further described here.

[0048] According to one embodiment of the present invention, in step S23 of the present invention, all image codes corresponding to prompt texts containing the same attribute category are obtained from the training set, and the average value of all the obtained image codes is calculated, and the calculated average value is used as the semantic center of the attribute category to obtain fairness benchmarks for different attribute categories.

[0049] According to one embodiment of the present invention, a collaborative distance constraint strategy is adopted in the present invention to guide the training process of the CLIP text encoder. Specifically, a preset first loss function and a preset second loss function are configured according to the calculated semantic center, so that when training the CLIP text encoder, the distance from the text encoding to the semantic center of each category is balanced to reduce the inherent bias of the text encoding, wherein the preset first loss function is used to ensure that the stable diffusion model has consistent probability when generating images of different opposing attributes in the same attribute category; the preset second loss function is used to ensure that the stable diffusion model has consistent quality when generating images of different opposing attributes in the same attribute category.

[0050] According to one embodiment of the present invention, in the present invention, the preset first loss function is expressed as:

[0051]

[0052]

[0053] in, Represents a set of all attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates a neutral prompt word, Represents the CLIP text encoder for neutral prompt words The text encoding, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

[0054] According to one embodiment of the present invention, in the present invention, the preset second loss function is expressed as:

[0055]

[0056]

[0057] in, Represents a set of all opposing attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates that it does not belong to the set The neutral prompt word of the corresponding attribute category, Indicates that the CLIP text encoder contains attributes With neutral prompt words The text encoding of the prompt text, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

[0058] According to one embodiment of the present invention, in step S23 of the present invention, the CLIP text encoder is trained to convergence using the LoRA training technology based on the training set obtained in the current iteration period and the preset first loss function and the preset second loss function configured in the current iteration period, and the CLIP text encoder in the stable diffusion model is trained to convergence, wherein, during the training process, the CLIP text encoder is trained based on the LoRA technology (Low-Rank Adaptation, a technology for fine-tuning pre-trained models), and the LoRA technology achieves parameter fine-tuning by introducing a low-rank structure into the weight matrix of the CLIP text encoder, so that the CLIP text encoder can be effectively adjusted without significantly increasing the number of parameters of the CLIP text encoder.

[0059] It should be noted that LoRA technology is a technology for efficiently fine-tuning pre-trained models, aiming to solve the computational and storage cost problems when fine-tuning large pre-trained models on specific tasks. This technology achieves parameter fine-tuning by introducing a low-rank structure in the model's weight matrix, so that the model can be effectively adjusted without significantly increasing the number of model parameters. For the present invention, the weight matrix W of the trained CLIP text encoder is decomposed into the product of two low-rank matrices, namely W=W0+ΔW, where W0 is the weight matrix of the original CLIP text encoder, and ΔW is the adjustment matrix obtained by multiplying the low-rank matrices. ΔW can be expressed as ΔW=AB, where A and B are two low-rank matrices. During the training process, only the parameters of the two matrices A and B will be updated, while the weight matrix W0 of the original CLIP text encoder remains unchanged. The number of parameters of the CLIP text encoder only increases the number of parameters of A and B, which is much smaller than the number of parameters of the entire weight matrix. It should be understood that using LoRA technology to perform training tasks is a well-known technology for those skilled in the art and will not be described in detail here.

[0060] According to one embodiment of the present invention, in the present invention, the original CLIP text encoder can obtain a debiased CLIP text encoder after multiple rounds of training in step S2. Although the debiased CLIP text encoder can remove the inherent bias in the original CLIP text encoder, directly using the debiased CLIP text encoder as the CLIP text encoder of the stable diffusion model may affect the quality and assurance of the raw image. Therefore, the present invention proposes a two-stage text-guided sampling strategy, in which the original CLIP text encoder of the pre-trained stable diffusion model and the debiased CLIP text encoder obtained by training are used together as the CLIP text encoder of the pre-trained stable diffusion model to construct a debiased stable diffusion model. Since low-frequency information (non-attribute information) is formed in the early denoising stage of the image generated by the stable diffusion model, and high-frequency information (attribute-related information) appears in the later denoising stage of the image generated by the stable diffusion model, the present invention also needs to configure the debiased stable diffusion model as follows: when using the debiased stable diffusion model to generate an image, the specified debiased stable diffusion model first uses the text encoding generated by the original text encoder to guide the denoising process to ensure the overall structure of the generated image is stable, and then uses the text encoding generated by the debiased CLIP text encoder to guide the denoising process to control the attributes of the final generated image.

[0061] In order to ensure the fairness of image generation and minimize the impact on image quality, it is necessary to let the debiased CLIP text encoder intervene in the appropriate denoising stage. According to one embodiment of the present invention, in the present invention, the selection time( represents the time required for the denoising process when the stable diffusion model generates an image), the original CLIP text encoder is switched to the debiased CLIP text encoder. It should be understood that the above-mentioned switching time of the CLIP text encoder is only an exemplary preferred embodiment and is not exhaustive. The implementer can configure it according to needs.

[0062] According to one embodiment of the present invention, the present invention also proposes an image generation method, which includes: constructing a stable diffusion model with debiased based on a stable diffusion model construction method based on a debiasing strategy proposed by the present invention, and using the constructed stable diffusion model with debiased to generate a corresponding image based on a prompt text describing the required image.

[0063] To more clearly illustrate the present invention, a specific example is provided below to describe how the method proposed in the present invention can be used to correct the gender attribute of the occupation "doctor." The Stable Diffusion v1.5 model is used as the model, and the steps for constructing the diffusion model include:

[0064] (1) Construct a low-rank adaptation (LoRA) version of the CLIP text encoder and randomly initialize the LoRA parameters to replace the CLIP text encoder of the original Stable Diffusion v1.5 model;

[0065] (2) Generate 50 images each using the Stable Diffusion v1.5 model and the prompts “Photo portrait of a male doctor, a person” and “Photo portrait of a female doctor, a person”;

[0066] (3) Use the “adaptive foreground extraction strategy” to extract the foreground of all generated images, and obtain 50 foreground images of male doctors and 50 foreground images of female doctors;

[0067] (4) Use CLIP's image encoder to encode each image into a feature vector with the same dimension. For the 50 male doctor images, average the 50 feature vectors to obtain the center vector of the male doctor image. Do the same for the female doctor image to obtain the center vector of the female doctor image.

[0068] (5) For the text “Photo portrait of a doctor, a person”, use the low-rank adaptation (LoRA) version of the text encoder to encode and obtain a vector of the doctor’s text;

[0069] (6) Calculate the loss function and , and back-propagate to update the parameters of the low-rank adaptation (LoRA) version of the text encoder;

[0070] (7) Repeat steps 2-6 for a total of 160 times to allow the model to converge. Finally, a trained low-rank adaptation (LoRA) version of the text encoder is obtained.

[0071] (8) The original CLIP text encoder and the trained low-rank adaptation (LoRA) version of the CLIP text encoder are used to replace the CLIP text encoder in the original Stable Diffusion v1.5 model to obtain the final stable diffusion model, and the stable diffusion model is made to be more efficient in the early stage ( ), the original text encoder of the Stable Diffusion v1.5 model is used for denoising, and in the later stage ( When , the trained low-rank adaptation (LoRA) version of the text encoder is used for denoising. Finally, a debiased image is obtained.

[0072] According to one embodiment of the present invention, the present invention proposes a method for constructing a stable diffusion model based on a debiasing strategy, which aims to eliminate bias in a pre-trained stable diffusion model. The scheme adopts a collaborative distance constrained debiasing strategy to improve the fairness of the stable diffusion model by optimizing the representation of the CLIP text encoder in the embedding space. At the same time, a two-stage text-guided sampling strategy is introduced, which switches different CLIP text encoders at key time points in the denoising process to eliminate potential bias and maintain image quality. Compared with the prior art, the present invention avoids expensive full-model fine-tuning and dependence on external reference models.

[0073] To demonstrate the beneficial effects of the present invention, the inventors conducted comparative experiments. Specifically, they used the LightFair model (a stable diffusion model with debiased properties constructed using the debiased stable diffusion model construction method proposed in the present invention) to generate a large number of images. The experiments were conducted on an NVIDIA 4090 GPU. The image generation task covered six occupations (doctor, CEO, taxi driver, nurse, artist, and teacher), all using a fixed prompt template: "Photo portrait of a {occupation}, a person." The initial learning rate varied slightly depending on the version of the generative model used and the specific attribute category. The overall configuration was and The batch size of all experiments is unified as 50, and the total number of training rounds is 160. The experimental results are as follows Figure 2As shown in the table, analyzing the data in the figure shows that the present invention achieves good performance in both fairness and image quality. Furthermore, it achieves excellent performance across different versions of the generative model (Stable Diffusion v1.5 and StableDiffusion v2.1) and across different attributes (gender and skin color).

[0074] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0075] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0076] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0077] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for constructing a stable diffusion model based on a debiasing strategy, wherein the stable diffusion model is used to generate a corresponding image based on a given prompt text, characterized in that: The method comprises: Step S1: obtaining a pre-trained stable diffusion model and a CLIP image encoder, wherein the pre-trained stable diffusion model includes an initial CLIP text encoder and a decoder, wherein the initial CLIP text encoder is used to generate a text code from a prompt text, and the decoder is used to generate a corresponding image based on the text code; Step S2: Perform multiple rounds of iterative training on the initial CLIP text encoder to obtain a debiased CLIP text encoder, wherein each round of training includes: Step S21: Obtain a training set, where the training set includes multiple samples, each sample including a prompt text and its corresponding image encoding, wherein the prompt text includes a neutral prompt word and a prompt word describing an attribute or multiple prompt words describing attributes that do not belong to the same attribute category, and the image encoding is obtained by sequentially using a CLIP image encoder to perform encoding operations on an image generated based on the prompt text using a pre-trained stable diffusion model, and performing preprocessing operations according to a preset attention strategy; Step S22: Calculate the semantic center of each attribute category based on all image codes in the training set, and configure a preset first loss function and a preset second loss function based on the calculated semantic center; Step S23: training the CLIP text encoder to convergence using the LoRA training technology based on the training set obtained in step S22 and the preset first loss function and the preset second loss function configured in step S22; S3, the debiased CLIP text encoder, the original CLIP text encoder and the decoder constitute a debiased stable diffusion model.

2. The method according to claim 1, characterized in that The preset attention strategy is: in, represents the activation function of the attention strategy, 、 and represent the query matrix, key matrix, and value matrix of the attention strategy, respectively. represents the dimension of the bond matrix, Represents the CLIP image encoder based on the pre-trained stable diffusion model based on the inclusion property With neutral prompt words The original image encoding is obtained by encoding the image generated by the prompt text. Represents the new image encoding obtained by encoding the original image using the attention strategy, Indicates that the CLIP text encoder contains attributes With neutral prompt words The text encoding of the prompt text.

3. The method according to claim 1, characterized in that In step S23, the semantic center of each attribute category is calculated as follows: All image codes corresponding to the prompt text containing the same attribute category are obtained from the training set, the average value of all the obtained image codes is calculated, and the calculated average value is used as the semantic center of the attribute category.

4. The method according to claim 1, wherein The preset first loss function is: in, Represents a set of all attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates a neutral prompt word, Represents the CLIP text encoder for neutral prompt words The text encoding, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

5. The method according to claim 1, wherein The preset second loss function is: in, Represents a set of all opposing attributes in an attribute category. Representing a collection The size of Representing a collection The index of the attribute in , Representing a collection The properties, Indicates that it does not belong to the set The neutral prompt word of the corresponding attribute category, Indicates that the CLIP text encoder contains attributes With neutral prompt words The text encoding of the prompt text, Represents the set of configurations for the current iteration The semantic center of the corresponding attribute category, Indicates that the training set contains attributes With neutral prompt words The image code corresponding to the prompt text, Represents a function that calculates cosine distance.

6. The method according to claim 1, characterized in that Configure the debiased stable diffusion model as follows: When generating an image using a debiased stable diffusion model, the original CLIP text encoder is used to generate text encoding guidance based on the prompt text to construct the basic structure of the image. At a preset moment, the original CLIP text encoder is switched to the debiased CLIP text encoder, and the debiased CLIP text encoder is used to generate text encoding guidance based on the prompt text to construct the image attributes.

7. The method according to claim 1, characterized in that The preset time is ,in, The total time required to denoise the image generated by the debiased stable diffusion model.

8. An image generation method for generating a specified image, characterized in that: The method comprises: Step S1, constructing a debiased stable diffusion model using the method according to any one of claims 1 to 8; Step S2: Generate a designated image based on the prompt text describing the designated image using the constructed debiased stable diffusion model.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 9.

10. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method according to any one of claims 1 to 9 by executing the executable instructions.

Images