A debiasing method and system for image generation models based on enhanced cue word learning

Abstract

The present application relates to the field of machine learning, and more specifically, to a method and system for debiasing an image generation model based on enhanced prompt word learning. The method comprises: defining debiasing categories and obtaining a target distribution of the debiasing categories; obtaining original text and setting prompt words, and obtaining prompt text features based on the original text and the prompt words; obtaining a reference text set, and obtaining reference text features based on the reference text set; inputting the prompt text features into an image generation model to generate an image result; evaluating the image result, obtaining the actual distribution of debiasing categories in the image result, training the prompt text features based on the actual distribution, the target distribution, and the reference text features, and obtaining trained prompt text features; and then inputting the prompt text features into the image generation model again to obtain a debiased image result. This method can achieve efficient and controllable debiasing of the image generation model, reduce computational overhead, and improve the adaptability of the debiasing method.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and more specifically, to a method and system for debiasing an image generation model based on reinforcement cue word learning. Background Art

[0002] Existing image generation models, particularly text-to-image (T2I) models, can generate images based on user-provided text descriptions. However, during model training, dataset feature imbalances can cause the model to inherit or acquire bias within the dataset or algorithm, resulting in imbalanced features in the generated images. Unbalanced image generation can negatively impact social diversity and balance, making research on methods to eliminate model bias crucial. Currently, methods for eliminating model bias can be broadly categorized as data-driven and model-driven. Data-driven methods minimize bias by modifying or rebalancing training data. This typically involves retraining the model or fine-tuning parameters, but data-driven methods often suffer from high computational complexity. Model-driven methods control biased output during model generation by adjusting the model structure or introducing a debiasing module. However, these methods are sensitive to specific model architectures, necessitating improvements to model debiasing methods. Summary of the Invention

[0003] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide a method and system for debiasing an image generation model based on reinforced cue word learning, thereby achieving efficient and controllable debiasing of the image generation model, reducing computational overhead, and improving the adaptability of the debiasing method.

[0004] According to a first aspect of the present application, a method for debiasing an image generation model based on reinforcement cue word learning is provided, the method comprising:

[0005] defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category;

[0006] Obtaining an original text, setting a prompt word according to the original text and the debiased category, and obtaining a prompt text feature according to the original text and the prompt word;

[0007] Acquire a reference text set according to the original text and the debiased category, and acquire a plurality of reference text features according to the reference text set;

[0008] Inputting the prompt text features into a pre-trained image generation model to generate an image result;

[0009] Evaluating the image result, obtaining an actual distribution of debiased categories in the image result, and training the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the image result evaluation and the prompt text feature training are performed on a pre-trained evaluation model;

[0010] The trained prompt text features are input into the image generation model again to obtain a debiased image result.

[0011] It is understandable that the image results generated by the prompt text features are statistically distributed, and the actual distribution is compared with the target distribution of the debiased category, and the prompt text features are trained according to the deviation value of the actual distribution and the target distribution, so as to obtain the debiased image results; this method does not require adjustment of the parameters inside the image generation model, nor does it modify the internal components in the image generation model. It only adds a prompt word to the input original text and trains it. The sample data required for training is only simple text data and a small amount of image data, which can save a lot of computing resources for adjusting the parameters and structure of the image generation model; furthermore, in the step of training the prompt text features, the concept of vector is introduced, which can guide the prompt text features to train slowly and directionally in the direction corresponding to the target distribution, improve the accuracy of training, and prevent the problem of overfitting caused by excessive training; at the same time, there is no need to adjust the parameters and structure of the image generation model, and it can be applied to image generation models of different versions and types, thereby improving the adaptability of the debiasing method.

[0012] Optionally, obtaining prompt text features according to the original text and the prompt word includes:

[0013] Vectorize the original text to obtain the original text vector ;

[0014] Vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector;

[0015] The zero-initialized hint word vector Add to the original text vector Get the prompt text vector ;

[0016] The prompt text vector Perform text encoding to obtain prompt text features .

[0017] It is understandable that the prompt word is related to the debiasing category and is used to place words related to the debiasing category in the original text, so that the debiasing type information can be reflected in the original text; it is understandable that when the image generation model is trained, the input samples are statistically unbalanced in the debiasing category, so that the trained image generation model is inevitably biased in the debiasing category; if the user's demand is to use a balanced image generated by text, the image generation model must be debiased; here, after the original text and the prompt word are vectorized, the prompt word vector is added to the original text vector, so that the original text has a control word for the debiasing category, and the prompt word can be adjusted to control the proportion of the image result, and the vectorized data of the text is used, which can save more computing overhead and improve the efficiency of training.

[0018] Optionally, the acquiring of a plurality of reference text features according to the reference text set includes:

[0019] Set reference text set ,in is the i-th reference text, and n represents the total value of the reference texts;

[0020] Each of the reference texts in the reference text set After vectorization, perform text encoding to obtain the corresponding reference text features .

[0021] It can be understood that the reference text is a multiple text value based on the original text and the debiased category, representing a single direction of each debiased category; the reference text is introduced here to provide a clear optional direction for training the prompt text feature, so that the prompt text feature training refers to the direction of each debiased category, and achieves a balanced training result for each debiased category; the reference text is also vectorized here, and can use a unified data format with the prompt text, and the vector has directionality, which can make the prompt text training more accurate.

[0022] Optionally, evaluating the image result to obtain an actual distribution of debiased categories in the image result includes:

[0023] Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result;

[0024] According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

[0025] It's understandable that the prompt text initially input to the image generation model is equivalent to the user's original text, as the added prompt words are zero-initialized. The resulting image, however, reflects the model's inherent bias. This bias needs to be statistically recorded on labels to determine the actual distribution of debiased categories before further debiasing can be performed. The image results are evaluated using a pre-trained evaluation model, which quickly determines the debiasing level of the image generation model and improves the efficiency of the entire debiasing method.

[0026] Optionally, the training of the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature includes:

[0027] Comparing the actual distribution with the target distribution; if the actual distribution conforms to the target distribution, taking the image result as a debiased image result, and completing the training of the prompt word feature; if the actual distribution does not conform to the target distribution, obtaining the deviation between the actual distribution and the target distribution, and calculating a reward R based on the deviation, wherein the deviation and the reward R are obtained through the evaluation model;

[0028] Adjust the distance between the prompt text feature and each reference text feature according to the reward R and the loss function to obtain the adjusted prompt text feature;

[0029] The adjusted prompt text feature is input into the image generation model again to generate a new image result, and the new image result is continuously evaluated until the training of the prompt word feature is completed to obtain the trained prompt text feature.

[0030] It is understandable that the above-mentioned training method of prompt text features is aimed at ensuring that the actual distribution of the acquired image results conforms to the target distribution of the debiased category, wherein the distance between the prompt text features and each reference text feature is adjusted according to the deviation between the actual distribution and the target distribution, so that the generated image results conform to the proportion of the target distribution, thereby achieving a debiased effect; this method can gradually and slowly reduce the bias of the image generation model, and adjust the bias in combination with the reward R and loss function to improve the efficiency of training.

[0031] Optionally, the loss function specifically includes:

[0032]

[0033] in, is the distance difference between the prompt text feature and all the reference text features, represents the similarity calculation between the prompt text feature and the i-th reference text feature, is the reward component, representing the reward value of the prompt text feature and the i-th reference text feature, Indicates the Reference text features.

[0034] It can be understood that calculating the similarity between the prompt text feature and the reference text feature is equivalent to obtaining the similarity between the prompt text feature and the i-th reference text feature, and calculating the distance difference between the prompt text feature and the i-th reference text feature based on the reward R obtained by the distribution deviation, and subsequently training the prompt text feature based on this difference; this method can train the prompt text feature through the vectorized direction and specific data of the distribution deviation, so that the training process has a clear direction and measurement, avoids the overfitting problem caused by excessive training or the problem of poor debiasing effect, and improves the accuracy of the debiasing effect.

[0035] Optionally, adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function includes:

[0036] The reward The amount of each reward Includes positive and negative values;

[0037] If the reward amount If it is a positive value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. direction adjustment, reducing the difference between the prompt text feature and the corresponding i-th reference text feature distance;

[0038] If the reward amount If it is a negative value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. Adjust in the opposite direction to increase the difference between the prompt text feature and the corresponding i-th reference text feature distance.

[0039] Understandably, the reward It can reflect the deviation between the real-time distribution and the target distribution, and affect the loss function through positive and negative data, thereby affecting the direction of the prompt text feature adjustment; this is conducive to the precise movement of the prompt text feature in the direction of training, and achieves the effect of slowly fine-tuning the prompt text feature.

[0040] According to a second aspect of the present application, a system for debiasing an image generation model based on reinforcement cue word learning is provided, comprising:

[0041] A target distribution acquisition module is used to define a debiasing category and acquire a target distribution of the debiasing category according to the debiasing category;

[0042] A prompt text feature acquisition module is used to acquire original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words;

[0043] A reference text acquisition module, configured to acquire a reference text set based on the original text and the debiased category, and acquire a plurality of reference text features based on the reference text set;

[0044] An image generation module is used to input the prompt text features into a pre-trained image generation model to generate an image result;

[0045] a training module for evaluating the image result, obtaining an actual distribution of debiased categories in the image result, and iteratively training the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed on a pre-trained evaluation model;

[0046] A debiased image module is generated, which is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0047] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the image generation model debiasing method based on reinforcement prompt word learning as described in the first aspect above.

[0048] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed, it implements the image generation model debiasing method based on reinforcement prompt word learning as described in the first aspect above.

[0049] Based on any one of the above aspects, the embodiment of the present application provides a method for statistically analyzing the actual distribution of the image results generated by the prompt text features, comparing the actual distribution with the target distribution of the debiased category, and training the prompt text features according to the deviation value between the actual distribution and the target distribution, thereby obtaining a debiased image result; this method does not require adjustment of the parameters within the image generation model, nor does it modify the internal components of the image generation model. It only adds a prompt word to the input original text and trains it, and introduces the concept of vector in the step of training the prompt text features, which can guide the prompt text features to train slowly and directionally in the direction corresponding to the target distribution, thereby improving the accuracy of the training, realizing efficient and controllable debiasing of the image generation model, reducing computational overhead, and improving the adaptability of the debiasing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work. Figure 1 This is a flow chart of a method for debiasing an image generation model based on reinforcement cue word learning provided in this embodiment.

[0051] Figure 2 This is a flow chart of obtaining prompt text features in this embodiment.

[0052] Figure 3 This is a flowchart of obtaining reference text features in this embodiment.

[0053] Figure 4 This is a flowchart of training prompt text features in this embodiment.

[0054] Figure 5 This is a diagram of adjusting the distance between the prompt text feature and each reference text feature in this embodiment.

[0055] Figure 6 This is a structural diagram of an image generation model debiasing system based on enhanced cue word learning in this embodiment.

[0056] Figure 7 This is a device structure diagram of the electronic device provided in this embodiment. DETAILED DESCRIPTION

[0057] The figures in this application are for illustrative purposes only and are not to be construed as limiting the present application. To better illustrate the following embodiments, some components in the figures may be omitted, enlarged, or reduced in size, and do not represent actual product dimensions. Those skilled in the art will appreciate that some well-known structures and their descriptions may be omitted from the figures.

[0058] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] In existing technologies, sample data often suffers from imbalances regarding certain categories, so image generation models trained using this sample data can be biased against certain categories. For example, one of the key areas of bias research in text-to-image (T2I) models within image generation models is targeting demographic features. During training, T2I models may inherit or develop biases against these features from the data or algorithms, resulting in a tendency to generate images favoring specific groups while neglecting other groups. For example, when a T2I model generates an image of a doctor without specifying the doctor's characteristics in the text, the generated image may be dominated by white males, with only a small proportion of other groups. Such image generation results can negatively impact the diversity and balance of society, making research on methods to eliminate model bias crucial.

[0061] Currently, methods for eliminating bias in T2I models can be divided into data-driven methods and model-driven methods. Data-driven methods minimize imbalances in training data by modifying or rebalancing the training data, which usually involves retraining or fine-tuning the parameters of the T2I model; model-driven methods control the biased output during the T2I model generation process by adjusting the structure of the T2I model or introducing a debiasing module. Although these methods have alleviated the bias problem to a certain extent, data-driven methods face the problem of high computational complexity, while model-driven methods are sensitive to specific model architectures. Therefore, it is necessary to improve the debiasing methods of image generation models to achieve efficient and controllable debiasing effects, thereby reducing computational overhead and adapting to different image generation models.

[0062] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below with reference to the accompanying drawings.

[0063] For example, the flowchart of the method for debiasing an image generation model based on enhanced prompt word learning is provided in the embodiment of the present application. Figure 1 As shown, the method can be divided into the following steps:

[0064] S110, defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category;

[0065] In this embodiment, the image generation model used is a pre-trained image generation model that can be put into use directly. Preferably, in this embodiment, a text-to-image (T2I) model of the image generation model is studied. It is understandable that when training an image generation model, the training data used has certain deviations and limitations, such as:

[0066] Data imbalance: If certain groups are over- or under-represented in the training data, the image generation model will be affected by these biases during the learning process, resulting in biased output. For example, if the training dataset contains far more male images than female images, the generated images may be more likely to show masculine features.

[0067] Single data source: The training data used is from the internet, which often reflects specific social and cultural backgrounds and may contain various biases and stereotypes.

[0068] Or when building an image generation model, there are inherent flaws in the algorithm design built into the image generation model, such as:

[0069] Lack of diversity: The importance of diversity may not be fully considered when designing image generation models, resulting in a lack of diversity and inclusiveness in the generated images;

[0070] Optimization goal deviation: The optimization goal of the image generation model may deviate from human values. For example, in some cases, the image generation model may focus more on image beauty rather than diversity or balance.

[0071] Therefore, based on the bias defects generated by the image generation model during construction and training mentioned above, it is necessary to eliminate these biases of the image generation model. It is understandable that in the debiasing work, it is necessary to define a debiasing category, which is the type of debiasing that the user wants to obtain in the image results generated by the image generation model with a balanced ratio. For example, if the user needs to generate an image result with a balanced ratio of male and female images, [gender] is selected as the debiasing category, and the debiasing category [gender] includes [male, female] values;

[0072] It is understandable that the debiasing category is not limited to a binary variable. You can also select a debiasing category that includes a polytomous variable. For example, [race] is selected as the debiasing category, including values ​​such as yellow race, black race, and white race.

[0073] Based on the selected debiasing category and the value of the debiasing category, the target distribution formed by the ratio of each value in the image result desired by the user is obtained. For example, if gender is the debiasing category, the user hopes to obtain an image with a 1:1 male-female ratio. Therefore, if the debiasing category conforms to the binary uniform distribution, the binary uniform distribution is used as the target distribution of the debiasing category.

[0074] It is understandable that the target distribution of the debiased category can be set to other statistical distributions, which are not listed here one by one.

[0075] S120: Obtaining an original text, setting prompt words according to the original text and the debiasing category, and obtaining prompt text features according to the original text and the prompt words;

[0076] In this embodiment, the user inputs a raw text into the image generation model, which serves as the initial tone for generating the image result. For example, "a group of doctors' photos" is input as the raw text. In the above steps, preferably, [gender] is selected as the debiasing category, including the values ​​[male, female], and the target distribution is a binary uniform distribution. It is understandable that in general image generation model training, the sample data may have a higher proportion of male doctors than female doctors. Therefore, the image generation model will be biased, and it is necessary to debias the image generation model so that the generated image result conforms to the target distribution.

[0077] According to the value of the debiasing category and the attributes of the original text, a prompt word is set to be inserted into the appropriate semantic position of the original text, which plays the role of prompting the debiasing category in the original text; for example, it can be simply understood that the prompt text with the prompt word added is "a group of images of [gender] doctors".

[0078] like Figure 2 As shown, the acquisition of prompt text features based on the original text and the prompt word can be subdivided into the following steps:

[0079] S121, vectorizing the original text to obtain an original text vector ;

[0080] In this embodiment, the original text, such as "a group of doctor's photos", is vectorized to obtain the original text vector. This enables the image generation model to understand and process the text content, facilitates the implementation of text content recognition, and brings format convenience to subsequent deviation calculation and training work.

[0081] S122, vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector;

[0082] In this application, we introduce hint words to debias the image generation model, drawing on the existing method of adding balance descriptions to text. The difference is that this method attempts to dynamically optimize this balance description in the continuous vector space of the reference text, rather than directly giving it in the form of natural language. Using the hint word learning method, we add an additional learnable hint word vector to the vectorized original text, denoted as , and optimize during training , enabling the image generation model to generate more balanced results. Unlike other methods, this paper uses the reinforcement learning paradigm to optimize the parameters of the prompt word. This design has two advantages:

[0083] 1) No pre-collected training data is required; the data used comes from the interaction between the model and the environment, that is, the data comes from the images generated by the image generation model itself;

[0084] 2) This debiasing method is highly generalizable. It only requires that the image generation model use a pre-trained text encoder to encode the text separately. There are no special requirements for other architectures of the image generation model. Using a text encoder to encode text is widely used in most current mainstream text-to-image generation models. Therefore, the debiasing method of this application is applicable to most text-to-image generation models.

[0085] In this embodiment, Zero initialization is performed to avoid The initial introduction of the hint word vector into the original text has a semantic impact, while also ensuring that the image generated by the original text with the hint word vector added for the first time is the original bias of the image generation model and can be used as a comparison benchmark for training results.

[0086] S123, initialize the prompt word vector after zero Add to the original text vector Get the prompt text vector ;

[0087] In this embodiment, the prompt word vector is added to the original text vector to obtain the prompt text vector, and then the prompt text vector is input into the text encoder to obtain the prompt text feature, which facilitates the image generation model to better process the text, capture the features of the text, and generate an image result that conforms to the text.

[0088] S124, the prompt text vector Perform text encoding to obtain prompt text features .

[0089] In this embodiment, the prompt text vector needs to be text encoded. Preferably, a text encoder can be used to encode the prompt text vector to obtain prompt text features, which can better compare the prompt text features and the reference text features in the same format and better calculate the deviation.

[0090] S130, obtaining a reference text set according to the original text and the debiased category, and obtaining a plurality of reference text features according to the reference text set;

[0091] In this embodiment, in addition to the need to introduce prompt words, it is also necessary to provide the direction and environment for prompt word training. Here, reference text is obtained based on the original text and the debiased category. For example, in the data mentioned above, reference texts of "a group of male doctor images" and "a group of female doctor images" can be obtained. These are two values ​​of the debiased category integrated into the original text, and are also the two directions of prompt word training. It can be understood that, unlike the original text, the reference text directly indicates the gender attribute of the generated image result content in the description. When the original text of the image generation model does not contain gender attributes, the gender ratio of the image results it generates is often uncertain and unbalanced. However, when the gender attribute is specified in the input text, the generated results are generally certain. For example, for the reference text "a group of male doctor images", the generated images are 100% male results, which is inconsistent with the user's desire to obtain a balanced image result. Therefore, it is necessary to use different reference texts as directions to train the prompt text features, so as to obtain a balanced image result.

[0092] It is understandable that the reference text is obtained according to the debiasing category value, so multiple reference texts can be obtained to form a reference text set, which represents multiple optimization directions for prompting text feature training.

[0093] Specifically, if Figure 3 As shown, the step of obtaining a plurality of reference text features according to the reference text set can be subdivided into the following steps:

[0094] S131, set the reference text set to ,in is the i-th reference text, and n represents the total value of the reference texts;

[0095] S132: Each of the reference texts in the reference text set After vectorization, perform text encoding to obtain the corresponding reference text features .

[0096] In this embodiment, several reference texts also need to be vectorized and then text encoded. Preferably, a text encoder can be used to encode the reference texts to obtain several reference text features, which can enable better comparison of the prompt text features and the reference text features in the same format, and better calculation of deviations.

[0097] S140, inputting the prompt text features into a pre-trained image generation model to generate an image result;

[0098] Understandably, debiasing methods for image generation models primarily focus on two areas. One approach fine-tunes the parameters of the image generation model to remove bias. However, these methods often involve significant computational overhead, especially when the entire image generation model needs to be retrained. This is not only computationally expensive but also places very demanding hardware requirements. Furthermore, these methods often require the collection of specific datasets, such as those for different types of bias. Another approach primarily optimizes certain components of the image generation model, such as debiasing the text encoding. This approach only considers one step in the text-to-image generation process, ignoring other steps in image generation that may also introduce or amplify bias. Furthermore, many current debiasing methods are designed for specific image generation model architectures, which often makes them less applicable as model architectures evolve and iterate. As image generation models continue to be optimized and updated, the structures and mechanisms of newer models may not align with the assumptions of existing debiasing methods, making these methods difficult to adapt to new architectures and scenarios.

[0099] In this embodiment, the generated image results are directly used as the sample set for training prompt text features, so that the debiasing method does not need to collect and process a large number of images, saving a lot of computational overhead. At the same time, there are no special requirements for the internal structure of the image generation model, which can improve the applicability of the debiasing method.

[0100] S150, evaluating the image result, obtaining an actual distribution of debiased categories in the image result, and training the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed on a pre-trained evaluation model;

[0101] In this embodiment, the generated image results need to be processed to obtain their actual distribution. It is understandable that the actual distribution obtained for the first time can reflect the bias that originally existed in the image generation model; the subsequent actual distributions reflect that the bias of the image generation model has been debiased after each training.

[0102] Specifically, evaluating the image result to obtain the actual distribution of debiased categories in the image result includes:

[0103] Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result;

[0104] According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

[0105] In this embodiment, the debiasing category of the image results is judged. It is understandable that the debiasing category of each image in a set of generated images can be identified, and finally the debiasing category ratio of the image results is obtained, thereby obtaining the actual distribution of the image results and providing necessary training data for subsequent prompt text feature training.

[0106] Specifically, if Figure 4 As shown, the training of the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature can be subdivided into the following steps:

[0107] S151, comparing the actual distribution with the target distribution. If the actual distribution conforms to the target distribution, the image result is regarded as a debiased image result, and the training of the prompt word feature is terminated. If the actual distribution does not conform to the target distribution, the deviation between the actual distribution and the target distribution is obtained, and a reward R is calculated based on the deviation, wherein the deviation and the reward R are obtained by the evaluation model.

[0108] In this embodiment, in order to guide the optimization of the prompt text vector, we evaluate the actual distribution and target distribution The reward is calculated based on the deviation between For example, taking the binary gender attribute as an example, let ,in , the target distribution is a binary uniform distribution, which comes from our optimization goal of balance, so .award for:

[0109]

[0110] S152, adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function, and obtaining an adjusted prompt text feature;

[0111] Specifically, the loss function includes:

[0112]

[0113] in, is the distance difference between the prompt text feature and all the reference text features, represents the similarity calculation between the prompt text feature and the i-th reference text feature, is a reward component, representing the reward value of the prompt text feature and the i-th reference text feature.

[0114] Specifically, if Figure 5 As shown, the distance between the prompt text feature and each reference text feature is adjusted according to the reward R and the loss function, which can be subdivided into the following cases:

[0115] The reward The amount of each reward Includes positive and negative values;

[0116] If the reward amount If it is a positive value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. direction adjustment, reducing the difference between the prompt text feature and the corresponding i-th reference text feature distance;

[0117] If the reward amount If it is a negative value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. Adjust in the opposite direction to increase the difference between the prompt text feature and the corresponding i-th reference text feature distance.

[0118] In this embodiment, the reward R components There are positive and negative, and the sum is zero, the component A positive value encourages prompt words in the prompt text feature The parameters are optimized in the direction of the corresponding reference text features, and negative values ​​encourage the parameters to move away from the corresponding direction of the corresponding reference text features. Intuitively, using the reward R and the loss function can guide the prompt words in the prompt text features. According to the reward feedback, the distance between itself and each reference text feature is adjusted to achieve a more balanced generation.

[0119] S153: input the adjusted prompt text feature into the image generation model again to generate a new image result, and continue to evaluate the new image result until the training of the prompt text feature is completed to obtain the trained prompt text feature.

[0120] For example, according to the above method, the prompt word in the prompt text feature Training, finally It will contain content that controls the gender attribute of the generated image results, and because it is a continuous optimization based on gradients, the distribution of the male-female ratio in the generated image results will change with the training and gradually change in the direction of the supervisory signal, eventually achieving a certain gender control result. We expect the training to stop at a certain moment in the middle, which is limited to the moment when the gender attribute distribution is a binary uniform distribution. It can be regarded as a hint word containing the semantics of balance. Therefore, we consider introducing the reinforcement learning method, where each reference text feature The set of components can be viewed as an action space, which refers to The optimization direction at a specific moment.

[0121] S160: Input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0122] The trained prompt text features can guide the image generation model to generate image results with a balanced proportion of corresponding debiased categories, thereby achieving a debiasing effect.

[0123] For example, this application also combines other debiasing methods to complete a comparative experiment:

[0124] The compared methods include: directly generating images using unprocessed Stable Diffusion, UCE (Gandikota R et al., 2024), ID (Li H et al., 2024), and the method of the present invention;

[0125] As shown in Table 1, the performance of the four methods on the Winobias dataset is compared. This dataset contains 36 occupational names. These occupational names, combined with certain text templates, can be used to prompt the image generation model to generate corresponding character images. By statistically analyzing the attributes of the generated character images, the degree of bias of the image generation model regarding these attributes can be evaluated. In addition, the experiment also examined the debiasing performance of the method when dealing with different text templates, such as adding the modifier "successful" to the text to form a sentence similar to "A headshot of a successful <profession>” text, such modifications usually aggravate the bias of the generated images, and the corresponding experimental results are recorded as "Gender+" and "Race+" in the table. The indicator used to evaluate the debiasing effect of different methods is the offset rate Δ, which is calculated as follows:

[0126]

[0127] in Represents the number of values ​​for the debiased category, such as when evaluating a binary gender attribute, , Represents the total number of generated images, Represents the number of images with the highest number of categories across all debiased categories. The lower the deviation rate, the less biased the image generation model is, and the better the debiasing effect of the method is. The best results are highlighted in bold in the table.

[0128] Table 1

[0129]

[0130] It can be seen that in different input texts, or in different debiasing categories, the offset rate of the debiasing method of the present application is generally lower than that of other methods, and the debiasing method of the present application is better.

[0131] like Figure 6 As shown, the embodiment of the present application also provides an image generation model debiasing system based on reinforcement prompt word learning. Optionally, the system may include:

[0132] Obtain target distribution module 211, obtain prompt text feature module 212, obtain reference text module 213, generate image module 214, training module 215, generate debiased image module 216, wherein:

[0133] Obtaining target distribution module 211, used to define debiasing categories and obtain target distribution of debiasing categories according to the debiasing categories;

[0134] In this embodiment, the target distribution acquisition module 211 can be used to perform Figure 1 As shown in step S110 , for a detailed description of the target distribution obtaining module 211 , reference may be made to the description of step S110 .

[0135] A prompt text feature acquisition module 212 is configured to acquire the original text, set prompt words based on the original text and the debiased category, and acquire prompt text features based on the original text and the prompt words;

[0136] In this embodiment, the prompt text feature acquisition module 212 can be used to perform Figure 1 As shown in step S120 , for a detailed description of the prompt text feature obtaining module 212 , reference may be made to the description of step S120 .

[0137] A reference text acquisition module 213 is configured to acquire a reference text set based on the original text and the debiased category, and acquire a plurality of reference text features based on the reference text set;

[0138] In this embodiment, the acquisition prompt acquisition reference text module 213 can be used to perform Figure 1 As shown in step S130 , for a detailed description of the reference text obtaining module 213 , reference may be made to the description of step S130 .

[0139] An image generation module 214 is configured to input the prompt text features into a pre-trained image generation model to generate an image result;

[0140] In this embodiment, the image generation module 214 can be used to perform Figure 1 As shown in step S140 , for a detailed description of the image generating module 214 , reference may be made to the description of step S140 .

[0141] A training module 215 is configured to evaluate the image result, obtain an actual distribution of debiased categories in the image result, and train the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed on a pre-trained evaluation model;

[0142] In this embodiment, the training module 215 can be used to perform Figure 1 As shown in step S150, for the detailed description of the training module 215, please refer to the description of step S150.

[0143] The module 216 for generating a debiased image is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0144] In this embodiment, the depolarized image generation module 216 can be used to perform Figure 1 As shown in step S160 , for a detailed description of the module 216 for generating the depolarized image, reference may be made to the description of step S160 .

[0145] The present application also provides an electronic device, the structure of which is as follows: Figure 7 As shown, the electronic device includes a memory 311, a processor 312, a communication module 313 and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313 and the input / output interface 314 can be connected and communicated through a bus 315.

[0146] The memory 311 is used to store one or more computer programs and transmit the code of the computer program to the processor 312; when the one or more computer programs are executed by the processor 311, an image generation model debiasing method based on reinforcement prompt word learning in an embodiment of the present application is implemented.

[0147] Optionally, the electronic device can be connected to a network via the communication module 313 to communicate with other devices, such as a terminal or a server, via the network to achieve data interaction. The electronic device can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers, or other types of computers. The electronic device can also be various forms of mobile terminals, such as smartphones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0148] Optionally, the electronic device can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 314. The electronic device itself can have a display device, and can also be connected to other display devices through the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 314, so that data in the electronic device can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device, or it can be an external device connected to the electronic device when needed.

[0149] Optionally, the memory 311 can be a volatile memory and / or a non-volatile memory, the volatile memory can be a random access memory, etc., and the non-volatile memory can be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0150] Optionally, the computer program stored in the processor 311 may be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to implement the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device.

[0151] Optionally, the processor 312 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various specialized artificial intelligence computing chips, various processors for running machine learning model algorithms, and may also be any appropriate controller, microcontroller, processor, etc. The processor 312 executes the various methods and processes of this embodiment, such as, for example, a method for debiasing a graph model based on enhanced cue word learning in an embodiment of the present application.

[0152] Optionally, the bus 315 may include a path for transmitting information. The bus 315 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Depending on their functions, the bus 315 may be categorized as an address bus, a data bus, a control bus, or the like.

[0153] In an optional implementation, the present embodiment further provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-described method embodiment. Part or all of the computer program can be loaded and / or installed into the memory 311 of the electronic device. When the computer program is executed by the processor 312, one or more steps of the image generation model debiasing method based on reinforcement cue word learning according to the embodiment of the present application can be performed.

[0154] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0155] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the scope of protection of the claims of the present application.< / profession>

Claims

1. A debiasing method for an image generation model based on reinforcement cue word learning, characterized in that: The method comprises: defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category; Obtaining an original text, setting a prompt word according to the original text and the debiased category, and obtaining a prompt text feature according to the original text and the prompt word; Acquire a reference text set according to the original text and the debiased category, and acquire a plurality of reference text features according to the reference text set; Inputting the prompt text features into a pre-trained image generation model to generate an image result; Evaluating the image result, obtaining an actual distribution of debiased categories in the image result, and training the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the image result evaluation and the prompt text feature training are performed on a pre-trained evaluation model; The trained prompt text features are input into the image generation model again to obtain a debiased image result.

2. The image generation model debiasing method based on enhanced prompt word learning according to claim 1, characterized in that: The acquiring of prompt text features according to the original text and the prompt word includes: Vectorize the original text to obtain the original text vector ; Vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector; The zero-initialized hint word vector Add to the original text vector Get the prompt text vector ; The prompt text vector Perform text encoding to obtain prompt text features .

3. The image generation model debiasing method based on enhanced prompt word learning according to claim 1, characterized in that: The acquiring of a plurality of reference text features according to the reference text set includes: Set the reference text set to ,in is the i-th reference text, and n represents the total value of the reference texts; Each of the reference texts in the reference text set After vectorization, perform text encoding to obtain the corresponding reference text features .

4. The image generation model debiasing method based on enhanced prompt word learning according to claim 1, characterized in that: The evaluating the image result to obtain the actual distribution of the debiased categories in the image result includes: Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result; According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

5. The image generation model debiasing method based on enhanced cue word learning according to any one of claims 1 to 4, characterized in that: The step of training the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature includes: Comparing the actual distribution with the target distribution; if the actual distribution conforms to the target distribution, taking the image result as a debiased image result, and completing the training of the prompt word feature; if the actual distribution does not conform to the target distribution, obtaining the deviation between the actual distribution and the target distribution, and calculating a reward R based on the deviation, wherein the deviation and the reward R are obtained through the evaluation model; Adjust the distance between the prompt text feature and each reference text feature according to the reward R and the loss function to obtain the adjusted prompt text feature; The adjusted prompt text feature is input into the image generation model again to generate a new image result, and the new image result is continuously evaluated until the training of the prompt word feature is completed to obtain the trained prompt text feature.

6. The image generation model debiasing method based on enhanced prompt word learning according to claim 5, characterized in that: The loss function specifically includes: in, is the distance difference between the prompt text feature and all the reference text features, represents the similarity calculation between the prompt text feature and the i-th reference text feature, is the reward component, representing the reward value of the prompt text feature and the i-th reference text feature, Indicates the Reference text features.

7. The image generation model debiasing method based on enhanced prompt word learning according to claim 5, characterized in that: The adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function includes: The reward The amount of each reward Includes positive and negative values; If the reward amount If it is a positive value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. direction adjustment, reducing the difference between the prompt text feature and the corresponding i-th reference text feature distance; If the reward amount If it is a negative value, the prompt text feature is converted to the corresponding i-th reference text feature according to the loss function. Adjust in the opposite direction to increase the difference between the prompt text feature and the corresponding i-th reference text feature distance.

8. A debiasing system for image generation models based on reinforcement cue word learning, characterized in that: include: A target distribution acquisition module is used to define a debiasing category and acquire a target distribution of the debiasing category according to the debiasing category; A prompt text feature acquisition module is used to acquire original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words; A reference text acquisition module, configured to acquire a reference text set based on the original text and the debiased category, and acquire a plurality of reference text features based on the reference text set; An image generation module is used to input the prompt text features into a pre-trained image generation model to generate an image result; a training module, configured to evaluate the image result, obtain an actual distribution of debiased categories in the image result, and train the prompt text feature based on the actual distribution, the target distribution, and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed on a pre-trained evaluation model; A debiased image module is generated, which is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

9. An electronic device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the image generation model debiasing method based on reinforcement prompt word learning as described in any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for debiasing an image generation model based on reinforcement prompt word learning as described in any one of claims 1 to 7 is implemented.

Images