Picture generation model depolarization method and system based on enhanced cue word learning

Abstract

The invention relates to the field of machine learning, in particular to an image generation model depolarization method and system based on enhanced cue word learning. The method comprises the following steps: defining depolarization categories, and obtaining target distribution of the depolarization categories; obtaining an original text, setting prompt words, and obtaining prompt text features according to the original text and the prompt words; obtaining a reference text set, and obtaining reference text features according to the reference text set; inputting the prompt text features into an image generation model to generate an image result; evaluating the image result, obtaining the actual distribution of the depolarization categories in the image result, and training the prompt text features according to the actual distribution, the target distribution and the reference text features to obtain trained prompt text features; and inputting the image into the image generation model again to obtain a depolarized image result. According to the method, efficient and controllable depolarization of the image generation model can be realized, the calculation overhead is reduced, and the adaptability of the depolarization method is improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and more specifically, to a method and system for debiasing an image generation model based on reinforcement cue word learning. Background Art

[0002] In the prior art, image generation models, especially text-to-image (T2I) models, can generate corresponding images based on the text description provided by the user. However, during the model training process, due to the imbalance of the data set in features, the model may inherit or generate bias in the data set or algorithm, resulting in imbalance in certain features (such as gender, race) of the generated images. Unbalanced image generation results will have a negative impact on social diversity and fairness, so the study of model bias elimination methods is very important. At present, the methods for eliminating model bias can be roughly divided into data-driven methods and model-driven methods: data-driven methods minimize bias in training data by modifying or rebalancing training data, which usually involves retraining the model or fine-tuning parameters, but data-driven methods generally have the problem of high computational complexity; model-driven methods control bias output in the model generation process by adjusting the structure of the model or introducing a debiasing module, but model-driven methods are sensitive to specific model architectures, so it is necessary to make necessary improvements to the model debiasing method. Summary of the invention

[0003] The present invention aims to overcome at least one defect (shortcoming) of the above-mentioned prior art and provide an image generation model debiasing method and system based on reinforcement cue word learning, so as to achieve efficient and controllable debiasing of the image generation model, reduce computational overhead, and improve the adaptability of the debiasing method.

[0004] According to a first aspect of the present application, a method for debiasing an image generation model based on reinforcement cue word learning is provided, the method comprising:

[0005] defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category;

[0006] Acquire the original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words;

[0007] Acquire a reference text set according to the original text and the debiased category, and acquire a plurality of reference text features according to the reference text set;

[0008] Inputting the prompt text features into a pre-trained image generation model to generate an image result;

[0009] The image result is evaluated to obtain the actual distribution of the debiased categories in the image result, and the prompt text feature is trained according to the actual distribution, the target distribution and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model;

[0010] The trained prompt text features are input into the image generation model again to obtain a debiased image result.

[0011] It is understandable that the image results generated by the prompt text features are statistically distributed, and the actual distribution is compared with the target distribution of the debiased category, and the prompt text features are trained according to the deviation values ​​of the actual distribution and the target distribution, so as to obtain the debiased image results; this method does not need to adjust the parameters inside the image generation model, nor does it modify the internal components in the image generation model. It only adds a prompt word to the input original text and trains it. The sample data required for training is only simple text data and a small amount of image data, which can save a lot of computing resources for adjusting the parameters and structure of the image generation model; furthermore, in the step of training the prompt text features, the concept of vector is introduced, which can guide the prompt text features to be trained slowly and directionally in the direction corresponding to the target distribution, improve the accuracy of training, and prevent the problem of overfitting caused by excessive training; at the same time, there is no need to adjust the parameters and structure of the image generation model, and it can be applicable to image generation models of different versions and types, thereby improving the adaptability of the debiasing method.

[0012] Optionally, acquiring prompt text features according to the original text and the prompt word includes:

[0013] The original text is vectorized to obtain the original text vector T = {v1, v2, ..., v n};

[0014] Vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector;

[0015] The zero-initialized prompt word vector e fair Add to the original text vector T = {v1, v2, ..., v n}Get the prompt text vector T'={v1,v2,…v n-1 ,e fair ,v n};

[0016] The prompt text vector T′={v1,v2,…v n-1 ,e fair ,v n}Perform text encoding to obtain the prompt text feature E(T′).

[0017] It is understandable that the prompt word is related to the debiasing category and is used to place words related to the debiasing category in the original text, so that the information of the debiasing type can be reflected in the original text; it is understandable that when the image generation model is trained, the input samples are statistically unbalanced in the debiasing category, so that the trained image generation model inevitably has bias in the debiasing category; if the user's demand is to use a balanced image generated by text, the image generation model must be debiased; here, after the original text and the prompt word are vectorized, the prompt word vector is added to the original text vector, so that the original text has a control word for the debiasing category, and the prompt word can be adjusted to control the proportion of the image result, and the vectorized data of the text is used, which can save more computing overhead and improve the efficiency of training.

[0018] Optionally, the acquiring a plurality of reference text features according to the reference text set includes:

[0019] Set the reference text set S = set(s1,s2,…,s i ,…,s n ), where s i is the i-th reference text, and n represents the total value of the reference text;

[0020] Each of the reference texts in the reference text set i After vectorization, the text is encoded to obtain the corresponding reference text features E(s i ).

[0021] It can be understood that the reference text is a multiple text value based on the original text and the debiasing category, representing a single direction of each debiasing category; the reference text is introduced here to provide a clear optional direction for training the prompt text feature, so that the prompt text feature training refers to the direction of each debiasing category, and achieves a balanced training result for each debiasing category; the reference text is also vectorized here, and can use a unified data format with the prompt text, and the vector has directionality, which can make the prompt text training more accurate.

[0022] Optionally, evaluating the image result to obtain an actual distribution of debiased categories in the image result includes:

[0023] Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result;

[0024] According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

[0025] It is understandable that the prompt text that is first input into the image generation model is equivalent to the original text of the user, because the added prompt words are zero-initialized; and the image result obtained by inputting the original text into the image generation model is the original bias of the image generation model. It is necessary to record the bias on the label and get the actual distribution of the debiased category before debiasing can be performed. Here, the image result is evaluated using a pre-trained evaluation model, which can quickly get the debiasing degree of the image generation model and improve the efficiency of the entire debiasing method.

[0026] Optionally, the step of training the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature includes:

[0027] The actual distribution is compared with the target distribution. If the actual distribution conforms to the target distribution, the image result is used as the debiased image result, and the training of the prompt word feature is completed; if the actual distribution does not conform to the target distribution, the deviation between the actual distribution and the target distribution is obtained, and the reward R is calculated according to the deviation, wherein the deviation and the reward R are obtained through the evaluation model;

[0028] Adjust the distance between the prompt text feature and each reference text feature according to the reward R and the loss function to obtain the adjusted prompt text feature;

[0029] The adjusted prompt text feature is input into the image generation model again to generate a new image result, and the new image result is continuously evaluated until the training of the prompt word feature is completed to obtain the trained prompt text feature.

[0030] It can be understood that the training method of the prompt text feature introduced above has the goal of making the actual distribution of the acquired image results conform to the target distribution of the debiased category, wherein the distance between the prompt text feature and each reference text feature is adjusted according to the deviation between the actual distribution and the target distribution, so that the generated image result is in proportion to the target distribution, thereby obtaining a debiased effect; this method can gradually and slowly reduce the bias of the image generation model, and adjust the bias in combination with the reward R and the loss function to improve the efficiency of training.

[0031] Optionally, the loss function specifically includes:

[0032]

[0033] Among them, L rl is the distance difference between the prompt text feature and all the reference text features, sim(·,·) represents the similarity calculation between the prompt text feature and the i-th reference text feature, R i is a reward component, representing the reward value of the prompt text feature and the i-th reference text feature.

[0034] It can be understood that calculating the similarity between the prompt text feature and the reference text feature is equivalent to obtaining the similarity between the prompt text feature and the i-th reference text feature, and calculating the distance difference between the prompt text feature and the i-th reference text feature based on the reward R obtained by the distribution deviation, and subsequently training the prompt text feature based on this difference; this method can train the prompt text feature through the vectorized direction and specific data of the distribution deviation, so that the training process has a clear direction and measurement, avoids the overfitting problem caused by excessive training or the problem of poor debiasing effect, and improves the accuracy of the debiasing effect.

[0035] Optionally, adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function includes:

[0036] The reward components R of the reward R i Includes positive and negative values;

[0037] If the reward component R i is a positive value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) direction adjustment, reducing the distance between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance;

[0038] If the reward component R i is a negative value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) in the opposite direction, increasing the difference between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance.

[0039] It is understandable that the reward R can reflect the deviation between the real-time distribution and the target distribution, as well as affect the loss function through positive and negative data, thereby affecting the direction of the prompt text feature adjustment; this is conducive to the precise movement of the prompt text feature in the direction of training, and achieves the effect of slowly fine-tuning the prompt text feature.

[0040] According to a second aspect of the present application, there is provided an image generation model debiasing system based on reinforcement cue word learning, comprising:

[0041] A target distribution acquisition module is used to define a debiasing category and acquire a target distribution of the debiasing category according to the debiasing category;

[0042] A prompt text feature acquisition module is used to acquire the original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words;

[0043] A reference text acquisition module, used for acquiring a reference text set according to the original text and the debiased category, and acquiring a plurality of reference text features according to the reference text set;

[0044] An image generation module is used to input the prompt text features into a pre-trained image generation model to generate an image result;

[0045] A training module, used for evaluating the image result, obtaining the actual distribution of the debiased categories in the image result, and iteratively training the prompt text feature according to the actual distribution, the target distribution and the plurality of reference text features to obtain the trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model;

[0046] A debiased image module is generated, which is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0047] According to a third aspect of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a method for debiasing an image generation model based on reinforcement cue word learning as described in the first aspect.

[0048] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed, the image generation model debiasing method based on reinforcement prompt word learning as described in the first aspect above is implemented.

[0049] Based on any one of the above aspects, the embodiment of the present application provides a method for statistically analyzing the actual distribution of the image results generated by the prompt text features, comparing the actual distribution with the target distribution of the debiased category, and training the prompt text features according to the deviation values ​​of the actual distribution and the target distribution, thereby obtaining a debiased image result; this method does not require adjustment of the parameters inside the image generation model, nor does it modify the internal components in the image generation model. It only adds a prompt word to the input original text and trains it, and in the step of training the prompt text features, introduces the concept of vector, which can guide the prompt text features to slowly and directionally train in the direction corresponding to the target distribution, thereby improving the accuracy of the training, achieving efficient and controllable debiasing of the image generation model, and reducing computational overhead, thereby improving the adaptability of the debiasing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flow chart of a method for debiasing an image generation model based on reinforcement cue word learning provided in this embodiment.

[0052] Figure 2 This is a flow chart of obtaining prompt text features according to this embodiment.

[0053] Figure 3 This is a flow chart of obtaining reference text features according to this embodiment.

[0054] Figure 4 This is a flow chart of training prompt text features in this embodiment.

[0055] Figure 5 This is a diagram of adjusting the distance between the prompt text feature and each reference text feature in this embodiment.

[0056] Figure 6 This is a structural diagram of an image generation model debiasing system based on reinforcement cue word learning in this embodiment.

[0057] Figure 7 This is a device structure diagram of the electronic device provided in this embodiment. DETAILED DESCRIPTION

[0058] The drawings of this application are only used for illustrative purposes and should not be construed as limiting the present application. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0059] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0060] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] In the prior art, sample data generally have the problem of imbalance in a certain category, so the image generation model trained with the sample data will have bias towards a certain category. For example, one of the focuses of bias research in the text-to-image (T2I) model in the image generation model is on demographic-related features, such as gender, race, etc. During the training process, the T2I model may inherit or generate bias against the above features from the data or algorithm, resulting in the T2I model tending to generate images of a specific race or gender when generating images, while ignoring other groups. For example, when an image of a doctor is generated through the T2I model, but the gender and racial characteristics of the doctor are not specified in the text, the generated image may be dominated by white men, while other groups only account for a small part. Such image generation results will have a negative impact on social diversity and fairness, so it is very important to study methods to eliminate model bias.

[0062] At present, the methods for eliminating bias in T2I models can be divided into data-driven methods and model-driven methods. Data-driven methods minimize the imbalance in training data by modifying or rebalancing the training data, which usually involves retraining or parameter fine-tuning of the T2I model; model-driven methods control the biased output in the T2I model generation process by adjusting the structure of the T2I model or introducing a debiasing module. Although these methods have alleviated the bias problem to a certain extent, data-driven methods face the problem of high computational complexity, while model-driven methods are sensitive to specific model architectures. Therefore, it is necessary to improve the debiasing method of the image generation model to achieve an efficient and controllable debiasing effect, so as to reduce the computational overhead and adapt to different image generation models.

[0063] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0064] For example, a flowchart of a method for debiasing an image generation model based on enhanced cue word learning is provided in an embodiment of the present application. Figure 1 As shown, the method can be subdivided into the following steps:

[0065] S110, defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category;

[0066] In this embodiment, the image generation model used is an image generation model that has been trained in advance and can be directly put into use; preferably, in this embodiment, the text generation image (Text-to-image, T2I) model of the image generation model is studied. It is understandable that when training the image generation model, the training data used has certain deviations and limitations, such as:

[0067] Data imbalance: If certain groups are over-represented or under-represented in the training data, the image generation model will be affected by these biases during the learning process, resulting in biased output results. For example, if there are far more male images than female images in the training data set, the generated images may be more likely to show male characteristics;

[0068] Single data source: The training data used is from the Internet, which often reflects a specific social and cultural background and may contain various biases and stereotypes.

[0069] Or when building an image generation model, there are inherent flaws in the algorithm design built into the image generation model, such as:

[0070] Lack of diversity: The importance of diversity may not be fully considered when designing image generation models, resulting in a lack of diversity and inclusiveness in the generated images;

[0071] Optimization goal deviation: The optimization goal of the image generation model may deviate from human values. For example, in some cases, the image generation model may pay more attention to the beauty of the image rather than diversity or fairness;

[0072] Or when training an image generation model, the image generation model will be affected by social and cultural factors, such as:

[0073] Gender discrimination: When image generation models generate character images based on text, they often follow traditional gender stereotypes, portraying women as weak and sexy, and men as strong and powerful. This gender bias not only limits people's diverse understanding of gender roles, but may also have a negative impact on women's career development and social status;

[0074] Ageism: When image generation models generate images of people based on text, they tend to present young and healthy images, while the images of the elderly are relatively rare and inaccurate. This bias may further deepen the stereotype of the elderly in society, leading to the underestimation of the status and value of the elderly in society;

[0075] Class discrimination: When generating images of urban life based on text, image generation models may show more scenes of the lives of the wealthy class and pay less attention to the living conditions of the poor class. This bias may exacerbate social inequality and make people have a wrong perception of the gap between social classes.

[0076] Therefore, based on the bias defects generated by the construction and training of the above-mentioned image generation model, it is necessary to eliminate these biases of the image generation model. It is understandable that in the debiasing work, it is necessary to define a debiasing category, which is the type of debiasing that the user hopes to obtain in the image results generated by the image generation model with a balanced ratio. Exemplarily, if the user needs to generate an image result with a balanced ratio of male and female pictures, [Gender] is selected as the debiasing category, and the debiasing category [Gender] includes [Male, Female] values;

[0077] It is understandable that the debiasing category is not limited to a binary variable, and a debiasing category including a polytomous variable can also be selected. For example, [race] is selected as the debiasing category, including values ​​such as yellow race, black race, and white race;

[0078] According to the selected debiasing category and the value of the debiasing category, the target distribution formed by the ratio of each value in the image result desired by the user is obtained; for example, if gender is the debiasing category, the user hopes to obtain an image with a 1:1 ratio of male to female, so if the debiasing category conforms to the binary uniform distribution, the binary uniform distribution is used as the target distribution of the debiasing category;

[0079] It is understandable that the target distribution of the debiased category can be set to other statistical distributions, which are not listed here one by one.

[0080] S120, obtaining an original text, and setting a prompt word according to the original text and the debiasing category, and obtaining a prompt text feature according to the original text and the prompt word;

[0081] In this embodiment, the user inputs a sentence of original text to the image generation model, which is used as the initial tone for generating the image result; illustratively, "a group of photos of doctors" is input as the original text; in the above steps, preferably, [gender] is selected as the debiasing category, including the value [male, female], and the target distribution is a binary uniform distribution. It is understandable that in the general image generation model training, the sample data may have a higher proportion of male doctors than female doctors, so the image generation model will be biased, and it is necessary to debias the image generation model so that the generated picture results meet the target distribution;

[0082] According to the value of the debiasing category and the attributes of the original text, a prompt word is set to be inserted into the appropriate semantic position of the original text, so as to play the role of prompting the debiasing category in the original text; for example, it can be simply understood that the prompt text with the prompt word added is "a group of pictures of [gender] doctors".

[0083] like Figure 2 As shown, the acquisition of prompt text features according to the original text and the prompt word can be subdivided into the following steps:

[0084] S121, vectorize the original text to obtain an original text vector T = {v1, v2, ..., v n};

[0085] In this embodiment, the original text, such as "a group of doctor's photos", is vectorized to obtain the original text vector, which enables the image generation model to understand and process the text content, helps to realize the recognition of the text content, and brings format convenience to the subsequent deviation calculation and training work;

[0086] S122, vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector;

[0087] In this application, the cue words are introduced to debias the image generation model. The existing method of adding fairness description to text is used for reference. The difference is that the method of this application attempts to dynamically optimize this fairness description in the continuous vector space of the reference text, rather than directly giving it in the form of natural language. Using the cue word learning method, an additional learnable cue word vector is added to the vectorized original text, denoted as e fair , and optimize e during training fair , so that the image generation model can generate fairer results. Unlike other methods, the present invention uses the reinforcement learning paradigm to optimize the parameters of the prompt word. This design has two advantages:

[0088] 1) There is no need to prepare and collect training data in advance. The data used comes from the interaction between the model and the environment, that is, the data comes from the images generated by the image generation model itself;

[0089] 2) The debiasing method has strong generalization ability. It only requires that the image generation model uses a pre-trained text encoder to encode the text separately. There are no special requirements for other architectures of the image generation model. The use of a text encoder to encode text is widely used in most of the current mainstream text-generated image models. Therefore, the debiasing method of this application can be applied to most text-generated image models;

[0090] In this embodiment, fair Zero initialization is performed to avoid e fair The initial introduction of the hint word vector into the original text has a semantic impact, and at the same time, it can also ensure that the image result generated by the original text with the hint word vector added for the first time is the original bias of the image generation model, which can be used as a comparison benchmark for training effects.

[0091] S123, initialize the prompt word vector e after zero fair Add to the original text vector T = {v1, v2, ..., v n}Get the prompt text vector T′={v1,v2,…v n-1 ,e fair ,v n};

[0092] In this embodiment, the prompt word vector is added to the original text vector to obtain the prompt text vector, and then the prompt text vector is input into the text encoder to obtain the prompt text feature, so that the image generation model can better process the text, capture the features of the text, and generate a picture result that conforms to the text.

[0093] S124, the prompt text vector T′={v1,v2,…v n-1 ,e fair ,v n}Perform text encoding to obtain the prompt text feature E(T′).

[0094] In this embodiment, the prompt text vector needs to be text encoded. Preferably, a text encoder can be used to encode the prompt text vector to obtain prompt text features, which can better compare the prompt text features and reference text features in the same format and better calculate the deviation.

[0095] S130, obtaining a reference text set according to the original text and the debiased category, and obtaining a plurality of reference text features according to the reference text set;

[0096] In this embodiment, in addition to the need to introduce prompt words, it is also necessary to provide the direction and environment for prompt word training. Here, the reference text is obtained based on the original text and the debiased category. Exemplarily, in the data mentioned above, reference texts of "a group of pictures of male doctors" and "a group of pictures of female doctors" can be obtained. These are two values ​​of the debiased category integrated into the original text, and are also the two directions of prompt word training. It can be understood that, unlike the original text, the reference text directly indicates the gender attribute of the generated image result content in the description. When the original text of the picture generation model does not contain gender attributes, the gender ratio of the generated picture results is often uncertain and unbalanced, but when the gender attribute is specified in the input text, the generated results are generally certain. For example, for the reference text "a group of pictures of male doctors", the generated image is 100% male, so it is inconsistent with the user's desire to obtain a balanced proportion of picture results. Therefore, it is necessary to use different reference texts as directions to train the prompt text features, so as to obtain a balanced proportion of picture results;

[0097] It is understandable that the reference text is obtained according to the debiasing category value, so multiple reference texts can be obtained to form a reference text set, which represents multiple optimization directions for prompting text feature training.

[0098] Specifically, Figure 3 As shown, the step of obtaining a plurality of reference text features according to the reference text set can be subdivided into the following steps:

[0099] S131, set the reference text set to S=set(s1,s2,…,s i ,…,s n ), where s i is the i-th reference text, and n represents the total value of the reference text;

[0100] S132: Each of the reference texts in the reference text set i After vectorization, the text is encoded to obtain the corresponding reference text features E(s i ).

[0101] In this embodiment, several reference texts also need to be vectorized and then text encoded. Preferably, a text encoder can be used to encode the reference texts to obtain several reference text features, so that the prompt text features and the reference text features can be better compared in the same format, and the deviation can be better calculated.

[0102] S140, inputting the prompt text features into a pre-trained image generation model to generate an image result;

[0103] Understandably, the methods for debiasing image generation models mainly focus on two aspects. One type of method removes bias by fine-tuning the parameters of the image generation model. However, these methods usually involve huge computational overhead, especially when the entire image generation model needs to be retrained. Not only is the computational cost high, but the hardware requirements are also very demanding. In addition, such methods often require the collection of specific datasets, such as collecting corresponding datasets for different types of bias. Another type of method mainly optimizes some components of the image generation model, such as debiasing the encoding part of the text. This method only considers one link in the text-to-image generation process, while ignoring other links of image generation, which may also introduce or amplify bias. In addition, many current debiasing methods are designed for specific image generation model architectures, which often makes them lose applicability in the process of continuous development and iteration of model architectures. With the continuous optimization and updating of image generation models, the structure and mechanism of the new generation of models may not be consistent with the assumptions of existing debiasing methods, making it difficult for these methods to adapt to new architectures and scenarios.

[0104] In this embodiment, the generated image results are directly used as the sample set for training prompt text features, so that the debiasing method does not need to collect and process a large number of images, saving a lot of computing overhead. At the same time, there are no special requirements for the internal structure of the image generation model, which can improve the applicability of the debiasing method.

[0105] S150, evaluating the image result, obtaining the actual distribution of the debiased categories in the image result, and training the prompt text feature according to the actual distribution, the target distribution and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model;

[0106] In this embodiment, the generated image result needs to be processed to obtain its actual distribution; it is understandable that the actual distribution obtained for the first time can reflect the bias that originally existed in the image generation model; the subsequent actual distribution reflects that the bias of the image generation model has been debiased after each training;

[0107] Specifically, the evaluating the image result to obtain the actual distribution of the debiased categories in the image result includes:

[0108] Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result;

[0109] According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

[0110] In this embodiment, the debiasing category of the image result is judged. It can be understood that the debiasing category of each picture in a set of generated pictures can be identified, and finally the debiasing category ratio of the image result is obtained, thereby obtaining the actual distribution of the image result, and providing necessary training data for subsequent prompt text feature training.

[0111] Specifically, Figure 4 As shown, the training of the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature can be subdivided into the following steps:

[0112] S151, comparing the actual distribution with the target distribution. If the actual distribution conforms to the target distribution, the image result is used as the debiased image result, and the training of the prompt word feature is completed; if the actual distribution does not conform to the target distribution, the deviation between the actual distribution and the target distribution is obtained, and a reward R is calculated according to the deviation, wherein the deviation and the reward R are obtained through the evaluation model;

[0113] In this embodiment, in order to guide the optimization of the prompt text vector, we evaluate the actual distribution d obs and the target distribution d target The reward R is calculated by the deviation between d and d. For example, taking the binary gender attribute as an example, let d obs =(d obs,1 ,d obs,2 ), where d obs,1 +d obs,2 =1, the target distribution is a binary uniform distribution, which comes from our optimization goal of fairness, so d target =(d target,1 ,d target,2 )=(0.5,0.5). The reward R is:

[0114] R=d target -d obs =(d target,1 -dobs,1 ,d target,2 -d obs,2 )

[0115] S152, adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function, and obtaining an adjusted prompt text feature;

[0116] Specifically, the loss function includes:

[0117]

[0118] Among them, L rl is the distance difference between the prompt text feature and all the reference text features, sim(·,·) represents the similarity calculation between the prompt text feature and the i-th reference text feature, R i is a reward component, representing the reward value of the prompt text feature and the i-th reference text feature.

[0119] Specifically, Figure 5 As shown, the distance between the prompt text feature and each reference text feature is adjusted according to the reward R and the loss function, which can be subdivided into the following cases:

[0120] The reward components R of the reward R i Includes positive and negative values;

[0121] If the reward component R i is a positive value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) direction adjustment, reducing the distance between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance;

[0122] If the reward component R i is a negative value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) in the opposite direction, increasing the difference between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance.

[0123] In this embodiment, the reward R components R i There are positive and negative components, and the sum is zero. i A positive value encourages the prompt word e in the prompt text feature fair The parameter is optimized in the direction of the corresponding reference text feature, and a negative value encourages the parameter to move away from the corresponding direction of the reference text feature. Intuitively, the use of reward R and loss function can guide the prompt word e in the prompt text feature.fair According to the reward feedback, the distance between itself and each reference text feature is adjusted to achieve fairer generation.

[0124] S153, inputting the adjusted prompt text feature into the image generation model again to generate a new image result, and continuing to evaluate the new image result until the training of the prompt text feature is completed to obtain the trained prompt text feature.

[0125] For example, according to the above method, the prompt word e in the prompt text feature is ffair Training, finally fair It will contain content that controls the gender attribute of the generated image results, and because it is a continuous optimization based on gradients, the distribution of the male-female ratio in the generated image results will change with the training and gradually change in the direction of the supervisory signal, eventually achieving a certain gender control result. We expect the training to stop at a certain moment in the middle, which is limited to the moment when the gender attribute distribution is a binary uniform distribution. At this time, e fair can be regarded as a hint word containing fairness semantics. Therefore, we consider introducing a reinforcement learning method. At this time, each reference text feature E(s I ) can be regarded as an action space, which refers to e fair The optimization direction at a specific moment.

[0126] S160: Input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0127] The trained prompt text features can guide the image generation model to generate image results with a balanced proportion of corresponding debiased categories, thereby achieving a debiasing effect.

[0128] For example, this application also combines other debiasing methods to complete a comparative experiment:

[0129] The comparison methods include: directly generating images using unprocessed Stable Diffusion, UCE (Gandikota R et al., 2024), ID (Li H et al., 2024), and the method of the present invention;

[0130] As shown in Table 1, the performance of the four methods on the Winobias dataset is compared. This dataset contains a total of 36 occupational names. Using these occupational names in combination with certain text templates, the image generation model can be prompted to generate corresponding character images. By statistically analyzing the attributes such as gender and race of the generated character images, the degree of bias of the image generation model regarding these attributes can be evaluated. In addition, the experiment also examined the debiasing performance of the method when dealing with different text templates, such as adding the modifier "successful" to the text to form a sentence similar to "A headshot of a successful <profession>", such modifications usually aggravate the bias of the generated images, and the corresponding experimental results are recorded as "Gender+" and "Race+" in the table. The indicator used to evaluate the debiasing effect of different methods is the offset rate Δ, which is calculated as follows:

[0131]

[0132] Where C represents the number of values ​​of the debiased category. For example, when evaluating the binary gender attribute, C = 2, and N represents the total number of generated images. c Represents the number of images in the most common category among all debiased categories. The lower the deviation rate, the lower the bias of the image generation model, the better the debiasing effect of the method, and the best results are bolded in the table.

[0133] Table 1

[0134]

[0135] It can be seen that in different input texts, or in different debiasing categories, the deviation rate of the debiasing method of the present application is generally lower than that of other methods, and the debiasing method of the present application is better.

[0136] like Figure 6 As shown, the embodiment of the present application also provides an image generation model debiasing system based on reinforcement prompt word learning. Optionally, the system may include:

[0137] Obtain target distribution module 211, obtain prompt text feature module 212, obtain reference text module 213, generate image module 214, train module 215, generate debiased image module 216, wherein:

[0138] The target distribution acquisition module 211 is used to define a debiasing category and acquire a target distribution of the debiasing category according to the debiasing category;

[0139] In this embodiment, the target distribution acquisition module 211 can be used to execute Figure 1 As shown in step S110 , for a detailed description of the target distribution acquisition module 211 , reference may be made to the description of step S110 .

[0140] A prompt text feature acquisition module 212 is used to acquire the original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words;

[0141] In this embodiment, the prompt text feature acquisition module 212 can be used to perform Figure 1 As shown in step S120 , for a detailed description of the prompt text feature obtaining module 212 , reference may be made to the description of step S120 .

[0142] A reference text acquisition module 213 is used to acquire a reference text set according to the original text and the debiased category, and acquire a plurality of reference text features according to the reference text set;

[0143] In this embodiment, the acquisition prompt acquisition reference text module 213 can be used to execute Figure 1 As shown in step S130 , for a detailed description of the reference text obtaining module 213 , reference may be made to the description of step S130 .

[0144] An image generation module 214 is used to input the prompt text features into a pre-trained image generation model to generate an image result;

[0145] In this embodiment, the image generation module 214 may be used to perform Figure 1 As shown in step S140 , for a detailed description of the image generating module 214 , reference may be made to the description of step S140 .

[0146] The training module 215 is used to evaluate the image result, obtain the actual distribution of the debiased categories in the image result, and train the prompt text feature according to the actual distribution, the target distribution and the plurality of reference text features to obtain the trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model;

[0147] In this embodiment, the training module 215 can be used to perform Figure 1 As shown in step S150, for the specific description of the training module 215, reference may be made to the description of step S150.

[0148] The module 216 for generating a debiased image is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

[0149] In this embodiment, the debiased image generation module 216 may be used to perform Figure 1 As shown in step S160, for a detailed description of the module 216 for generating the depolarized image, reference may be made to the description of step S160.

[0150] The present application also provides an electronic device, the structure of which is as follows: Figure 7 As shown, the electronic device includes a memory 311, a processor 312, a communication module 313 and an input / output interface 314, etc. Optionally, the memory 311, the processor 312, the communication module 313 and the input / output interface 314 can be connected and communicated through a bus 315.

[0151] The memory 311 is used to store one or more computer programs and transmit the code of the computer program to the processor 312; when the one or more computer programs are executed by the processor 311, an image generation model debiasing method based on reinforcement prompt word learning in an embodiment of the present application is implemented.

[0152] Optionally, the electronic device can be connected to a network via a communication module 313 to communicate with other devices, such as a terminal or a server, through the network to achieve data interaction. The electronic device can be various forms of digital computers, such as desktop computers, servers, workstations, mainframe computers or other types of computers. The electronic device can also be various forms of mobile terminals, such as smart phones, tablet computers, wearable devices (such as helmets, glasses, watches, etc.) and other similar mobile terminals.

[0153] Optionally, the electronic device can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 314. The electronic device itself can have a display device, and can also be connected to other display devices through the input / output interface 314. Optionally, a storage device, such as a hard disk, can also be connected through the input / output interface 314, so that data in the electronic device can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 311. It can be understood that the input / output interface 314 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 314 can be a component of the electronic device, or it can be an external device connected to the electronic device when needed.

[0154] Optionally, the memory 311 may be a volatile memory and / or a non-volatile memory, the volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory or a flash memory, etc.

[0155] Optionally, the computer program stored in the processor 311 may be divided into one or more modules, which are stored in the memory 311 and executed by the processor 312 to complete the method provided in the embodiment. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the electronic device.

[0156] Optionally, the processor 312 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 312 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various dedicated artificial intelligence computing chips, various processors running machine learning model algorithms, and any appropriate controller, microcontroller, processor, etc. The processor 312 executes the various methods and processes of this embodiment, illustratively, such as a method for debiasing a graph model based on enhanced prompt word learning in an embodiment of the present application.

[0157] Optionally, the bus 315 may include a path for transmitting information. The bus 315 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 315 may be divided into an address bus, a data bus, a control bus, etc.

[0158] In an optional implementation, the embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer is enabled to perform the method of the above method embodiment. Part or all of the computer program can be loaded and / or installed on the memory 311 of the electronic device. When the computer program is executed by the processor 312, one or more steps of an image generation model debiasing method based on reinforcement prompt word learning in the embodiment of the present application can be performed.

[0159] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or the like.

[0160] Obviously, the above embodiments of the present application are merely examples for clearly illustrating the technical solution of the present application, and are not intended to limit the specific implementation methods of the present application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present application shall be included in the protection scope of the claims of the present application.< / profession>

Claims

1. A method for debiasing an image generation model based on reinforcement cue word learning, characterized in that: The method comprises: defining a debiasing category, and obtaining a target distribution of the debiasing category according to the debiasing category; Acquire the original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words; Acquire a reference text set according to the original text and the debiased category, and acquire a plurality of reference text features according to the reference text set; Inputting the prompt text features into a pre-trained image generation model to generate an image result; The image result is evaluated to obtain the actual distribution of the debiased categories in the image result, and the prompt text feature is trained according to the actual distribution, the target distribution and the plurality of reference text features to obtain a trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model; The trained prompt text features are input into the image generation model again to obtain a debiased image result.

2. The image generation model debiasing method based on enhanced cue word learning according to claim 1, characterized in that: The acquiring of prompt text features according to the original text and the prompt word includes: The original text is vectorized to obtain the original text vector T = {v1, v2, ..., v n }; Vectorizing the prompt word to obtain a prompt word vector, and zero-initializing the prompt word vector; The zero-initialized prompt word vector e fair Add to the original text vector T = {v1, v2, ..., v n }Get the prompt text vector T'={v1,v2,…v n-1 ,e fair ,v n }; The prompt text vector T′={v1,v2,…v n-1 ,e fair ,v n }Perform text encoding to obtain the prompt text feature E(T′).

3. The image generation model debiasing method based on enhanced cue word learning according to claim 1, characterized in that: The acquiring of a plurality of reference text features according to the reference text set includes: Set the reference text set to S = set (s1, s2, ..., s i ,…,s n ), where s i is the i-th reference text, and n represents the total value of the reference text; Each of the reference texts in the reference text set i After vectorization, the text is encoded to obtain the corresponding reference text features E(s i ).

4. The image generation model debiasing method based on enhanced cue word learning according to claim 1, characterized in that: The step of evaluating the image result to obtain an actual distribution of debiased categories in the image result includes: Performing debiasing category judgment on the image result to obtain a debiasing category label of the image result; According to the debiased category labels of the image results, the debiased category ratios of the image results are counted, and the actual distribution of the debiased categories in the image results is obtained according to the debiased category ratios.

5. The image generation model debiasing method based on enhanced cue word learning according to any one of claims 1 to 4, characterized in that: The step of training the prompt text feature according to the actual distribution, the target distribution, and the plurality of reference text features to obtain the trained prompt text feature includes: The actual distribution is compared with the target distribution. If the actual distribution conforms to the target distribution, the image result is used as the debiased image result, and the training of the prompt word feature is completed; if the actual distribution does not conform to the target distribution, the deviation between the actual distribution and the target distribution is obtained, and the reward R is calculated according to the deviation, wherein the deviation and the reward R are obtained through the evaluation model; Adjust the distance between the prompt text feature and each reference text feature according to the reward R and the loss function to obtain the adjusted prompt text feature; The adjusted prompt text feature is input into the image generation model again to generate a new image result, and the new image result is continuously evaluated until the training of the prompt word feature is completed to obtain the trained prompt text feature.

6. The image generation model debiasing method based on enhanced cue word learning according to claim 5, characterized in that: The loss function specifically includes: Among them, L rl is the distance difference between the prompt text feature and all the reference text features, sim(·,·) represents the similarity calculation between the prompt text feature and the i-th reference text feature, R i is a reward component, representing the reward value of the prompt text feature and the i-th reference text feature.

7. The method for debiasing an image generation model based on enhanced cue word learning according to claim 5, characterized in that: The step of adjusting the distance between the prompt text feature and each reference text feature according to the reward R and the loss function includes: The reward components R of the reward R i Includes positive and negative values; If the reward component R i is a positive value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) direction adjustment, reducing the distance between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance; If the reward component R i is a negative value, then the prompt text feature is converted to the corresponding i-th reference text feature E(s i ) in the opposite direction, increasing the difference between the prompt text feature and the corresponding i-th reference text feature E(s i ) distance.

8. A debiasing system for image generation model based on reinforcement cue word learning, characterized in that: include: A target distribution acquisition module is used to define a debiasing category and acquire a target distribution of the debiasing category according to the debiasing category; A prompt text feature acquisition module is used to acquire the original text, set prompt words according to the original text and the debiased category, and acquire prompt text features according to the original text and the prompt words; A reference text acquisition module, used for acquiring a reference text set according to the original text and the debiased category, and acquiring a plurality of reference text features according to the reference text set; An image generation module is used to input the prompt text features into a pre-trained image generation model to generate an image result; A training module, used for evaluating the image result, obtaining the actual distribution of the debiased categories in the image result, and training the prompt text feature according to the actual distribution, the target distribution and the plurality of reference text features to obtain the trained prompt text feature; wherein the evaluation of the image result and the training of the prompt text feature are performed in a pre-trained evaluation model; A debiased image module is generated, which is used to input the trained prompt text features into the image generation model again to obtain a debiased image result.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the image generation model debiasing method based on reinforcement prompt word learning as described in any one of claims 1-7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for debiasing an image generation model based on reinforcement cue word learning as described in any one of claims 1 to 7 is implemented.

Images