Portrait generation quality partition optimization system based on Chinese context text map scene

Abstract

The invention discloses a portrait generation quality partition optimization system based on a Chinese context text map scene, and particularly relates to the technical field of artificial intelligence. A preference learning theory is used as a framework, a portrait preference data set is constructed by using an attention aggregation algorithm, a ChineseCLIP model is used as a base model, and training of a portrait reward model is completed by minimizing a cross entropy between annotation preference distribution and a sample normalization score. Therefore, the method can receive input of pictures and prompts and output a score as portrait quality evaluation, automatic evaluation of the portrait quality of the pictures in the Chinese context is completed, and the method has good robustness and generalization. Moreover, a portrait partition preference data set can be constructed based on the portrait reward model, and an effective reward signal is fed back to preference optimization of the text map model by using a partition preference optimization algorithm, so that the problem of low portrait generation quality in a current Chinese context text map scene is solved; and the portrait quality of the text graph in the Chinese scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a system for optimizing the quality of portrait generation based on a Chinese context literary image scene. Background Art

[0002] Text-to-image technology is the main research direction in the field of text-to-image multimodality, which refers to generating images corresponding to given text conditions. The latent diffusion model is an important model representative for text-to-image tasks, which can generate and perform image generation tasks based on externally given text prompts.

[0003] In terms of image quality optimization of the text-based image model, current research mainly focuses on the English context, and has not yet conducted in-depth research on the text-based image model in the Chinese context. In addition, in terms of portrait generation, collapse often occurs, that is, the generated image does not match the distribution of real human images. This means that in the Chinese context, the generation quality of portraits in text-based image scenes is difficult to guarantee, so that the generated image effect cannot meet expectations. To this end, we propose a quality partitioning optimization system for portrait generation in text-based image scenes in the Chinese context. Summary of the invention

[0004] The present invention aims to provide a system for optimizing the quality of portrait generation based on a Chinese context Wenshengtu scene, which solves the problem that the quality of portrait generation in the current Wenshengtu scene cannot meet expectations in the Chinese context.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows: a system for optimizing the quality of portrait generation based on a Chinese contextual literary image scene, comprising the following steps:

[0006] S1. Construct data annotation rules based on the collapse problem of generating portraits to construct a portrait annotation dataset;

[0007] S2, using the attention aggregation algorithm to transform the portrait annotation dataset into a portrait annotation preference dataset;

[0008] S3, using the pre-trained ChineseCLIP model as the base model, and using the portrait annotation preference dataset and the preset cross entropy function to adjust the parameters of the ChineseCLIP model to obtain the portrait reward model;

[0009] S4, collect Chinese portrait prompts and generate portrait images through the Chinese literary image model, use the portrait reward model to give portrait image samples reward signals, and construct a portrait partition preference dataset;

[0010] S5. Use a partition preference optimization algorithm for the Chinese text graph model to adjust the parameters of the Chinese text graph model until the training is completed.

[0011] Furthermore, step S1 data source: collect commonly used or low-quality portrait prompts, and use the Chinese literary image model to generate corresponding different images.

[0012] Furthermore, the labeling standard of step S1 is to construct corresponding labeling standards through the limb number problem, limb shape problem, hand problem, interaction problem and face problem in the collapse problem.

[0013] Furthermore, the aggregation rule of step S2 is: using the attention aggregation algorithm, that is, for each annotation, different types of collapse will be assigned a different weight, representing the user's attention to this type of collapse problem, and finally the annotation results will be integrated; for each image sample, there will be a total attention, which is the sum of the weights corresponding to the problems annotated by all annotators. The smaller the attention level, the smaller the degree of portrait collapse, and vice versa; an image sample and a sample with a higher attention level form a preference data pair, thereby forming a portrait annotation preference data set.

[0014] Furthermore, in step S3, the ChineseCLIP model training process is to minimize the cross entropy between the annotated preference distribution and the normalized score of the sample, so that the portrait reward model obtains human preference knowledge of portrait quality.

[0015] Furthermore, the method of step S4 is as follows: after collecting a large number of portrait prompts and generating corresponding portrait images, the portrait image samples are scored for quality using the portrait reward model, and then arranged in descending order and the score areas are evenly separated, and image samples are extracted from different score areas to construct.

[0016] Furthermore, in step S5, the partition preference optimization algorithm optimizes the Chinese literary image model to be optimized based on the portrait partition preference dataset and the direct preference loss to improve the portrait generation quality of the Chinese literary image model.

[0017] Another technical solution provided by the present invention is: a computer-readable storage medium, which stores a computer program, wherein the computer-readable storage medium stores the above-mentioned optimization system, and / or controls the device where the computer-readable storage medium is located to execute the above-mentioned optimization system when the computer program is running.

[0018] Compared with the prior art, this solution has the following beneficial effects:

[0019] 1. This solution builds a portrait quality partition preference optimization system based on preference learning, carefully designs annotation criteria and aggregation algorithms, trains the reward model by minimizing the relative entropy between the annotation preference distribution and the sample normalized score, and finally uses the trained reward model to score the portrait quality of the portrait image data, builds a portrait preference data set, and uses the designed partition preference optimization algorithm for the Wenshengtu model to improve the portrait generation quality of the Wenshengtu model. In order to prove the effectiveness and generalization of the invention, through manual black box evaluation, the portrait generation quality of the optimized model is significantly improved compared with the unoptimized model, which proves the generalization of the reward model and the effectiveness of the system.

[0020] 2. This scheme constructs a dataset related to portraits in the Chinese context through the proposed attention aggregation algorithm, and obtains a Chinese context portrait reward model through post-training based on the pre-trained Chinese visual-linguistic contrastive learning model (ChineseCLIP). This reward model is suitable for automatic scoring of the quality of portrait generation in literary image scenes in the Chinese context, which significantly improves the model's ability to distinguish the quality of portraits. Based on this model, a partition preference optimization algorithm is proposed based on the preference learning theory to improve the portrait generation quality of the Chinese literary image model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the quality partition optimization system of portrait generation in the Chinese context cultural image scene of the present invention;

[0022] Figure 2 This is a case diagram of the portrait generation quality zoning optimization system based on the Chinese context cultural image scene of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below through specific embodiments:

[0024] Example 1

[0025] like Figure 1 and Figure 2 As shown in FIG. 1 , a quality partitioning optimization system for portrait generation based on a Chinese contextual literary image scene includes the following steps:

[0026] S1. Construct data annotation rules based on the five common collapse problems of Chinese literary image model in generating portraits to construct a portrait annotation dataset. Among them, data source: collect commonly used or low-quality portrait prompts, and use Chinese literary image model to generate corresponding different pictures. Annotation standard: Based on the collapse problems of Chinese literary image model in generating portraits, corresponding annotation standards are constructed, namely, the limb number problem, limb shape problem, hand problem, interaction problem and face problem in the collapse problem. Each major problem will have different detailed classifications and degrees to facilitate the annotator to annotate more accurately.

[0027] S2. Use the attention aggregation algorithm to convert the portrait annotation dataset into a portrait annotation preference dataset; where the aggregation rule is: use the attention aggregation algorithm, that is, for each annotation, different types of collapse will be assigned a different weight, representing the user's attention to this type of collapse problem, and finally the annotation results will be integrated. For each image sample, there will be a total attention, which is the sum of the weights of all annotators in their annotated issues. The lower the attention, the lower the portrait collapse, and vice versa; an image sample and a sample with a higher attention level form a preference data pair, thereby forming a portrait annotation preference dataset.

[0028] S3. Use the pre-trained ChineseCLIP model as the base model (the ChineseCLIP model in this embodiment is implemented using but not limited to the content recorded in the document "Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese", and the alignment of Chinese visual information and language information has been achieved internally), which can not only meet the needs of Chinese scenes, but also obtain the ability of pre-trained Chinese semantic alignment. In order to add the ability to distinguish portrait quality to the base model, the training combines the base model with the above-mentioned portrait annotation preference data set to perform reward model training, and fine-tuning is completed by minimizing the cross entropy between the annotation preference distribution and the sample normalized score. That is, the cross entropy between the annotation preference distribution and the normalized score of the sample is calculated. Specifically, given two samples, if the first sample is of higher quality, p = [1, 0], otherwise p = [0, 1], and the scorer s (c, x) accepts the prompt c and the portrait picture x,

[0029] make

[0030] The loss function is: According to the above results, the gradients of all parameters to be optimized in the model are calculated, and then the gradient descent method is used to update all parameters to be optimized in the model to complete the portrait reward model training. This step improves the consistency of the existing model in the quality of Chinese portraits based on human preferences. The accuracy of human preference prediction on the test set relative to the base model is increased from the original 49.02% to 92.46%.

[0031] S4, collect Chinese portrait prompts and generate portrait images through the Chinese literary image model, use the portrait reward model to give the portrait image samples a reward signal, and construct a portrait partition preference dataset; the method is as follows: after collecting a large number of portrait prompts and generating corresponding portrait images, use the portrait reward model to score the quality of the portrait image samples, arrange them in descending order and evenly separate the score areas, and extract image samples from different score areas to construct, that is, collect Chinese portrait prompts, and use the initial Chinese literary image model ∈ θ (x, t, c) generates an image dataset, where x is the image, t is the sampling time step in the diffusion model, and c is the text prompt, and uses the reward model trained by the above process to assign an image quality score to each sample in the above image dataset to represent the reward signal.

[0032] S5. A partition preference optimization algorithm is used for the Chinese literary image model. Preference learning is a machine learning method that builds a preference prediction model based on known observable preference information. Specifically, it refers to collecting human feedback and manually annotating data samples according to pre-defined standards, and then aggregating these annotated data to build a corresponding data set to optimize the model. That is, the Chinese literary image model to be optimized is optimized based on the portrait partition preference data set and direct preference loss to improve the portrait generation quality of the Chinese literary image model. Adjust the parameters of the Chinese literary image model until the training is completed.

[0033] That is, for the N image samples with the same text prompt, they are first sorted in descending order and evenly divided into c score regions S1, ..., S c , then the image samples in the score area with a smaller subscript have higher scores. In order to construct data pairs with obvious sample quality differences, we select image samples in different score areas to construct preferred data pairs (c, x w , x l ), that is, x w ∈S i , x l ∈S j , i<j, thus constructing the portrait partition preference dataset

[0034] Based on the above portrait partition preference dataset Optimizing according to the direct preference loss model

[0035]

[0036] Among them, ∈ ref Is to use ∈ θ Initialized reference model, is a multidimensional Gaussian distribution, α tand σ t is a predefined diffusion schedule parameter, and β is a hyperparameter that controls the degree of regularization. Through the above-constructed portrait partition preference dataset and calculating the loss of preference optimization, the gradient descent method is used to update all parameters to be optimized in the Wensheng graph model to complete the model training.

[0037] The optimization system of this embodiment is used to optimize the portrait generation quality of the Chinese literary image model, and the optimized Chinese literary image model and the unoptimized version are subjected to a black box human comparison test, and the comparison is made under the conditions of unknown prompts and generated seeds.

[0038] The above black box test fed back the following scores. The higher the score, the higher the portrait quality. There are two groups in the experiment, using the normal 25-step sampling strategy and the low-step 8-step sampling strategy respectively. The test method is as follows: Under the same sampling strategy and environmental conditions, use the models before and after optimization to generate pictures. Subsequently, three evaluators evaluated the generated pictures and selected the one with better quality, or chose "tie" to indicate that the quality of the two pictures is equivalent, and the following value is the number of selected better ones. It can be seen from the experimental results that whether in the normal sampling strategy or the low-step sampling strategy, the optimization system proposed in this embodiment has significantly improved the quality of portrait generation of the Chinese literary model, which proves the generalization of the reward model and the effectiveness of this optimization system.

[0039]

[0040] Example 2

[0041] A computer-readable storage medium storing a computer program, wherein the computer-readable storage medium stores the optimization system of Example 1, and / or controls a device where the computer-readable storage medium is located to execute the optimization system of Example 1 when the computer program is running.

[0042] The above are only embodiments of the present invention, and the common knowledge such as the known specific structures and / or characteristics in the scheme are not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A quality zoning optimization system for portrait generation in Chinese contextual literary and artistic scenes, characterized by: The steps include: S1. Construct data annotation rules based on the collapse problem of generating portraits to construct a portrait annotation dataset; S2, using the attention aggregation algorithm to transform the portrait annotation dataset into a portrait annotation preference dataset; S3, using the pre-trained ChineseCLIP model as the base model, and using the portrait annotation preference dataset and the preset cross entropy function to adjust the parameters of the ChineseCLIP model to obtain the portrait reward model; S4, collect Chinese portrait prompts and generate portrait images through the Chinese literary image model, use the portrait reward model to give portrait image samples reward signals, and construct a portrait partition preference dataset; S5. Use a partition preference optimization algorithm for the Chinese text graph model to adjust the parameters of the Chinese text graph model until the training is completed.

2. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1 is characterized by: Step S1 Data source: Collect commonly used or low-quality portrait prompts, and use the Chinese literary image model to generate corresponding different images.

3. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1 is characterized by: The labeling standard of step S1: construct the corresponding labeling standard through the limb number problem, limb shape problem, hand problem, interaction problem and face problem in the collapse problem.

4. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1 is characterized by: Aggregation rule of step S2: Adopt the attention aggregation algorithm, that is, for each annotation, different crash types will be assigned a different weight, representing the user's attention to this type of crash problem, and finally the annotation results will be integrated; For each image sample, there is a total attention degree, which is the sum of the weights of all annotators corresponding to the questions they annotated. The smaller the attention degree, the less the portrait collapses, and vice versa. An image sample and a sample with a higher attention degree form a preference data pair, thus forming a portrait annotation preference dataset.

5. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1 is characterized by: In step S3, the ChineseCLIP model training process is to minimize the cross entropy between the annotated preference distribution and the normalized score of the sample, so that the portrait reward model obtains human preference knowledge of portrait quality.

6. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1, characterized in that: The method of step S4 is as follows: after collecting a large number of portrait prompts and generating corresponding portrait images, the portrait image samples are scored for quality using the portrait reward model, arranged in descending order and the score areas are evenly separated, and image samples are extracted from different score areas to construct.

7. The system for optimizing the quality of portrait generation based on Chinese contextual images according to claim 1 is characterized by: In step S5, the partition preference optimization algorithm optimizes the Chinese language image model to be optimized based on the portrait partition preference data set and the direct preference loss to improve the portrait generation quality of the Chinese language image model.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer-readable storage medium stores an optimization system as described in any one of claims 1 to 7, and / or controls the device where the computer-readable storage medium is located to execute the optimization system as described in any one of claims 1 to 7 when the computer program is running.

Images