Image generation model optimization method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to an optimization method and device for an image generation model, electronic equipment and a storage medium, and the method comprises the steps: employing each training sample, taking a noise prediction result of a reference model as reference information, carrying out the multi-round iteration training of a basic image generation model, and obtaining a target model; in a round of iterative training, executing the following operations: based on the read training sample, obtaining a target noise reduction time step, a predicted first noise map and a predicted second noise map of the image generation model in random sampling, and obtaining a predicted third noise map and a predicted fourth noise map of the reference model in the target noise reduction time step; and adjusting the image generation model based on the difference between the object areas in the first noise map and the third noise map and the difference between the object areas in the second noise map and the fourth noise map. Therefore, the image generation effect of the object area can be effectively improved.

Description

Background Art

[0002] In the field of text-to-image generation, to solve the problem of poor generation effects of specified types of objects in images, the existing image generation models are usually optimized.

[0003] Under the related technologies, when optimizing the existing image generation model, the image encoding and decoding network in the image generation model can be adjusted first, and then the adjusted image generation model can be trained to obtain a target model with better generation effects; alternatively, other functional networks can be mounted on the image generation model to obtain an image with better generation effects.

[0004] However, in the method of optimizing and training the adjusted image generation model, more training time is required to train the newly adjusted model network, which increases the training cost as a whole; while in the method of predicting and optimizing by mounting other functional networks, additional deployment costs are required when deploying and implementing the complete model for image generation. In addition, due to the above two methods, the model network participating in image generation is changed, so the optimized target model may not be able to effectively run in the original deployment environment, thus unable to guarantee the model optimization effect and reducing the image generation efficiency. Summary of the Invention

[0005] The embodiments of the present application provide an optimization method, device, electronic device and storage medium for an image generation model, so as to obtain an image generation model with better image generation effects without changing the model network participating in image generation.

[0006] In a first aspect, an optimization method for an image generation model is proposed, including:

[0007] Obtain each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image for indicating the sample object regions in the positive sample image and the negative sample image;

[0008] Use the training samples to perform multiple rounds of iterative training on a basic image generation model with the noise prediction result of a reference model as reference information to obtain a target model; the reference model is obtained by copying the image generation model before training; wherein, in one round of iterative training, the following operations are performed:

[0009] Based on the read training sample, obtain a first noise map predicted by the image generation model for the positive sample image and a second noise map predicted by the image generation model for the negative sample image at a randomly sampled target noise reduction time step, and obtain a third noise map predicted by the reference generation model for the positive sample image and a fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step;

[0010] Adjust the image generation model based on the object region differences in the first noise map and the third noise map, and the object region differences in the second noise map and the fourth noise map.

[0011] In a second aspect, an optimization device for an image generation model is proposed, including:

[0012] An acquisition unit for acquiring each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image for indicating the sample object regions in the positive sample image and the negative sample image;

[0013] A training unit for using each of the training samples and taking the noise prediction result of the reference model as reference information to perform multiple rounds of iterative training on a basic image generation model to obtain a target model; the reference model is obtained by copying the image generation model before training; wherein, in one round of iterative training, the following operations are performed:

[0014] Based on the read training sample, obtain a first noise map predicted by the image generation model for the positive sample image and a second noise map predicted by the image generation model for the negative sample image at a randomly sampled target noise reduction time step, and obtain a third noise map predicted by the reference generation model for the positive sample image and a fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step;

[0015] Adjust the image generation model based on the object region differences in the first noise map and the third noise map, and the object region differences in the second noise map and the fourth noise map.

[0016] Optionally, each of the training samples is constructed by the acquisition unit in the following manner:

[0017] Obtain each candidate image containing an object of the target type generated based on each candidate prompt text;

[0018] Among the candidate images, screen out each basic image that respectively contains at least one target object region, and perform redrawing on the target object region for each basic image to obtain each target image that respectively contains at least one sample object region; wherein, the image quality evaluation value in each target object region is lower than a first threshold; the image quality evaluation value in each sample object region is higher than a second threshold, and the region coincidence degree with a corresponding target object region is higher than a coincidence degree threshold;

[0019] For each target image, perform the following operations: construct a mask image corresponding to the sample object region, and combine the corresponding base image and the candidate prompt text to construct a training sample.

[0020] Optionally, when screening out each base image that contains at least one target object region from the candidate images, the obtaining unit is configured to:

[0021] For each candidate image, perform the following operations respectively:

[0022] Perform object region detection of the target type object on the candidate image to obtain first region information, and perform image quality detection on the region content of each object region indicated by the first region information to obtain the image quality evaluation value corresponding to each object region;

[0023] When it is determined that there are target object regions in the object regions whose image quality evaluation values are higher than the first threshold, determine the candidate image as the base image.

[0024] Optionally, when redrawing the target object region for each base image respectively, the obtaining unit is configured to:

[0025] For each base image, perform the following operations respectively:

[0026] Crop out at least one target object region in the base image respectively to obtain at least one sub-image;

[0027] Replace the corresponding target object region in the base image with at least one object image generated based on the at least one sub-image and the preset optimized prompt text respectively to obtain a redrawn image.

[0028] Optionally, when obtaining each target image that respectively contains at least one sample object region after screening, the obtaining unit is configured to:

[0029] For each redrawn image obtained by redrawing, perform the following operations respectively:

[0030] Perform object region detection of the target type object on the redrawn image to obtain second region information, and determine at least one candidate sample region corresponding to at least one target object region in the corresponding base image according to the region content indicated by the second region identification information;

[0031] For the at least one candidate sample region, determine the corresponding image quality evaluation value and the region coincidence degree with the corresponding target object region respectively;

[0032] When it is determined that there is a sample object region in the at least one candidate sample region whose image quality evaluation value is higher than the second threshold and whose region overlap degree is higher than the overlap degree threshold, the redrawn image is determined as the target image.

[0033] Optionally, when constructing the mask image corresponding to the sample object region, the obtaining unit is used for:

[0034] Based on the object region information of the sample object region according to the training basis and the corresponding set expansion coefficient, obtain the expanded region information;

[0035] In a constructed image of the same image size, set the pixel values of each pixel point in the expanded region indicated by the expanded region information to 1, and set the pixel values of each pixel point in other regions to 0 to obtain the mask image.

[0036] Optionally, when combining the corresponding base image and the candidate prompt text to construct a training sample, the obtaining unit is used for:

[0037] Use the corresponding base image as the negative sample image in the training sample, use the target image as the positive sample image that is more preferred relative to the negative sample image in the training sample, and use the candidate prompt text corresponding to the base image as the sample prompt text in the training sample;

[0038] In the training sample, add a mask image for indicating the sample object region according to the training basis, and the image quality evaluation value in the sample object region in the positive sample image is higher than the second threshold; the image quality evaluation value in the sample object region in the negative sample image is lower than the first threshold.

[0039] Optionally, when obtaining the first noise map predicted by the image generation model for the positive sample image and the second noise map predicted for the negative sample image at the randomly sampled target noise reduction time step, the training unit is used for:

[0040] Obtain a target noise reduction time step randomly sampled within the noise reduction time step range during the noise reduction process;

[0041] Use the image generation model to perform image noise reduction based on the sample prompt text and the positive sample image read, obtain the first noise map predicted at the target noise reduction time step, and perform image noise reduction based on the sample prompt text and the negative sample image read in the training sample to obtain the second noise map predicted at the target noise reduction time step.

[0042] Optionally, when obtaining the third noise map predicted for the positive sample image and the fourth noise map predicted for the negative sample image by the reference model at the target noise reduction time step, the training unit is configured to:

[0043] Use the reference model to perform image denoising based on the sample prompt text and the positive sample image to obtain a third noise map predicted at the target noise reduction time step, and perform image denoising generation processing based on the sample prompt text and the negative sample image to obtain a fourth noise map predicted at the target noise reduction time step.

[0044] Optionally, when adjusting the image generation model based on the object region differences between the first noise map and the third noise map, and the object region differences between the second noise map and the fourth noise map, the training unit is configured to:

[0045] Obtain a first Gaussian map and a second Gaussian map respectively determined for the positive sample image and the negative sample image during the noise addition process;

[0046] For the sample object region indicated by the mask image, determine a first region difference between the first Gaussian map and the first noise map, and a third region difference between the first Gaussian map and the third noise map, and determine a second region difference between the second Gaussian map and the second noise map, and a fourth region difference between the second Gaussian map and the fourth noise map;

[0047] Adjust the image generation model based on the object region difference determined by the difference result between the first region difference and the third region difference, and the object region difference determined by the difference result between the fourth region difference and the second region difference.

[0048] Optionally, when determining the first region difference between the first Gaussian map and the first noise map based on the mask image, the training unit is configured to:

[0049] Adjust the image size of the mask image according to the image sizes of the first Gaussian map and the first noise map, and based on the adjusted mask image, determine the Gaussian image content and the noise image content corresponding to the sample object region in the first Gaussian map and the first noise map respectively;

[0050] Obtain the first region difference by calculating the f2 norm between the Gaussian image content and the noise image content.

[0051] Optionally, the target model includes a text encoding network, a subject generation network, an image encoding network, and an image decoding network; after obtaining the target model, the apparatus further includes a processing unit, and the processing unit is configured to:

[0052] Obtain a target prompt text;

[0053] Use the text encoding network to encode the target prompt text into text features in the latent space, and use the subject generation network to perform multiple denoising processes based on randomly generated latent space noise with the text features as guiding conditions to obtain latent space features;

[0054] Use the image decoding network to decode the latent space features to obtain an image to be fed back.

[0055] In a third aspect, an electronic device is proposed, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above method is implemented.

[0056] In a fourth aspect, a computer-readable storage medium is proposed, on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0057] In a fifth aspect, a computer program product is proposed, including a computer program. When the computer program is executed by a processor, the above method is implemented.

[0058] The beneficial effects of this application are as follows:

[0059] This application proposes an optimization method, apparatus, electronic device, and storage medium for an image generation model, and proposes to first obtain each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and marked with a preference relationship, and a mask image for indicating the sample object area in the positive sample image and the negative sample image; then use each training sample to perform multiple rounds of iterative training on the basic image generation model with the noise prediction result of the reference model as reference information to obtain the target model; the reference model is obtained by copying the image generation model before training; this makes the target model obtained after the optimized training be trained based on the basic image generation model. Therefore, when optimizing the image generation model, neither the model network in the image generation model is adjusted, nor other functional networks are mounted on the basis of the image generation model, so that the optimization process is carried out for the basic image generation model itself. Therefore, in the process of deploying the optimized model, there is no problem that it cannot be deployed in the original deployment environment, so that the optimized target model can effectively run in the original deployment environment, ensuring the model optimization effect and helping to improve the image generation efficiency;

[0060] Moreover, taking one round of iterative training as an example, the following operations are proposed to be performed during one round of iterative training: Based on the read training samples, obtain the first noise map predicted by the image generation model for the positive sample image and the second noise map predicted by the image generation model for the negative sample image at the randomly sampled target noise reduction time step, and obtain the third noise map predicted by the reference generation model for the positive sample image and the fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step; then, based on the object region differences between the first noise map and the third noise map, and the object region differences between the second noise map and the fourth noise map, adjust the image generation model. In this way, during the specific training process, by comparing the object region differences between the first noise map and the third noise map, the noise prediction differences of the image generation model and the reference model for the object region can be obtained for the positively labeled sample images with annotation preferences; similarly, by comparing the fourth noise map and the second noise map, the noise prediction differences of the reference model and the image generation model for the object region can be obtained for the negatively labeled sample images that are not preferred; furthermore, based on the object region differences from two perspectives, the preference optimization training of the image generation model can be achieved. Moreover, since the object region differences are focused on during the training process, the image generation effect of the object region can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of a possible application scenario in an embodiment of the present application;

[0062] Figure 2 Schematic diagram of the optimization process of the image generation model in an embodiment of the present application;

[0063] Figure 3 Schematic diagram of the construction process of each training sample in an embodiment of the present application;

[0064] Figure 4 Schematic diagram of a possible process for obtaining a sub-image in an embodiment of the present application;

[0065] Figure 5 Schematic diagram of another possible process for obtaining a sub-image in an embodiment of the present application;

[0066] Figure 6 Schematic diagram of the process for obtaining a redrawn image in an embodiment of the present application;

[0067] Figure 7 Schematic diagram of the process for determining the region overlap degree in an embodiment of the present application;

[0068] Figure 8 Schematic diagram of the process for screening and determining the target image in an embodiment of the present application;

[0069] Figure 9 Schematic diagram of the possible construction process of the mask image in an embodiment of the present application;

[0070] Figure 10 It is a possible structural schematic diagram of the image generation model in the embodiments of the present application;

[0071] Figure 11 It is a processing schematic diagram of one round of iterative training in the embodiments of the present application;

[0072] Figure 12 It is a logical structural schematic diagram of the optimization device of the image generation model in the embodiments of the present application;

[0073] Figure 13 It is a hardware composition structural schematic diagram of an electronic device applying the embodiments of the present application;

[0074] Figure 14 It is a hardware composition structural schematic diagram of another electronic device applying the embodiments of the present application. Specific embodiments

[0075] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the technical solutions of the present application. Based on the embodiments recorded in this application document, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the technical solutions of the present application.

[0076] Terms such as "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here.

[0077] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit including the function of that module or unit.

[0078] The following explains some terms in the embodiments of the present application to facilitate the understanding of those skilled in the art.

[0079] Preference relation: used to describe human preferences for image generation; specifically, for positive and negative sample images labeled with preference relations, a positive sample image is a high-quality image that annotators prefer to generate based on a sample prompt text, while a negative sample image is a low-quality image that annotators do not prefer to generate based on the sample prompt text by comparison.

[0080] Target type object: It is a type of object to be optimized. It refers to the object corresponding to the image content with very poor image quality determined by pre-analysis when it appears in the images generated by the basic image generation model; in other words, it refers to the object with poor generation effect in the images generated by the basic image generation model; or it can be understood as the specified type of object selected from various objects with poor generation effects in the image generation model; the target type object is equivalent to indicating the image content that needs to be optimized for generation; for example, the target type object can specifically be one of various organisms in various states, various organism organs, and various static items, etc. For example, the target type object can be a "small face", or it can be an "eye", or it can be a "clothes", etc.; among them, a small face refers to a face where the proportion of the face area in the whole image is lower than the set threshold.

[0081] Basic image generation model: It refers to the model targeted by the optimization training method proposed in this application. It should be understood that the basic image generation model usually has basic image generation capabilities and can achieve good generation effects when generating certain types of object content in the image; for example, the image generation model has the ability to generate adult face images, where an adult face image refers to an image in which the proportion of the face area in the generated image is higher than the first threshold.

[0082] Small face image: It refers to an image that contains at least one face area with a proportion lower than the set threshold relative to the whole image, or an image containing a small face. For example, a small face image can specifically be embodied as a full-body image, a panoramic image of a person, etc.

[0083] Target denoising time step: It refers to a denoising time step randomly sampled during the reverse generation process (or denoising process) of an image generation model. It should be understood that the reverse generation process refers to the process of gradually removing noise to achieve image generation. During the reverse generation process of an image generation model, there are multiple denoising time steps, and each denoising time step corresponds to an operation of removing part of the noise. In one denoising time step, denoising is specifically achieved by predicting and removing noise. Additionally, the total number of denoising time steps in the reverse generation process of the image generation model is set according to actual training needs. Moreover, the total number of denoising time steps adopted by the trained image generation model during the denoising process may be different from the total number of denoising time steps adopted during the training process. During model training, skip sampling is usually performed among the total denoising time steps to determine the target denoising time step. For example, the total number of denoising time steps during model training can be 1000, and after model training, without acceleration, the total number of denoising time steps adopted is generally around 25 - 40 steps.

[0084] The following briefly introduces the design concept of the embodiments of the present application:

[0085] Currently, in the field of text-to-image generation, to solve the problem of poor generation effect of specified types of objects in images, existing image generation models are usually optimized.

[0086] For example, to address the problem of poor generation effect of small face images, existing image generation models need to be optimized and trained to improve the generation effect of small object images.

[0087] When conceiving an optimization scheme to improve the generation effect of an image generation model, the applicant first thought that model optimization could be achieved by adjusting the image quality in the training samples.

[0088] Specifically, a combination of artificial intelligence (AI) screening and manual screening can be used to screen out pictures with extremely high picture quality for training. Moreover, considering the relatively high requirements for image quality, personnel with relatively rich art backgrounds such as photographers and painters are usually introduced for processing, so as to directionally collect high-quality data related to the target type of object, thereby improving the generation effect of text-to-image for images containing the target type of object.

[0089] However, since constructing high-quality training samples requires a large amount of time and labor costs, it is difficult to ensure the generation efficiency of training samples. Additionally, limited by the model expression ability of the image generation model and the training objective based on Mean Squared Error (MSE), relying solely on data-level optimization cannot effectively improve the generation effect of images containing the target type of object.

[0090] Therefore, effective optimization of the image generation model cannot be achieved by adjusting the image quality in the training samples.

[0091] Furthermore, the applicant conceives that model optimization can be achieved by adjusting some of the networks within the image generation model.

[0092] Specifically, considering that a basic image generation model usually includes four parts: a main body generation network, a text encoding network (text-encoder), an image encoding network, and an image decoding network; moreover, the combined effect of the image decoding network, the main body generation network, and the image decoding network is to compress the input image into the latent variable space, generate in the latent variable space, and finally restore the generated image based on the generated latent variable. Therefore, in view of the fact that common image encoding networks and image decoding networks usually compress an image into a set of features with a dimension of 4, and the compression ratio is too large, so details and other features of the target type object will naturally be lost, resulting in a poor generation effect for images containing the target type object. The applicant conceives that the image encoding network and the image decoding network (abbreviated as the image codec network) can be adjusted to an image codec network with a relatively low compression ratio.

[0093] However, although adjusting the image codec network can, to a certain extent, solve the problem of poor generation of images containing the target type object, this adjustment not only requires training a new image codec network, but also requires training other network structures in the overall image generation model. The overall training time, training cost, and data consumed by training are relatively large. Moreover, because there are many common image generation deployment environments that only support several common image codec networks, the optimized target model may not be able to run effectively in these deployment environments, thus unable to guarantee the model optimization effect and reducing the image generation efficiency.

[0094] Therefore, effective optimization of the image generation model cannot be achieved by adjusting some of the networks within the image generation model either.

[0095] Furthermore, the applicant conceives that by mounting other functional networks on the basis of the basic image generation model, and then, during the image generation process, jointly processing with the image generation model and other functional networks.

[0096] Specifically, considering that the basic image generation model mainly generates images based on the iterative denoising method, the applicant conceives that during the processing of the denoising time step, the corresponding noisy results can be taken out, the object key point information can be predicted for the target type object, and the prediction results can be sent to the added ControlNet network for prediction guidance. Alternatively, based on the image generation model, a functional network for realizing object area redrawing can be added to ensure the final generated image effect.

[0097] However, whether adding the ControlNet network or adding a functional network for realizing object area redrawing, other functional networks need to be added on the basis of the basic image generation model, so that in the final target model participating in image generation, it includes both the basic image generation model and the added other functional networks; this will bring more deployment costs and image generation time costs. Moreover, the optimized target model may not be able to effectively run in the deployment environment of the basic image generation model, thus unable to guarantee the model optimization effect and reducing the image generation efficiency.

[0098] Therefore, adding other functional networks on the basis of the image generation model cannot effectively optimize the image generation model either.

[0099] In view of the above-mentioned optimized method, the present application proposes an optimization method, device, electronic device and storage medium for an image generation model, and first obtains each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image for indicating the sample object area in the positive sample image and the negative sample image; then uses each training sample, with the noise prediction result of the reference model as the reference information, to perform multiple rounds of iterative training on the basic image generation model to obtain the target model; the reference model is obtained by copying the image generation model before training; this makes the target model obtained after the optimized training based on the basic image generation model. Therefore, when optimizing the image generation model, neither the model network in the image generation model is adjusted, nor other functional networks are mounted on the basis of the image generation model, so that the optimization process is carried out for the basic image generation model itself. Therefore, in the process of deploying the optimized model, there is no problem that it cannot be deployed in the original deployment environment, so that the optimized target model can effectively run in the original deployment environment, guaranteeing the model optimization effect and helping to improve the image generation efficiency;

[0100] Moreover, taking one round of iterative training as an example, the following operations are proposed during one round of iterative training: Based on the read training samples, obtain the first noise map predicted by the image generation model for the positive sample image and the second noise map predicted by the image generation model for the negative sample image at the randomly sampled target noise reduction time step, and obtain the third noise map predicted by the reference generation model for the positive sample image and the fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step; then, based on the object region differences between the first noise map and the third noise map, and the object region differences between the second noise map and the fourth noise map, adjust the image generation model. In this way, during the specific training process, by comparing the object region differences between the first noise map and the third noise map, the noise prediction differences of the image generation model and the reference model for the object region can be obtained for the positively labeled sample images with annotation preferences; similarly, by comparing the fourth noise map and the second noise map, the noise prediction differences of the reference model and the image generation model for the object region can be obtained for the negatively labeled sample images without preferences; furthermore, based on the object region differences from two perspectives, the preference optimization training of the image generation model can be realized, and moreover, since the object region differences are emphasized during the training process, the image generation effect of the object region can be effectively improved.

[0101] The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0102] Refer to Figure 1 As shown, it is a schematic diagram of a possible application scenario in an embodiment of the present application. In this schematic diagram of the application scenario, it includes a client device 110 and a processing device 120.

[0103] In a feasible embodiment of the present application, the processing device 120 can respond to an image generation instruction triggered by a business object on the client device 110 based on the input image generation basis, use the optimized target model to perform image generation processing based on the obtained image generation basis, and send the generated image to be feedback to the client device 110, so that the client device 110 presents the image to be feedback to the business object. Among them, the image generation instruction triggered by the business object can be initiated on any one of a mini-program application, a client application, and a web application. The present application does not make specific limitations on this; the business object is a user with an image generation need.

[0104] In addition, in some feasible embodiments, the image generation basis can be a target prompt text; in other feasible embodiments, the image generation basis can be a target prompt text and a target prompt image; in still other feasible embodiments, the image generation basis can be a target prompt image.

[0105] The client device 110 includes, but is not limited to, mobile phones, tablet computers, laptops, e - book readers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft, etc.

[0106] The processing device 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0107] In the embodiments of the present application, the client device 110 and the processing device 120 can communicate through a wired network or a wireless network. In the following description, the optimization process of the image generation model is described only from the perspective of the processing device 120.

[0108] The following is a schematic description of the application scenarios for processing using the optimized target model:

[0109] Application scenario 1: When the target type object is a small face, generate a full - body image.

[0110] Specifically, the processing device can, according to the full - body image generation requirement proposed by the user, adopt the optimized target model, and on the premise of ensuring the content generation quality of the face area, generate a full - body image that meets the requirements based on the image generation basis in the full - body image generation requirement, where the image generation basis includes at least one or a combination of the following: target prompt text; target prompt image.

[0111] For example, based on a face image and a target prompt text, generate a corresponding full - body image.

[0112] Application scenario 2: Generate a group image.

[0113] Specifically, the processing device can, according to the group image generation requirement proposed by the user, adopt the optimized target model, and on the premise of ensuring the generation quality of the target type object in the image, generate a group image that meets the requirements based on the image generation basis in the group image generation requirement, where the image generation basis includes at least one or a combination of the following: target prompt text; target prompt image.

[0114] For example, when the target type object is a small face, the group image can be an image of a crowd.

[0115] Application scenario 3: Generate an original painting image during the original painting design process.

[0116] Specifically, the processing device can, according to the image generation requirements proposed by the original painting designer, adopt the optimized target model, and on the premise of ensuring the generation quality of the target type object in the image, generate the original painting image required by the original painting designer based on the image generation basis in the image generation requirements, where the image generation basis includes at least one or a combination of the following: target prompt text; target prompt image.

[0117] For example, the processing device can generate the required original painting image according to the image material drawn by the original painting designer and the image scene described by the original designer in the target prompt text.

[0118] Application scenario 4: Generate material images.

[0119] Specifically, the processing device can, according to the image generation requirements proposed by the user, adopt the optimized target model, and on the premise of ensuring the generation quality of the target type object in the image, generate the required material image based on the image generation basis in the image generation requirements, where the image generation basis includes at least one or a combination of the following: target prompt text; target prompt image.

[0120] For example, in the process of generating AI materials, various required material images can be generated in batches based on the image generation requirements.

[0121] In addition, it should be understood that in the specific implementation manner of this application, which involves the optimization process of the image generation model, when the embodiments described in this application are applied to specific products or technologies, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0122] The following will, from the perspective of the processing device, explain the optimization process of the image generation model with reference to the accompanying drawings:

[0123] Refer to Figure 2 as shown, which is a schematic diagram of the optimization process of the image generation model in the embodiment of this application. The following will, in combination with the attached Figure 2 , make a specific explanation:

[0124] Step 201: The processing device obtains each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image used to indicate the sample object area in the positive sample image and the negative sample image.

[0125] In the embodiments of the present application, before optimizing the basic image generation model, the processing device needs to first obtain each training sample relied on during the optimization training process. Moreover, the sample object area indicated in the masked image can be specifically understood as the image area relied on for training.

[0126] In the optimization method proposed in the present application, it focuses on optimizing the generation quality of the object area where the target type object is located in the image generation model, so that when specifically generating an image containing the target type object, a very good generation effect can be obtained.

[0127] In a feasible implementation manner, the target type object can be a small face.

[0128] For example, the image containing a small face can specifically be any one of the following: full-body images in various backgrounds, full-body images in various styles, crowd images in various backgrounds, crowd images in various styles, multi-entity images in various styles, multi-entity images in various backgrounds, etc. Among them, the entity types corresponding to the multi-entities may be any one or combination of the following: people, various animals, and various static items.

[0129] In a feasible implementation manner, the target type object can be a small face part, where the small face part refers to a face where the proportion of a single face area in the overall image is lower than a set threshold; for the generation optimization of the small face part in the image, it can specifically be the optimization of the human face area, or it can be the optimization of the face areas of various animals. The present application does not make specific limitations on this; moreover, the sub-objects covered by the generation optimization of the small face part can be any one category or combination of the following: humans, and various animals.

[0130] It should be understood that in the embodiments of the present application, there may be many types of objects to be optimized. In some feasible implementation manners, each type of object to be optimized can be used as the target type object respectively, and the basic image generation model can be optimized and trained according to each training sample constructed for the target type object to obtain a target model, so that different target models can be trained for different types of objects to be optimized; in other feasible implementation manners, each type of object to be optimized can be used as the target type object respectively, and for each target type object, the corresponding training samples can be constructed, and the basic image generation model can be adjusted according to each training sample corresponding to each target type object respectively, and finally a general target model can be obtained.

[0131] Moreover, whether it is to optimize and obtain the corresponding target models respectively for different types of objects to be optimized, or to optimize and obtain a general target model based on different types of objects to be optimized, the same optimization logic is used for processing during the relevant optimization process, and there are differences in the training samples relied on during the optimization process.

[0132] In the following description of this application, only the generation optimization of the target type of object by the image generation model is taken as an example to schematically illustrate the construction process of each training sample; moreover, it should be understood that each training sample can be constructed by the processing device itself, or can be obtained by the processing device from other devices, and this application does not make specific restrictions on this. In the following description, only the case where the processing device constructs each training sample by itself is taken as an example for schematic illustration.

[0133] Refer to Figure 3 As shown, it is a schematic diagram of the construction process of each training sample in the embodiment of this application. The following will be specifically described in conjunction with the attached Figure 3 , as follows:

[0134] Step 301: The processing device obtains each candidate image containing the target type of object generated based on each candidate prompt text.

[0135] In the embodiment of this application, when the processing device obtains each candidate image, in some feasible implementation manners, the processing device can directly obtain each candidate image containing the target type of object generated in advance based on each candidate prompt text; in other feasible implementation manners, the processing device can first obtain each candidate prompt text for generating an image containing the target type of object, and then use a basic image generation model to generate corresponding candidate images respectively under the guidance of each candidate prompt text, where the total number of generated candidate images is set according to actual processing needs.

[0136] For example, about 100,000 candidate images can be generated.

[0137] In some embodiments of obtaining each candidate prompt text that are feasible, the processing device can screen out each candidate prompt text from the historical prompt texts on which the historical image generation process is based by means of online return.

[0138] For example, the processing device can obtain each historical prompt text used by each historical user in the image generation process, and according to the feedback results of each historical object, first screen out each initial prompt text that guides the generation of historical images satisfactory to the historical user; then, from each initial prompt text, screen out each candidate prompt text containing the prompt keywords for generating an image of the target type of object, where the image of the target type of object indicates an image containing the target type of object.

[0139] Also for example, continuing the above example, the "satisfactory" result is specifically obtained by analyzing the feedback of the historical user on the historical image; for example, the historical user no longer adds prompt content, or the content feedback by the historical user has a praising meaning, etc. Among them, the judgment of whether it has a praising meaning is obtained by performing sentiment analysis on the feedback content, or can be determined by analyzing whether the feedback content contains sentiment keywords expressing a praising meaning.

[0140] For another example, the processing device may obtain each historical prompt text used by each historical user during the image generation process, and then directly screen out each candidate prompt text containing the prompt keywords for generating the image of the target type object from each historical prompt text.

[0141] In some other feasible embodiments of obtaining each candidate prompt text, the processing device may generate each candidate prompt text with the help of a large language model.

[0142] For example, the processing device may input the basic prompt text into the large language model to obtain multiple candidate prompt texts output by the large language model.

[0143] For another example, the basic prompt text input into the large language model may be: Please simulate the sentence "A girl is taking a full-body photo" to generate 100 similar sentences.

[0144] In still some other feasible embodiments of obtaining each candidate prompt text, the processing device obtains each candidate prompt text obtained by editing each labeled object, where each labeled object may specifically be a developer who sorts out the training samples.

[0145] It should be noted that in the embodiments of the present application, the prompt keywords for generating the image of the target type object in each candidate prompt text are specifically used to indicate the content including the target type object in the generated image.

[0146] For example, the target type object is a small face, and the content of the target type object is the content of the small face. Among them, the content of the small face refers to the regional content in a face area with a proportion lower than the set threshold relative to the overall image; the face area where the content of the small face is located may also be referred to as the small face area.

[0147] For example, continuing with the example where the target type object is a small face, the prompt keywords may be various texts that can express generating the content of the small face in the image. It can be "small-sized face", "low-resolution face", "whole body", and "crowd", etc.

[0148] Step 302: The processing device screens out each basic image that respectively contains at least one target object area from each candidate image, and respectively performs target object area redrawing on each basic image to obtain each target image that respectively contains at least one sample object area.

[0149] Among them, the image quality evaluation value in each target object area is lower than the first threshold; the image quality evaluation value in each sample object area is higher than the second threshold, and the regional coincidence degree with a corresponding target object area is higher than the coincidence degree threshold.

[0150] In an embodiment of the present application, when screening out basic images each containing at least one target object region from each candidate image, the processing device may perform the following operations for each candidate image respectively: perform object region detection of target type objects on the candidate image to obtain first region information, and perform image quality detection on the region content of each object region indicated by the first region information respectively to obtain an image quality evaluation value corresponding to each object region; and when determining that there are target object regions in each object region with an image quality evaluation value higher than a first threshold, determine the candidate image as a basic image.

[0151] Specifically, taking the processing of one candidate object as an example, when performing object region detection, the processing device may use a target detection model matching the target type object to perform object region detection on the candidate image to obtain first region information, where the first region information is used to indicate the object region where the target type object is located and its position in the candidate image; then, corresponding to at least one object region indicated by the first region information, crop at least one image to be evaluated from the candidate image, and use a matching image quality evaluation model to obtain corresponding image quality evaluation values respectively based on each image to be evaluated; afterwards, when determining that there are object regions with an image quality evaluation value lower than the first threshold, take at least one object region with an image quality evaluation value lower than the first threshold as a target object region respectively, and determine the candidate image as a basic image.

[0152] For the target detection model used for detection and the image quality evaluation model for evaluating image quality, a matching model can be obtained according to actual detection needs.

[0153] For example, assume that the target type object is a small face. Then, the matching models are a face detection model and a face quality assessment model. The face detection model can be constructed using any network structure capable of face region detection, such as Faster Regions with Convoutional Nerual Networks (Faster R-CNN), You Only Look Once (YOLO) series networks, Single Shot MultiBox Detector (SSD) networks, and Multi-task Cascaded Convolutional Networks (MTCNN). The face quality assessment model can be constructed using a Convolutional Neural Network (CNN) or any network structure capable of face quality assessment. This application does not specifically limit the construction process and training process of the face detection model and the face quality assessment model.

[0154] Another example, assume that the target type object is a small face part. Considering that the small face part may cover small faces and various small animal faces, etc., when obtaining the matching target detection model and image quality assessment model, models capable of target detection and image quality assessment for human faces and various animal faces need to be obtained and processed respectively. This application does not explain the related model construction and training processes here.

[0155] Another example, assume that the target type object is an eye. Then, a target detection model with the eye as the detection target and an image quality assessment model capable of evaluating the image quality of the eye region can be used as the matching models.

[0156] Specifically, in some feasible embodiments, when screening the basic images, the screening can also be combined with the region size of the target object region, so that when it is determined that at least one target object region in the candidate image has an image quality assessment value lower than the first threshold and the proportion of the region content in the overall image is lower than the set threshold, the candidate image is used as the screened basic image. Based on this, after obtaining the image quality assessment values of each object region in the candidate image, the proportion of the region content of each object region in the overall image can be calculated. Furthermore, when it is determined that there is an object region with an image quality assessment value lower than the first threshold and the corresponding proportion lower than the set threshold, at least one object region with an image quality assessment value lower than the first threshold and the proportion of the region content in the overall image lower than the set threshold is used as the target object region respectively, and the candidate image is determined as the basic image.

[0157] For example, in the case where the target type object is a small face or a small human face, when it is determined that at least one target object region with very poor image quality and a very small face region is included in the candidate image, the candidate object can be used as the base image.

[0158] Optionally, in some other feasible embodiments, the processing device may determine a candidate image in which each included object region is a target object region as the base image.

[0159] In this way, a base image including a target type object with very low image quality can be screened out from the candidate images.

[0160] In a feasible implementation manner of the present application, the processing device may respectively use the determined base images as candidate sample images, save them in the negative sample candidate set, and record the first region information for determining the object regions in each candidate sample image, the indication information for indicating the target object regions in the candidate samples, and the candidate prompt text corresponding to each candidate sample image, where the first region information may specifically be the position information of the face detection frame.

[0161] Further, when redrawing the target object regions for each base image respectively, the processing device may perform the following operations for each base image respectively: respectively crop at least one target object region in the base image to obtain at least one sub-image; then respectively use at least one object image generated based on at least one sub-image and a preset optimized prompt text to replace the corresponding target object regions in the base image to obtain a redrawn image.

[0162] In a feasible implementation manner, the obtained at least one sub-image may be at least one image to be evaluated based on which the image quality score of the candidate image is calculated, or may be a sub-image re-cropped from the candidate image according to the first region information and the indication information of the target object region.

[0163] It should be noted that in the embodiments of the present application, for a obtained sub-image, in some feasible implementation manners, it may be directly obtained by cropping a target object region from the base image; in some other feasible implementation manners, it may be obtained by expanding the region according to a preset expansion ratio on the basis of a target object region and then cropping the expanded target object region from the candidate image, where the preset expansion ratio is set according to actual processing needs, and the present application does not make specific limitations on this; when expanding a target object region, a reference pixel point may be selected in the target object region, and the region may be expanded while keeping the position of the reference pixel point unchanged.

[0164] For example, refer to Figure 4As shown, it is a schematic diagram of a possible process for obtaining a sub-image in an embodiment of the present application. According to the Figure 4 content shown, assuming that the target type object is a small face, in the process of obtaining the sub-image, face region detection can be first performed on the candidate image to determine the face region indicated by the first region information in the candidate image. Then, the face region is cropped from the candidate image to obtain an image to be evaluated. After that, face quality evaluation is performed on the image to be evaluated. When it is determined that the face quality evaluation value (i.e., the image quality evaluation value) is lower than the first threshold and the proportion of the region content of the face region in the overall image is lower than the set threshold, the candidate image is used as the base image, and the cropped image to be evaluated is used as the sub-image.

[0165] For another example, refer to Figure 5 As shown, it is another schematic diagram of a possible process for obtaining a sub-image in an embodiment of the present application. According to the Figure 5 content shown, when obtaining the sub-image, face region detection can be first performed on the candidate image to determine the face region indicated by the first region information, and then the enlarged face region is determined according to the preset enlargement ratio. After that, the enlarged face region is cropped from the candidate image to obtain an image to be evaluated. Further, face quality evaluation is performed on the image to be evaluated. When it is determined that the face quality evaluation value is lower than the first threshold and the proportion of the region content of the face region in the overall image is lower than the set threshold, the candidate image is determined as the base image, and the cropped image to be evaluated is used as the sub-image.

[0166] In addition, when generating at least one object image based on at least one sub-image and the preset optimization prompt text, in some feasible implementation manners, the basic image generation model in the present application can be used for image generation; in other feasible implementation manners, other models capable of generating high-quality object images can be used for image generation; wherein, the preset optimization prompt text is used to indicate generating an object image of the target type object with higher quality based on the sub-image.

[0167] For example, assuming that the target type object is a small face, the optimization prompt text can specifically be "Please generate an image with higher face quality based on this image", or "Please optimize the face content in this image", or "Beautiful face", or other texts that can express the same meaning.

[0168] Furthermore, after obtaining at least one object image, the at least one object image can be synthesized into the position corresponding to the at least one sub-image in the base image to obtain a redrawn image after pasting processing; the specific implementation manner of the pasting process is not limited in the present application.

[0169] For example, refer to Figure 6As shown, it is a schematic diagram of the process of obtaining the redrawn image in the embodiment of the present application. According to the content Figure 6 shown, after determining the base image and the sub-image, a higher-quality object image can be generated based on the sub-image. Then, by pasting the object image at the corresponding position in the base image, the redrawn image can be obtained.

[0170] Similarly, the processing device can execute the above process for each base image respectively to generate the corresponding redrawn image.

[0171] In this way, among the determined base images, there is at least one target type object with poor image quality; moreover, by redrawing the corresponding object image for the target object area of the base image again, and replacing the target object area in the base image with the object image, the pasting process of the base image can be realized, so that the base image can be adjusted in the direction of improving the image quality of the target type object.

[0172] After that, when the processing device redraws each base image to obtain each target image that respectively includes at least one sample object area, the processing device can perform image quality evaluation on the redrawn images obtained by redrawing the base images, so as to screen out each target image.

[0173] In addition, in a feasible implementation manner of the present application, the processing device can respectively use each redrawn image as each candidate sample image, save it in the positive sample candidate set, and record the second area information for determining the object area in each candidate sample image, the indication information for indicating the sample object area whose image quality evaluation value is higher than the second threshold, and the candidate prompt text corresponding to each candidate sample image, where the second area information can specifically be the position information of the detection frame of the target type object.

[0174] When specifically screening out the target images from the redrawn images, the processing device can perform the following operations for each redrawn image obtained by redrawing: perform object area detection on the redrawn image for the target type object to obtain the second area information, and determine at least one candidate sample area corresponding to at least one target object area in the corresponding base image according to the area content indicated by the second area identification information; then, respectively determine the corresponding image quality evaluation value and the area coincidence degree with the corresponding target object area for at least one candidate sample area; after that, when it is determined that there is a sample object area in at least one candidate sample area whose image quality evaluation value is higher than the second threshold and the area coincidence degree is higher than the coincidence degree threshold, the redrawn image is determined as the target image.

[0175] Among them, the methods for object region detection and image quality evaluation of the redrawn image are the same as those for object region detection and image quality evaluation of the basic image described above, and the present application will not elaborate on this; the value of the second threshold is set according to actual processing needs, and the present application does not make specific limitations on this.

[0176] Specifically, in the case where the second region identification information indicates multiple object regions, in the redrawn image, at least one candidate sample region corresponding to at least one target object region in the basic image can be determined first, and then the image quality evaluation values respectively determined for the at least one candidate sample region, and the region coincidence degree between each candidate sample region and the corresponding target object region can be determined; furthermore, the candidate sample regions with a region coincidence degree higher than the coincidence degree threshold and an image quality evaluation value higher than the second threshold are determined as sample object regions, and the redrawn image containing at least one sample object region is determined as the target image. Among them, the sample object regions meet the following screening conditions: the region coincidence degree with the corresponding target object region is higher than the coincidence degree threshold, and the corresponding image quality evaluation value is higher than the second threshold.

[0177] Among them, the calculation method of the region coincidence degree can specifically be: the ratio of the area of the coincidence region between the sample object region in the target image and the corresponding target object region in the basic image to the area of the union region; it should be understood that the corresponding relationship between the target object region and the candidate sample region can be determined according to the relative positions of the target object region and the candidate sample region in the image.

[0178] For example, after determining the distribution of the detection frames of each object region in the basic image and the detection frame corresponding to the target object region, a detection frame with the same relative position in the redrawn image can be determined as the detection frame of the corresponding candidate object region according to the relative positions of the detection frames, so as to determine the candidate object region corresponding to the target object region.

[0179] In some feasible implementation manners of the present application, in the case of pre-storing each basic image in the negative sample candidate set and each redrawn image in the positive sample candidate set, the processing device can respectively obtain the candidate sample images stored corresponding to the same candidate prompt text in the negative sample candidate set and the positive sample candidate set, and determine the region coincidence degree of the sample object region based on the corresponding first region information and second region information, and determine whether a training sample can be generated based on the current candidate sample image based on the relationship between the region coincidence degree and the preset coincidence degree threshold.

[0180] In some feasible embodiments, when, for each candidate sample region where the image quality evaluation value is higher than the second threshold, the corresponding determined region coincidence degree is higher than the coincidence degree threshold, that is, when it is determined that at least one candidate sample region can be used as the sample object region respectively, training samples can be constructed based on the current base image and the target image; or, in other feasible implementation manners, when, among the candidate sample regions where the image quality evaluation value is determined to be higher than the second threshold, at least one corresponding region coincidence degree exceeds the coincidence degree threshold, it can be determined that there is at least one sample object region, so training samples can be constructed based on the current base image and the target image.

[0181] For example, referring to Figure 7 As shown, it is a schematic diagram of the process of determining the region coincidence degree in an embodiment of the present application. Taking a small face as the target type object as an example, based on the first region information of a base image, the position of the detection frame for indicating the face region in the base image can be determined. Similarly, for the redrawn image obtained by redrawing the base image, based on the corresponding second region information, the position of the detection frame for indicating the face region in the redrawn image can be determined; furthermore, for the target object region in the base image and the corresponding candidate object region in the redrawn image, the coincidence region and the union region can be determined, and then by calculating the ratio between the area of the coincidence region and the area of the union region, the region coincidence degree can be obtained.

[0182] For another example, assume that three object regions, namely object regions 1-3, are identified for a candidate image, and through detection, it is determined that object regions 1 and 2 can be determined as target object regions, then the candidate image can be determined as the base image; furthermore, object regions 1 and 2 in the base image are redrawn to obtain a redrawn image; then, through object region detection of the redrawn image, corresponding to object regions 1 and 2 in the base image, two corresponding candidate sample regions, namely object regions 1' and 2', are determined in the redrawn image; furthermore, after respectively performing image quality evaluation, assume that it is determined that the image quality evaluation value of object region 2' is greater than the second threshold, and the region coincidence degree between object region 2' and object region 2 is higher than the coincidence degree threshold, then the redrawn image can be determined as the target image.

[0183] For yet another example, referring to Figure 8 As shown, it is a schematic diagram of the process of screening and determining the target image in an embodiment of the present application. According to the appendix Figure 8As can be seen from the schematic content, taking the small face as an example of the target type object, when there is only one identified object area, when determining whether the redrawn image constructed can be used as the target image containing high-quality face content, the processing device determines a target object area in the base image. Moreover, the processing device performs face area detection on the redrawn image obtained by redrawing the base image, and can determine the face area indicated by the second area information in the candidate image, that is, the candidate object area. Furthermore, for the area content of the candidate face area, image quality evaluation is performed, and when it is determined that the image quality evaluation value is lower than the second threshold and the area overlap degree with the corresponding target object area is higher than the overlap degree threshold, the redrawn image is determined as the target image.

[0184] In this way, based on the area overlap degree and the image quality evaluation value, the redrawn image containing at least one sample object area can be screened as the target image, so that the image with high-quality target type objects can be screened out, and for each at least one sample object area that may be included in the target image, the object entity of the target type object in the target image may be at least one.

[0185] Particularly, in a feasible embodiment, the processing device may also determine the redrawn image as the target image when it is determined that each object area in the redrawn image meets the screening conditions of the sample object area.

[0186] Step 303: The processing device performs the following operations for each target image: construct a mask image corresponding to the sample object area, and construct a training sample in combination with the corresponding base image and the candidate prompt text.

[0187] In a feasible embodiment of the present application, when the processing device constructs a mask image corresponding to the sample object area, it may obtain the extended area information based on the object area information of the sample object area for training and the corresponding set expansion coefficient; then, in the constructed image with the same image size, set the pixel values of each pixel point in the extended area indicated by the extended area information to 1, and set the pixel values of each pixel point in other areas to 0 to obtain the mask image.

[0188] Among them, the value of the expansion coefficient is set according to actual processing needs, and the present application does not make specific limitations on this.

[0189] When there are at least one sample object areas in the target image, for actual processing needs, a sample object area for training in a training sample can be determined, and then the sample object area for training can be expanded to obtain an extended area, and the extended area can be understood as the re-determined sample object area.

[0190] In some feasible implementation manners of the present application, each sample object region can be regarded as an independent training basis. Then, the processing device can perform region expansion on each sample object region in the target image respectively, and mask each expanded region (i.e., the re-determined sample object region), so that multiple masked images can be constructed corresponding to the target image.

[0191] In some other feasible implementation manners of the present application, the whole of at least one sample object region can be regarded as a training basis. Then, the processing device can perform region expansion on at least one sample object region in the target image, and mask the expanded region obtained as a whole, so that one masked image can be constructed corresponding to one target image.

[0192] For example, taking the value of the outer expansion coefficient as 0.25 and only one sample object region included in one target image as an example, when obtaining an expanded region by outer expansion, on the basis of the sample object region in the target image, it can be expanded 0.25 times in each of the four directions of up, down, left, and right, so as to obtain the expanded region information determined for positioning the expanded region in the constructed image; the size of the masked image is the same as that of the base image and the target image.

[0193] In this way, by constructing the masked image, the pixel region where the sample object region serving as the training basis is located can be indicated. Moreover, by optionally expanding the sample object region when constructing the masked image, the content integrity of the target type object can be guaranteed, so as to provide an effective processing basis for the generation optimization of the target type object.

[0194] Optionally, in the feasible manner of constructing the masked image, after determining the sample object region serving as the training basis, the sample object region serving as the training basis in the target image and the target object region correspondingly determined in the base image can be merged to obtain a merged region, and an expanded region can be obtained on the basis of the merged region, and the corresponding masked image can be obtained by masking the expanded region.

[0195] It should be noted that the processing device can indicate the merged region by means of the merged region information, where the merged region information can be jointly determined based on the position information of the sample object region in the target image and the position information of the target object region in the base image.

[0196] For example, assume that when constructing a training sample, in a target image containing 3 object regions, sample object regions 2' and 3' are determined as the sample object regions for training. Moreover, the target object regions corresponding to sample object regions 2' and 3' in the base image are target object regions 2 and 3. Therefore, based on sample object region 2 and target object region 2', the region position corresponding to a merged sub-region can be obtained. Similarly, for target object region 3 and sample object region 3', the region position corresponding to another merged sub-region can be obtained. Furthermore, the whole of the two merged sub-regions can be used as the merged region, and in the constructed image, after masking the merged region, a masked image can be obtained.

[0197] For another example, refer to Figure 9 shown, which is a schematic diagram of the construction process of a possible masked image in an embodiment of the present application. According to the content Figure 9 indicated by the schematic, in the case where the object regions in the base image and the target image are one, in a feasible way of constructing a masked image, the union region indicated by the first region information and the second region information can be first determined as the merged region, and while keeping the central position of the merged region unchanged, it is expanded 0.25 times in each of the up, down, left, and right directions to obtain the extended region information for positioning the extended region. Furthermore, by setting the pixel values in the extended region in the constructed image to 1 and setting the pixel values in other regions to 0, a corresponding masked image can be obtained.

[0198] Continuing to explain in combination with Figure 9 In another feasible way of constructing a masked image, the merged region containing the target object region indicated by the first region information and the sample object region indicated by the second region information can be first determined based on the first region information and the second region information. Furthermore, while keeping the central position of the merged region unchanged, it is expanded 0.25 times in each of the up, down, left, and right directions to obtain the extended region information for positioning the extended region. Furthermore, by setting the pixel values in the extended region in the constructed image to 1 and setting the pixel values in other regions to 0, a corresponding masked image can be obtained.

[0199] Furthermore, when the processing device constructs a corresponding training sample, the following configuration operations can be performed: using the corresponding base image as the negative sample image in the training sample, using the target image as the positive sample image in the training sample that is more preferred relative to the negative sample image, and using the candidate prompt text corresponding to the base image as the sample prompt text in the training sample; then adding a masked image for indicating the sample object region for training basis in the training sample, where the image quality evaluation value within the sample object region in the positive sample image is higher than the second threshold; and the image quality evaluation value within the sample object region in the negative sample image is lower than the first threshold.

[0200] Specifically, when constructing training samples, the base image containing the target type object with poor image quality can be used as the negative sample image in a training sample, and the target image that is redrawn based on this base image and has a higher image quality of the target type object can be used as the positive sample image in this training sample. Moreover, a mask image constructed for the sample object region based on the training basis is added to this training sample. In addition, the candidate prompt text corresponding to the base image is added to this training sample, and finally a complete training sample is obtained.

[0201] Optionally, the processing device can construct a positive sample image set to store the determined positive sample images, and construct a negative sample image set to store the determined negative sample images.

[0202] In a feasible implementation, when the target image contains multiple sample object regions, the training samples constructed corresponding to the target image may be one or more.

[0203] For example, assume that the target image includes sample object region 1 and sample object region 2. Then, when using this target image as the positive sample image, one or two training samples can be constructed. When constructing one training sample, the sample object regions 1 and 2 can be used as the sample object region based on the training basis. Therefore, in the mask image in the training sample, the mask content determined corresponding to the sample object regions 1 and 2 is included. When constructing two training samples, it can be configured that the mask image in one training sample includes the mask content determined corresponding to the sample object region 1, and the mask image in the other training sample includes the mask content determined corresponding to the sample object region 2.

[0204] In this way, the poor object image with the image content of the target type object being damaged can be used as the negative sample image in a training sample, and the corresponding high-quality image can be used as the positive sample image in the same training sample. By generating preferences through annotation between the positive sample image and the negative sample image, the positive and negative sample pairs based on which the direct preference optimization process is carried out can be determined.

[0205] Similarly, the processing device can construct each training sample in combination with the construction method in steps 301 - 303.

[0206] In this way, with the constructed training samples, the training data meeting the requirements of direct preference optimization (DPO) can be obtained, thereby providing an effective optimization basis for the optimization process of the image generation model. Moreover, for the positive sample image and the negative sample image included in a single training sample, it meets the characteristics that in the sample object region based on the training basis, there is one good and one poor face quality, and the detected region overlap degree is higher than the third threshold.

[0207] Step 202: The processing device uses each training sample, takes the noise prediction result of the reference model as the reference information, and performs multiple rounds of iterative training on the basic image generation model to obtain the target model.

[0208] When the processing device obtains each training sample, it can take the noise prediction result determined by the reference model based on the training sample as the reference information, and perform multiple rounds of iterative training on the basic image generation model to obtain the trained target model. Among them, the end condition of the multiple rounds of iterative training can be: the training round reaches a preset round threshold, or the number of times that the model loss value is continuously lower than the first set value reaches the second set value; the values of the round threshold, the first set value, and the second set value are set according to actual processing needs, and the present application does not make specific limitations on this.

[0209] It should be noted that in the embodiments of the present application, the basic image generation model can be a model capable of realizing image generation, such as can be selected from various text-to-image models; in a feasible embodiment, the image generation model is specifically composed of an image encoding network, an image decoding network, a text encoding network, and a main body generation network; in the noise reduction process included in the image generation model, pure noise can be gradually denoised, and there are multiple noise reduction time steps in the noise reduction process. During the processing of each time step, noise prediction can be realized, and the predicted noise can be removed.

[0210] For example, the basic image generation model can be constructed based on the generative model framework (diffusion), such as, specifically, it can be the stable diffusion model.

[0211] Again, for example, referring to Figure 10 shown, which is a possible structural schematic diagram of the image generation model in the embodiments of the present application. According to the content shown in the appendix Figure 10 it can be known that the basic image generation model can specifically be a model constructed based on stable diffusion, and moreover, the model includes an image encoding network, an image decoding network, a main body generation network, and a text encoding network.

[0212] The following takes the processing involved in one round of iterative training process as an example for specific description:

[0213] It should be noted that in the following description, only taking the batch size value as 1 as an example, the relevant processing process is described.

[0214] Step 2021: Based on the read training samples, the processing device obtains the first noise map predicted by the image generation model for the positive sample image and the second noise map predicted by the image generation model for the negative sample image at a randomly sampled target noise reduction time step, and obtains the third noise map predicted by the reference generation model for the positive sample image and the fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step.

[0215] When specifically obtaining the first noise map and the second noise map, the processing device can first obtain the target noise reduction time step randomly sampled within the noise reduction time step range of the noise reduction process; then, using the image generation model, perform image noise reduction based on the sample prompt text and the positive sample image in the read training samples to obtain the first noise map predicted at the target noise reduction time step, and perform image noise reduction based on the sample prompt text and the negative sample image in the read training samples to obtain the second noise map predicted at the target noise reduction time step.

[0216] Specifically, in some feasible embodiments of the present application, the positive sample image and the negative sample image can be input into the image generation model in parallel, and after copying the corresponding sample prompt text, input it into the prompt text input positions respectively configured for the positive sample image and the negative sample image, so that the image generation model can perform parallel processing on the positive sample image and the negative sample image internally.

[0217] It should be noted that the target noise reduction time step is randomly sampled within the noise reduction time step range corresponding to the noise reduction process, and the values of the target noise reduction time steps determined in different rounds of training processes may be different.

[0218] In this way, for the sampled target noise reduction time step, the noise maps predicted by the image generation model for the positive sample image and the negative sample image respectively during the noise reduction process can be obtained.

[0219] When specifically obtaining the third noise map and the fourth noise map, the processing device can use the reference model to perform image noise reduction based on the sample prompt text and the positive sample image to obtain the third noise map predicted at the target noise reduction time step, and perform image noise reduction generation processing based on the sample prompt text and the negative sample image to obtain the fourth noise map predicted at the target noise reduction time step.

[0220] Among them, when obtaining the third noise map and the fourth noise map, the target noise reduction time step used is the same as that when obtaining the first noise map and the second noise map; moreover, when the processing device inputs the read training samples into the reference model, it can be processed in the same input manner as the above image generation model, and the present application will not expand on this; in addition, the reference model can be obtained by copying the parameters of the image generation model before training.

[0221] In this way, for the target noise reduction time step obtained by corresponding sampling, it is possible to obtain the noise maps respectively predicted by the reference model for the positive sample image and the negative sample image during the noise reduction process.

[0222] Step 2022: The processing device adjusts the image generation model based on the object region differences between the first noise map and the third noise map, and the object region differences between the second noise map and the fourth noise map.

[0223] When performing step 2022, the processing device can obtain the first Gaussian map and the second Gaussian map respectively determined for the positive sample image and the negative sample image during the noise addition process; then, corresponding to the sample object region indicated by the mask image, determine the first region difference between the first Gaussian map and the first noise map, and determine the third region difference between the first Gaussian map and the third noise map, as well as determine the second region difference between the second Gaussian map and the second noise map, and determine the fourth region difference between the second Gaussian map and the fourth noise map; afterwards, based on the object region difference determined by the difference result between the first region difference and the third region difference, and the object region difference determined by the difference result between the fourth region difference and the second region difference, adjust the image generation model.

[0224] Specifically, the processing device can calculate the model loss value based on the difference result between the first region difference and the third region difference, and the difference result between the fourth region difference and the second region difference, and then adjust the model parameters of the image generation model based on the calculated model loss value.

[0225] In addition, it should be noted that since the mask image represents the positions of the sample object regions in the positive sample image and the negative sample image, the initially constructed mask image has the same size as the positive sample image and the negative sample image; then, taking the determination of the first region difference between the first Gaussian map and the first noise map based on the mask image as an example, the processing device needs to adjust the image size of the mask image according to the image sizes of the first Gaussian map and the first noise map, and based on the adjusted mask image, respectively determine the Gaussian image content and the noise image content corresponding to the sample object region in the first Gaussian map and the first noise map; then, by calculating the f2 norm between the Gaussian image content and the noise image content, the first region difference is obtained.

[0226] Among them, the method of adjusting the image size of the mask image can be obtained by sampling according to the image sizes of the first Gaussian map and the first noise map.

[0227] In this way, by adjusting the image size of the mask image, the size consistency between the mask image, the first Gaussian map, and the first noise map can be maintained, thereby providing a processing basis for specifically determining the differences in object regions.

[0228] Similarly, the processing device can obtain the second region difference, the third region difference, and the fourth region difference.

[0229] Specifically, when calculating the model loss value, the following formula can be used for calculation:

[0230]

[0231] Among them, L(θ) represents the calculated model loss value; E represents the mathematical expectation; is the positive sample image in a training sample, is the negative sample image in a training sample; D pos is the positive sample image set; D neg is the negative sample image set; t is a target denoising time step in the denoising process, and T is the total number of fixed denoising time steps set; β is the set hyperparameter; ε w is the Gaussian noise determined for the positive sample image, that is, the first Gaussian map, which conforms to the standard normal distribution N(0, I); ε l is the Gaussian noise determined for the negative sample image, that is, the second Gaussian map, which conforms to the standard normal distribution N(0, I); is the first noise map predicted by the image generation model for the positive sample image at the target denoising time step t; is the third noise map predicted by the reference model for the positive sample image at the target denoising time step t; The second noise map predicted by the image generation model for the negative sample image at the target denoising time step t; is the fourth noise map predicted by the reference model for the negative sample image at the target denoising time step t; represents the sampling result of the original data at the denoising time step t during the noise addition process for the positive sample image; represents the sampling result of the original data at the denoising time step t during the noise addition process for the negative sample image; ω(λ t ) is the given hyperparameter; M face is a mask image with the same size as various noise maps; represents the calculation of the f2 norm.

[0232] In this way, by introducing the influence of the masked image in the calculation process of the model loss value, the position where the bias optimization can play a role is restricted to the sample object area where the target type object is located, so that the generation quality of the target type object can be optimized and improved. Moreover, a reference model that does not participate in training is introduced to provide a generation preference direction for the image generation model, so that during the optimization process of the image generation model, the predicted noise obtained by the image generation model based on the positive sample image is closer to the predicted noise obtained by the reference model based on the positive sample image, and the predicted noise obtained by the image generation model based on the negative sample image is farther from the predicted noise obtained by the reference model based on the negative sample image, and the model parameters are adjusted accordingly.

[0233] Similarly, the processing device can perform multiple rounds of iterative training on the image generation model until a preset end condition is met, and obtain the trained target model.

[0234] During the actual test process, the applicant only uses tens of thousands of training samples for the overall training, and training for 10,000 - 20,000 rounds can meet the training needs. On an 8-card A100 machine, it takes about 3 hours to complete the training. It can be seen that the technical solution proposed in this application can train a target image that meets the business generation effect at a very low training cost and low training cost, and can avoid consuming a high inference and deployment cost during the deployment process.

[0235] Furthermore, when performing business processing with the aid of the target model, the processing device can flexibly use the model network in the target model according to the image generation basis for image generation, so as to generate images that meet the requirements.

[0236] Specifically, when the target model includes a text encoding network, a subject generation network, an image encoding network, and an image decoding network, when performing text-to-image processing, the processing device obtains the target prompt text; then uses the text encoding network to encode the target prompt text into text features in the latent space, and uses the subject generation network to perform multiple denoising processes based on the randomly generated latent space noise with the text features as the guiding conditions to obtain latent space features; afterwards, uses the image decoding network to decode the latent space features to obtain the image to be fed back.

[0237] Among them, when the image generation basis does not include the target prompt image, the processing device can construct the latent space noise by itself for denoising processing.

[0238] In this way, it is possible to perform text-to-image processing according to the target prompt text provided by the target object, and obtain the image to be fed back that meets the processing needs of the target object and has a high generation quality of the target type object.

[0239] Similarly, when the processing device obtains the image generation basis including the target prompt image and the target prompt text, it can jointly process according to the text encoding network, the subject generation network, the image encoding network, and the image decoding network, and use the image encoding network to encode the target prompt image and add noise to the image encoding result in the latent space to obtain the latent space noise.

[0240] Specifically, when the obtained image generation basis only includes the target prompt image, it can jointly process according to the subject generation network, the image encoding network, and the image decoding network, and finally generate the image to be fed back.

[0241] Next, taking the small face as an example of the target type object, the processing involved in one round of training process will be schematically described with reference to the accompanying drawings:

[0242] Refer to the attached Figure 11 which is a schematic diagram of the processing of one round of iterative training in the embodiment of the present application. Next, with reference to the attached Figure 11 , specific description will be made. According to the content shown in the attached Figure 11 , taking the processing process executed based on one training sample as an example, the processing device inputs the positive sample image and the negative sample image in the training sample into the image encoding network in the image generation model in parallel, and inputs the candidate prompt text into the text encoding network. And for the pre-sampled target noise reduction time step, after determining the processing result for the target noise reduction time step in the subject generation network, directly obtain the first noise reduction map predicted based on the positive sample image and the second noise reduction map predicted based on the negative sample image, where the first noise reduction map and the second noise reduction map can be obtained before the image generated by the image generation model; or, it can be obtained after the image generated by the image generation model. The present application does not make specific limitations on this. Furthermore, the processing device calculates the model loss value based on the first noise reduction map, the second noise reduction map, the third noise reduction map and the fourth noise reduction map predicted by the reference model in the same way, and the mask image in the training sample; and then adjusts the model parameters of the image generation model based on the model loss value.

[0243] In summary, the present application proposes a method for improving the quality of face generation based on direct preference optimization and related data construction, which enables the optimization of the problem of poor generation effect in the small face area in the field of text-to-image; moreover, by generally introducing the direct preference optimization algorithm and based on the generation preference of the face area, it is possible to realize the generation optimization of the small face area during the model training process.

[0244] Based on the same inventive concept, refer to Figure 12As shown, it is a schematic logical structure diagram of an optimization device for an image generation model in an embodiment of the present application. The optimization device 1200 for the image generation model includes an acquisition unit 1201 and a training unit 1202. Among them,

[0245] The acquisition unit 1201 is used to acquire each training sample; each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image used to indicate the sample object regions in the positive sample image and the negative sample image;

[0246] The training unit 1202 is used to use each training sample and take the noise prediction result of the reference model as reference information to perform multiple rounds of iterative training on the basic image generation model to obtain a target model; the reference model is obtained by copying the image generation model before training; among them, in one round of iterative training, the following operations are performed:

[0247] Based on the read training sample, obtain the first noise map predicted by the image generation model for the positive sample image and the second noise map predicted by the image generation model for the negative sample image at a randomly sampled target noise reduction time step, and obtain the third noise map predicted by the reference generation model for the positive sample image and the fourth noise map predicted by the reference generation model for the negative sample image at the target noise reduction time step;

[0248] Based on the object region differences between the first noise map and the third noise map, and the object region differences between the second noise map and the fourth noise map, adjust the image generation model.

[0249] Optionally, each training sample is constructed by the acquisition unit 1201 in the following manner:

[0250] Obtain each candidate image containing an object of the target type generated based on each candidate prompt text;

[0251] Among each candidate image, screen out each basic image that contains at least one target object region, and perform redrawing on the target object region for each basic image respectively to obtain each target image that contains at least one sample object region; among them, the image quality evaluation value in each target object region is lower than the first threshold; the image quality evaluation value in each sample object region is higher than the second threshold, and the region coincidence degree with a corresponding target object region is higher than the coincidence degree threshold;

[0252] For each target image, perform the following operations: construct a mask image corresponding to the sample object region, and combine the corresponding basic image and the candidate prompt text to construct a training sample.

[0253] Optionally, when screening out each basic image that contains at least one target object region among each candidate image, the acquisition unit 1201 is used to:

[0254] For each candidate image, the following operations are performed separately:

[0255] Perform object area detection of the target type object for the candidate image to obtain first area information, and perform image quality detection on the area content of each object area indicated by the first area information respectively to obtain the image quality evaluation value corresponding to each object area;

[0256] When it is determined that there are target object areas in each object area whose image quality evaluation value is higher than the first threshold, the candidate image is determined as the base image.

[0257] Optionally, when redrawing the target object area for each base image, the obtaining unit 1201 is used for:

[0258] For each base image, the following operations are performed separately:

[0259] Crop at least one target object area in the base image respectively to obtain at least one sub-image;

[0260] Replace the corresponding target object area in the base image with at least one object image generated based on at least one sub-image and the preset optimization prompt text respectively to obtain the redrawn image.

[0261] Optionally, when obtaining each target image that respectively contains at least one sample object area after screening, the obtaining unit 1201 is used for:

[0262] For each redrawn image obtained by redrawing, the following operations are performed separately:

[0263] Perform object area detection of the target type object on the redrawn image to obtain second area information, and determine at least one candidate sample area corresponding to at least one target object area in the corresponding base image according to the area content indicated by the second area identification information;

[0264] For at least one candidate sample area, determine the corresponding image quality evaluation value and the area coincidence degree with the corresponding target object area respectively;

[0265] When it is determined that there are sample object areas in at least one candidate sample area whose image quality evaluation value is higher than the second threshold and the area coincidence degree is higher than the coincidence degree threshold, the redrawn image is determined as the target image.

[0266] Optionally, when constructing the mask image corresponding to the sample object area, the obtaining unit 1201 is used for:

[0267] Based on the object area information of the sample object area according to the training and the corresponding set expansion coefficient, obtain the extended area information;

[0268] A mask image is obtained by setting the pixel values of each pixel point in the expansion area indicated by the expansion area information to 1 and the pixel values of each pixel point in other areas to 0 in a constructed image of the same image size.

[0269] Optionally, when constructing a training sample by combining a corresponding base image and a candidate prompt text, the acquisition unit 1201 is used for:

[0270] Taking the corresponding base image as the negative sample image in the training sample, taking the target image as the positive sample image that is more preferred than the negative sample image in the training sample, and taking the candidate prompt text corresponding to the base image as the sample prompt text in the training sample;

[0271] In the training sample, a mask image for indicating the sample object area of the training basis is added. The image quality evaluation value in the sample object area of the positive sample image is higher than the second threshold; the image quality evaluation value in the sample object area of the negative sample image is lower than the first threshold.

[0272] Optionally, when obtaining the first noise map predicted for the positive sample image and the second noise map predicted for the negative sample image at the target noise reduction time step randomly sampled by the image generation model, the training unit 1202 is used for:

[0273] Obtaining a target noise reduction time step randomly sampled within the noise reduction time step range of the noise reduction process;

[0274] Using the image generation model, performing image noise reduction based on the sample prompt text and the positive sample image read, obtaining the first noise map predicted at the target noise reduction time step, and performing image noise reduction based on the sample prompt text and the negative sample image read, obtaining the second noise map predicted at the target noise reduction time step.

[0275] Optionally, when obtaining the third noise map predicted for the positive sample image and the fourth noise map predicted for the negative sample image at the target noise reduction time step by the reference generation model, the training unit 1202 is used for:

[0276] Using the reference model, performing image noise reduction based on the sample prompt text and the positive sample image, obtaining the third noise map predicted at the target noise reduction time step, and performing image noise reduction generation processing based on the sample prompt text and the negative sample image, obtaining the fourth noise map predicted at the target noise reduction time step.

[0277] Optionally, when adjusting the image generation model based on the object area difference between the first noise map and the third noise map, and the object area difference between the second noise map and the fourth noise map, the training unit 1202 is used for:

[0278] Obtain a first Gaussian map and a second Gaussian map respectively determined for the positive sample image and the negative sample image during the noise addition process;

[0279] For the sample object region indicated by the corresponding mask image, determine a first regional difference between the first Gaussian map and the first noise map, and determine a third regional difference between the first Gaussian map and the third noise map, and determine a second regional difference between the second Gaussian map and the second noise map, and determine a fourth regional difference between the second Gaussian map and the fourth noise map;

[0280] Based on the object region difference determined by the difference result between the first regional difference and the third regional difference, and the object region difference determined by the difference result between the fourth regional difference and the second regional difference, adjust the image generation model.

[0281] Optionally, when determining the first regional difference between the first Gaussian map and the first noise map based on the mask image, the training unit 1202 is configured to:

[0282] According to the image sizes of the first Gaussian map and the first noise map, adjust the image size of the mask image, and based on the adjusted mask image, respectively determine the Gaussian image content and the noise image content corresponding to the sample object region in the first Gaussian map and the first noise map;

[0283] Obtain the first regional difference by calculating the f2 norm between the Gaussian image content and the noise image content.

[0284] Optionally, the target model includes a text encoding network, a main body generation network, an image encoding network, and an image decoding network; after obtaining the target model, the device further includes a processing unit, and the processing unit 1203 is configured to:

[0285] Obtain the target prompt text;

[0286] Use the text encoding network to encode the target prompt text into text features in the latent space, and use the main body generation network to perform multiple denoising processes based on the randomly generated latent space noise with the text features as the guiding conditions to obtain latent space features;

[0287] Use the image decoding network to decode the latent space features to obtain the image to be fed back.

[0288] For the convenience of description, the above parts are divided into respective modules (or units) according to functions and described separately. Of course, when implementing the present application, the functions of the respective modules (or units) can be implemented in the same or multiple software or hardware.

[0289] After introducing the optimization method and device of the image generation model according to the exemplary embodiments of the present application, next, an electronic device according to another exemplary embodiment of the present application will be introduced.

[0290] Those skilled in the art can understand that various aspects of the present application can be implemented as a system, a method, or a program product. Therefore, various aspects of the present application can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module", or "system" here.

[0291] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. Refer to Figure 13 As shown, it is a schematic diagram of the hardware composition structure of an electronic device applying the embodiment of the present application. In one embodiment, the electronic device can be Figure 1 the processing device 120 shown. In this embodiment, the structure of the electronic device can be as Figure 13 shown, including a memory 1301, a communication module 1303, and one or more processors 1302.

[0292] The memory 1301 is used to store the computer program executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and programs required to run the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0293] The memory 1301 can be a volatile memory, such as a random-access memory (RAM); the memory 1301 can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or the memory 1301 is any other medium that can be used to carry or store the desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1301 can be a combination of the above memories.

[0294] The processor 1302 may include one or more central processing units (CPUs) or be a digital processing unit, etc. The processor 1302 is used to implement the above-mentioned optimization method of the image generation model when calling the computer program stored in the memory 1301.

[0295] The communication module 1303 is used to communicate with the client device and the server.

[0296] In the embodiments of the present application, the specific connection medium between the above-mentioned memory 1301, communication module 1303 and processor 1302 is not limited. In the embodiments of the present application Figure 13 it is connected between the memory 1301 and the processor 1302 through a bus 1304, and the bus 1304 is Figure 13 described in thick lines. The connection manners between other components are only for illustrative purposes and are not limiting. The bus 1304 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, Figure 13 only one thick line is used to describe it in the figure, but it does not describe that there is only one bus or one type of bus.

[0297] The memory 1301 stores a computer storage medium, and the computer storage medium stores computer-executable instructions, which are used to implement the optimization method of the image generation model in the embodiments of the present application. The processor 1302 is used to execute the above-mentioned optimization of the image generation model, as Figure 2 shown.

[0298] In another embodiment, the electronic device may also be other electronic devices. Refer to Figure 14 shown, which is a schematic diagram of the hardware composition structure of another electronic device applying the embodiments of the present application. The electronic device may specifically be Figure 1 the client device 110 shown. In this embodiment, the structure of the electronic device may be as Figure 14 shown, including components such as a communication component 1410, a memory 1420, a display unit 1430, a camera 1440, a sensor 1450, an audio circuit 1460, a Bluetooth module 1470, and a processor 1480.

[0299] The communication component 1410 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module. The WiFi module belongs to short-range wireless transmission technology, and the electronic device can help users send and receive information through the WiFi module.

[0300] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions and data processing of the client device 110 by running the software programs or data stored in the memory 1420. In the present application, the memory 1420 can store an operating system and various application programs, and can also store computer programs related to implementing the optimization method of the prompt image generation model in the embodiments of the present application.

[0301] The display unit 1430 can also be used to display information input by the user or information provided to the user, as well as a graphical user interface (GUI) of various menus of the client device 110. Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the client device 110. The display unit 1430 can be used to display pages and the like.

[0302] The display unit 1430 can also be used to receive input numerical or character information and generate signal inputs related to the user settings and function control of the client device 110. Specifically, the display unit 1430 may include a touch screen 1431 disposed on the front of the client device 110, which can collect touch operations of the user thereon or nearby.

[0303] Among them, the touch screen 1431 can cover the display screen 1432, or the touch screen 1431 and the display screen 1432 can be integrated to implement the input and output functions of the client device 110. After integration, it can be simply referred to as a touch display screen. In this application, the display unit 1430 can display application programs and corresponding operation steps.

[0304] The camera 1440 can be used to capture static images. The user can post comments on the images captured by the camera 1440 through an application. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1480 to convert it into a digital image signal.

[0305] The client device may further include at least one sensor 1450, such as an acceleration sensor 1451, a distance sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The client device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0306] The audio circuit 1460, speaker 1461, and microphone 1462 can provide an audio interface between the user and the client device 110. The audio circuit 1460 can transmit the electrical signal converted from the received audio data to the speaker 1461, and the speaker 1461 converts it into a sound signal for output. On the other hand, the microphone 1462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1460 and then converted into audio data. The audio data is then output to the communication component 1410 for sending to, for example, another client device 110, or output to the memory 1420 for further processing.

[0307] The Bluetooth module 1470 is used to interact with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol.

[0308] The processor 1480 is the control center of the client device, connecting various parts of the entire terminal using various interfaces and lines. By running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420, it performs various functions of the client device and processes data. In some embodiments, the processor 1480 may include at least one processing unit; the processor 1480 may also integrate an application processor and a baseband processor. In this application, the processor 1480 can run the operating system, application programs, user interface display and touch response, as well as methods related to the optimization of the image generation model of the embodiments of this application. Additionally, the processor 1480 is coupled to the display unit 1430.

[0309] In some possible implementation manners, various aspects of the method for optimizing the image generation model provided in this application can also be implemented in the form of a program product, which includes a computer program. When the program product runs on an electronic device, the computer program is used to cause the electronic device to execute the steps in the method for optimizing the image generation model according to various exemplary embodiments of this application described above in this specification. For example, the electronic device can execute the steps as Figure 2 shown.

[0310] The program product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0311] The program product of the embodiments of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and may be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with a command execution system, apparatus, or device.

[0312] The readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a readable computer program is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0313] The computer program contained on the readable medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0314] The computer program for performing the operations of the present application may be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The computer program may be executed entirely on the user's electronic device, partially on the user's electronic device, executed as a stand-alone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the user's electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (for example, by connecting through the Internet service provider via the Internet).

[0315] It should be noted that although several units or subunits of the apparatus are mentioned in the foregoing detailed description, such a division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more units described above may be embodied in one unit. Conversely, the features and functions of one unit described above may be further divided and embodied by multiple units.

[0316] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution.

[0317] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) that contain computer-usable computer programs.

[0318] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0319] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0320] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A method for optimizing an image generation model, characterized in that: include: Obtain each training sample; Each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image for indicating a sample object area in the positive sample image and the negative sample image; Using the training samples and the noise prediction results of the reference model as reference information, the basic image generation model is trained for multiple rounds of iterations to obtain a target model; the reference model is obtained by copying the image generation model before training; wherein, in one round of iterative training, the following operations are performed: Based on the read training samples, obtaining a first noise map predicted by the image generation model at a randomly sampled target denoising time step, corresponding to a positive sample image, and a second noise map predicted by the image generation model at a randomly sampled target denoising time step, and obtaining a third noise map predicted by the reference generation model at the target denoising time step, corresponding to the positive sample image, and a fourth noise map predicted by the reference generation model at the target denoising time step; The image generation model is adjusted based on the object area difference between the first noise map and the third noise map, and the object area difference between the second noise map and the fourth noise map.

2. The method according to claim 1, characterized in that The training samples are constructed in the following manner: Obtain candidate images generated based on the candidate prompt texts and containing the target type object; In each of the candidate images, basic images each containing at least one target object region are screened out, and the target object region is redrawn for each basic image, to obtain screened target images each containing at least one sample object region; wherein the image quality evaluation value in each target object region is lower than a first threshold; the image quality evaluation value in each sample object region is higher than a second threshold, and the region overlap with a corresponding target object region is higher than an overlap threshold; For each target image, the following operations are performed: a mask image corresponding to the sample object area is constructed, and a training sample is constructed by combining the corresponding base image and the candidate prompt text.

3. The method according to claim 2, characterized in that The step of selecting basic images that each contain at least one target object region from the candidate images comprises: For each candidate image, perform the following operations: Performing object region detection of the target type object on the candidate image to obtain first region information, and performing image quality detection on the region content of each object region indicated by the first region information to obtain image quality assessment values ​​corresponding to each object region; When it is determined that the object regions include a target object region whose image quality evaluation value is higher than a first threshold, the candidate image is determined as a basic image.

4. The method according to claim 2, characterized in that The redrawing of the target object area for each basic image includes: For each base image, perform the following operations: Cut out at least one target object region in the base image respectively to obtain at least one sub-image; At least one object image generated based on the at least one sub-image and the preset optimization prompt text is respectively used to replace the corresponding target object area in the basic image to obtain a redrawn image.

5. The method according to claim 2, characterized in that The screened target images each containing at least one sample object area include: For each redrawn image, perform the following operations: Performing object region detection on the target type object for the redrawn image to obtain second region information, and determining at least one candidate sample region corresponding to at least one target object region in the corresponding basic image according to the region content indicated by the second region identification information; For the at least one candidate sample area, respectively determine a corresponding image quality assessment value and a region overlap degree between the candidate sample area and the corresponding target object area; When it is determined that there is a sample object region in the at least one candidate sample region whose image quality evaluation value is higher than a second threshold and whose region coincidence is higher than a coincidence threshold, the redrawn image is determined as a target image.

6. The method according to claim 2, characterized in that The step of constructing a mask image corresponding to the sample object area includes: Based on the object area information of the sample object area of ​​the training basis and the corresponding set external expansion coefficient, the expanded area information is obtained; The mask image is obtained by setting the pixel value of each pixel in the extended area indicated by the extended area information to 1 and setting the pixel value of each pixel in other areas to 0 in the constructed image of the same image size.

7. The method according to claim 2, characterized in that The step of combining the corresponding basic image and the candidate prompt text to construct a training sample includes: Using the corresponding basic image as a negative sample image in the training sample, using the target image as a positive sample image in the training sample that is more preferred than the negative sample image, and using the candidate prompt text corresponding to the basic image as a sample prompt text in the training sample; In the training sample, a mask image is added to indicate a sample object area serving as a training basis, and an image quality assessment value within the sample object area in the positive sample image is higher than a second threshold; and an image quality assessment value within the sample object area in the negative sample image is lower than a first threshold.

8. The method according to any one of claims 1 to 7, characterized in that: The obtaining of a first noise map corresponding to a positive sample image prediction and a second noise map corresponding to a negative sample image prediction at a randomly sampled target denoising time step of the image generation model comprises: Obtain the target denoising time step obtained by random sampling within the denoising time step range of the denoising process; The image generation model is used to perform image denoising based on the sample prompt text and positive sample images in the read training samples to obtain a first noise map predicted at the target denoising time step, and image denoising is performed based on the sample prompt text and negative sample images in the read training samples to obtain a second noise map predicted at the target denoising time step.

9. The method according to claim 8, characterized in that Obtaining a third noise map predicted by the reference model at the target denoising time step corresponding to the positive sample image and a fourth noise map predicted corresponding to the negative sample image, comprising: The reference model is used to perform image denoising based on the sample prompt text and the positive sample image to obtain a third noise map predicted at the target denoising time step, and image denoising generation processing is performed based on the sample prompt text and the negative sample image to obtain a fourth noise map predicted at the target denoising time step.

10. The method according to any one of claims 1 to 7, characterized in that: The adjusting the image generation model based on the difference of the object regions in the first noise map and the third noise map, and the difference of the object regions in the second noise map and the fourth noise map, comprises: Acquire a first Gaussian graph and a second Gaussian graph respectively determined for the positive sample image and the negative sample image during the noise addition process; Corresponding to the sample object area indicated by the mask image, determining a first regional difference between the first Gaussian map and the first noise map, determining a third regional difference between the first Gaussian map and the third noise map, determining a second regional difference between the second Gaussian map and the second noise map, and determining a fourth regional difference between the second Gaussian map and the fourth noise map; The image generation model is adjusted based on the object region difference determined by the difference result between the first region difference and the third region difference and the object region difference determined by the difference result between the fourth region difference and the second region difference.

11. The method according to claim 10, characterized in that The determining, based on the mask image, a first regional difference between the first Gaussian image and the first noise image comprises: Adjusting the image size of the mask image according to the image sizes of the first Gaussian image and the first noise image, and determining the Gaussian image content and the noise image content corresponding to the sample object area in the first Gaussian image and the first noise image respectively based on the adjusted mask image; The first region difference is obtained by calculating the f2 norm between the Gaussian image content and the noise image content.

12. The method according to any one of claims 1 to 7, characterized in that: The target model includes a text encoding network, a subject generation network, an image encoding network, and an image decoding network; after obtaining the target model, the method further includes: Get the target prompt text; The target prompt text is encoded into a text feature of a latent space by using the text encoding network, and the subject generation network is used to perform multiple noise reduction processes based on randomly generated latent space noise with the text feature as a guide condition to obtain latent space features; The image decoding network is used to decode the image to be fed back based on the latent space features.

13. An optimization device for an image generation model, characterized in that: include: An acquisition unit, used for acquiring each training sample; Each training sample includes: a sample prompt text, a positive sample image and a negative sample image constructed based on the sample prompt text and annotated with a preference relationship, and a mask image for indicating a sample object area in the positive sample image and the negative sample image; A training unit is used to use the training samples and the noise prediction results of the reference model as reference information to perform multiple rounds of iterative training on the basic image generation model to obtain a target model; the reference model is obtained by copying the image generation model before training; wherein, in one round of iterative training, the following operations are performed: Based on the read training samples, obtaining a first noise map predicted by the image generation model at a randomly sampled target denoising time step, corresponding to a positive sample image, and a second noise map predicted by the image generation model at a randomly sampled target denoising time step, and obtaining a third noise map predicted by the reference generation model at the target denoising time step, corresponding to the positive sample image, and a fourth noise map predicted by the reference generation model at the target denoising time step; The image generation model is adjusted based on differences in object regions between the first noise map and the third noise map, and differences in object regions between the second noise map and the fourth noise map.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images