Image generation model construction method
By constructing an image generation model containing common features and differentiated feature learning modules, the problem of difficulty in generating category images in the prior art is solved, and higher image authenticity and diversity are achieved.
Patent Information
- Application Number
- CN202411401149.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-02-18
AI Technical Summary
Existing personalized image generation methods are difficult to accurately generate images of a certain category, especially when the diversity of reference images increases, it is difficult to learn complex distributions, resulting in impairment of the authenticity and diversity of the generated samples.
A method for building an image generation model is proposed. By obtaining the text and images of the preset training set, an initial image generation model is constructed, including a pre-trained image generation model, a first feature learning module and a second feature learning module. These two feature learning modules are used to extract common and differential features of the image from the pre-trained image generation model, and update model parameters through multiple iterative training to ensure that the generated image contains discriminant features and is diverse.
The higher authenticity and diversity of generated images are achieved, especially in low data downstream tasks, which can accurately generate images of a certain category, avoiding the decline in the generation quality of existing methods when diversity increases.
Smart Images
Figure CN119478201A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of digital image processing, and more specifically, to a method for constructing an image generation model. Background Art
[0002] The advent of diffusion models, such as Stable Diffusion and DALL-E, has brought the possibility of generating realistic and creative images to a new level. Other variants of these large-scale text-to-image generation models, such as image editing, image stylization, etc., have found widespread applications in fields such as art and design. In a variety of scenarios, achieving controllable image content editing and generation relies on personalized image generation as the first step. The personalized image generation task refers to generating images that contain specific individuals or meet specific personalized preferences. Existing personalized image generation methods mainly utilize a given set of reference images to fine-tune the parameters of the network or text embedding. This fine-tuning process enables the network to learn to generate specific content in the reference image.
[0003] Existing technologies usually fine-tune the parameters of a pre-trained generative model based on a small number of given reference images to build an image generation model that can generate images containing a specific individual. In image generation tasks such as art design and certain low-data downstream tasks (such as small sample image classification and defect detection), it is crucial for the model to generate images of a certain category rather than individual images. Image samples of the same category usually contain some common features, but compared to individual images, category image samples also contain more complex variations. However, existing personalized image generation methods cannot accurately generate images of a certain category.
[0004] Some existing methods directly fine-tune the weights of the network to learn the reference data distribution. As the diversity of reference images increases, these methods find it difficult to accurately learn complex distributions, which greatly compromises the authenticity and diversity of the generated images. Another type of personalized method uses the prior knowledge contained in the pre-trained model to learn text embeddings corresponding to specific instances. When samples in the reference image are rare in the pre-training dataset or their diversity increases, it is difficult for such methods to extract accurate prior information from the pre-trained model to represent these samples. DreamDistribution is the only existing personalized image generation method optimized specifically for generating classified data. It learns more diverse data distributions by optimizing the distribution of text embeddings. It has similar limitations to the second type of method, namely, it is constrained by the prior information contained in the pre-trained model. In addition, due to the defects in the architectural design of DreamDistribution, it cannot use text to control image generation after training.
[0005] In summary, existing personalization methods mainly focus on fine-tuning pre-trained generative models to generate images containing specific individuals based on a small number of given reference images. However, these methods have difficulty generating images of a certain category when dealing with low-data downstream tasks. In addition, as the diversity of reference images increases, existing methods find it difficult to accurately learn complex distributions, which greatly damages the authenticity and diversity of generated samples. Another class of methods learns text embeddings corresponding to specific instances, leveraging prior knowledge in pre-trained models, but these methods also have difficulty extracting accurate prior information when reference images are rare or their diversity increases in the pre-training dataset.
[0006] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention
[0007] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for constructing an image generation model.
[0008] The objective of the present invention is achieved through the following technical solutions:
[0009] According to a first aspect of the present invention, a method for constructing an image generation model is proposed, which is used to construct a target image generation model. The method comprises: step S1, obtaining a preset training set, wherein the preset training set comprises one or more texts describing an image and a corresponding plurality of images, wherein the plurality of images have some common features with the target image; step S2, constructing an initial image generation model, wherein the initial image generation model comprises a pre-trained image generation model, a first feature learning module, and a second feature learning module, wherein the pre-trained image generation model comprises an encoding module and a decoding module, wherein: the encoding module comprises a plurality of network layers, which are used to extract features of the image and transmit them layer by layer in the network layer; The first feature learning module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the common features of the image and construct the common feature vector based on this; the second feature learning module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the differential features of the image and construct the differential feature vector based on this; the decoding module is used to decode and generate multiple target images according to the common feature vector, the differential feature vector and the basic feature vector; step S3, use the preset training set to perform multiple rounds of iterative training on the initial image generation model until convergence to obtain the target image generation model.
[0010] Preferably, the step S3 includes: step S31, constructing a common feature queue and a difference feature queue, wherein the common feature queue is used to save the latest preset number of common feature vectors generated in the first feature learning module, and the difference feature queue is used to save the latest preset number of difference feature vectors generated in the second feature learning module; step S32, using a preset training set to perform multiple rounds of iterative training on the initial image generation model, and during the training process, using a preset loss function to update the image generation model based on the constructed common feature vectors, difference feature vectors, common feature queues, and difference feature queues.
[0011] Preferably, in step S32, each round of iteration includes: a first stage, including using a preset training set to perform a first retraining and a second retraining on the image generation model updated in the previous round in sequence, wherein, during the first retraining, a first feature learning module is used to obtain common features of images in the preset training set to construct a common feature vector, and based on the constructed common feature vector and the common feature queue, a first preset loss function is used to update the first feature learning module to obtain the image generation model after the first retraining, and the common feature queue is updated based on the constructed common feature vector; during the second retraining, the image generation model after the first retraining is trained using the preset training set and a second preset loss function is used to update the first feature learning module and the second feature learning module. The second stage includes using a preset training set to train the image generation model trained in the first stage for a preset number of times, and each training is performed in turn with a third training and a fourth training, wherein, during the third training, the second feature learning module is used to obtain the differential features of the images in the preset training set to construct a differential feature vector, and based on the constructed differential feature vector and the differential feature queue, the third preset loss function is used to update the second feature learning module to obtain the image generation model after the third training, and the differential feature queue is updated based on the constructed differential feature vector; during the fourth training, the image generation model after the third training is trained using the preset training set and the second preset loss function is used to update the first feature learning module and the second feature learning module.
[0012] Preferably, the first preset loss function is configured as:
[0013]
[0014] in, represents the context loss, Represents the number of common feature vectors stored in the common feature queue, represents the number of channels of the pre-trained image generation model, represents the mean calculation function, represents the covariance calculation function, Represents the common feature vector currently traversed, Represents the i-th common feature vector in the common feature queue.
[0015] Preferably, the second preset loss function is configured as:
[0016]
[0017] in, represents the potential diffusion model loss, represents the time step, represents the initial latent variable, express The hidden variables of the time step, represents the conditional vector, represents noise, Represents the parameters of the image generation model, including the parameters of the pre-trained image generation model, the parameters of the first feature learning module, and the parameters of the second feature learning module. Represents the prediction noise of the pre-trained image generation model.
[0018] Preferably, the third preset loss function is configured as:
[0019]
[0020] in, represents the difference loss, Represents the number of difference feature vectors stored in the difference feature queue, represents the difference feature vector of the current traversal, represents the i-th difference feature vector in the difference feature queue, Represents the preset cosine threshold, represents a function that calculates the absolute value of the cosine of the angle between two vectors. Represents the cosine collocation function of two vectors.
[0021] Preferably, the common feature queue and the difference feature queue are updated in the following manner: the common feature queue is updated in the following manner: when the number of common feature vectors in the common feature queue does not reach the preset capacity, the constructed common feature vector is directly inserted into the tail of the common feature queue; when the number of common feature vectors in the common feature queue has not reached the preset capacity, the common feature vector at the head of the common feature queue is popped out and the constructed common feature vector is inserted into the tail of the common feature queue; the difference feature queue is updated in the following manner: when the number of difference feature vectors in the difference feature queue does not reach the preset capacity, the constructed difference feature vector is directly inserted into the tail of the difference feature queue; when the number of difference feature vectors in the difference feature queue has not reached the preset capacity, the difference feature vector at the head of the difference feature queue is popped out and the constructed difference feature vector is inserted into the tail of the difference feature queue.
[0022] Preferably, the neural network architecture of the pre-trained image generation model is configured as U-Net or DiffusionTransformer.
[0023] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. The computer program can be executed by an adjuster to implement the steps of any method described in the first aspect of the present invention.
[0024] According to a second aspect of the present invention, an electronic device is proposed, comprising: one or more adjusters; and a memory, wherein the memory is used to store executable instructions; the one or more adjusters are configured to implement the steps of any one of the methods described in the first aspect of the present invention by executing the executable instructions.
[0025] Compared with the prior art, the advantages of the present invention are:
[0026] The present invention proposes a scheme for constructing an image generation model, constructing a pre-trained image generation model into a target image generation model, utilizing two feature learning modules to extract common features and difference features of images from feature data transmitted between network layers of an encoding module in the pre-trained image generation model to construct common feature vectors and difference feature vectors, and generating a target image model based on the features, while ensuring that the generated image contains discriminative features, utilizing the difference features between images to enhance the diversity of the image, wherein during model training, an asynchronous optimization method is adopted and a plurality of loss functions are combined to update the parameters of the two feature learning modules, which not only realizes the coordinated optimization of the two feature learning modules, but also ensures that the target image generated by the target image generation model has higher authenticity and diversity. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:
[0028] Figure 1 is a flowchart of a method for constructing an image generation model according to an embodiment of the present invention;
[0029] Figure 2 A schematic diagram of the structure of an initial image generation model according to an embodiment of the present invention;
[0030] Figure 3 is a quantitative evaluation result of image quality according to an embodiment of the present invention;
[0031] Figure 4 is an image quality visualization evaluation result according to an embodiment of the present invention;
[0032] Figure 5 is a visual evaluation result of multi-text image quality according to an embodiment of the present invention;
[0033] Figure 6 is a downstream classification task result according to an embodiment of the present invention;
[0034] Figure 7 3D multi-view image quality visualization evaluation result according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0036] As mentioned in the background section, existing personalization methods mainly focus on fine-tuning pre-trained generative models to generate images containing specific individuals based on a small number of given reference images. However, these methods have difficulty generating images of a certain category when dealing with low-data downstream tasks. In addition, as the diversity of reference images increases, existing methods find it difficult to accurately learn complex distributions, which greatly damages the authenticity and diversity of generated samples. Another class of methods learns text embeddings corresponding to specific instances, leveraging prior knowledge in pre-trained models, but these methods also have difficulty extracting accurate prior information when reference images are rare in the pre-training dataset or when their diversity increases.
[0037] To solve the above problems, the present invention proposes an image generation model construction scheme, which is used to construct a pre-trained image generation model into a target image generation model. The scheme uses two feature learning modules to extract common features and difference features of images from feature data transmitted between network layers of an encoding module in the pre-trained image generation model and returns these features to the pre-trained image generation model, thereby ensuring that the generated image contains discriminative features, and at the same time, utilizing the difference features between images to enhance the diversity of the image. During model training, the parameters of the two feature learning modules are updated in an asynchronous optimization manner to achieve collaborative optimization of the two feature learning modules, while ensuring that the target image generated by the target image generation model has higher authenticity and diversity.
[0038] In summary, the present invention proposes a method for constructing an image generation model, the flow chart of which is as follows: Figure 1 As shown, it includes: step S1, obtaining a preset training set, the preset training set includes one or more texts describing an image and corresponding multiple images, and the multiple images have some common features with the target image; step S2, constructing an initial image generation model, the initial image generation model includes a pre-trained image generation model, a first feature learning module, and a second feature learning module, the pre-trained image generation model includes an encoding module and a decoding module, wherein: the encoding module includes multiple network layers, which are used to extract the features of the image and transmit feature data layer by layer in the network layer to generate a basic feature vector; the first feature learning module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the common features of the image and construct a common feature vector based on this; the second data adjustment module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the differential features of the image and construct a differential feature vector based on this; the decoding module is used to decode and generate multiple target images according to the common feature vector, the differential feature vector, and the basic feature vector; step S3, using the preset training set to perform multiple rounds of iterative training on the initial image generation model until convergence to obtain the target image generation model.
[0039] In order to better understand the present invention, each step is described in detail below in conjunction with specific embodiments.
[0040] According to an embodiment of the present invention, the plurality of images in the preset training set are images having common features and certain differences, and the preset training set also includes one or more texts describing these images.
[0041] According to one embodiment of the present invention, the pre-trained image generation model is a pre-trained image generation model, whose neural network structure is configured as U-Net or Diffusion Transformer. The image generation model can generate one or more corresponding images according to the input text.
[0042] According to one embodiment of the present invention, the first feature learning module is intended to extract common features from the feature data of the images in the preset training set, thereby ensuring that the images generated by the target image generation model contain discriminative features. The second feature learning module is intended to capture the reasonable changes between the feature data of each image in the preset training set, that is, the differential features between images, thereby ensuring the diversity of the images generated by the target image generation model.
[0043] According to one embodiment of the present invention, in the first feature learning module and the second feature learning module, the feature data transmitted between network layers in the encoding module are multiplied by the parameters in the first feature learning module to extract common features and thereby construct a common feature vector, the feature data transmitted between network layers are multiplied by the parameters in the second feature learning module to extract differential features and thereby construct a differential feature vector, and the feature data transmitted between network layers are multiplied by the parameters of a pre-trained image generation model to construct a basic feature vector. Therefore, the basic feature vector, the common feature vector, and the differential feature vector in the decoding module can be represented as the product of the feature data transmitted between network layers and the parameters of the pre-trained image generation model, the parameters of the first feature learning module, and the parameters of the second feature learning module, wherein the sum of the parameters of the pre-trained image generation model, the parameters of the first feature learning module, and the parameters in the second feature learning module is The configuration is:
[0044]
[0045] in, Denote the weights of the first feature learning module and the second feature learning module respectively. Generate the parameters of the model for the pre-trained images, represents the parameters of the first feature learning module, Represents the parameters of the second feature learning module.
[0046] According to one embodiment of the present invention, step S3 includes: step S31, constructing a common feature queue and a difference feature queue, wherein the common feature queue is used to save the latest preset number of common feature vectors generated in the first feature learning module, and the difference feature queue is used to save the latest preset number of difference feature vectors generated in the second feature learning module; step S32, using a preset training set to perform multiple rounds of iterative training on the initial image generation model, and during the training process, using a preset loss function to update the image generation model based on the constructed common feature vectors, difference feature vectors, common feature queues, and difference feature queues.
[0047] According to one embodiment of the present invention, the common feature queue and the difference feature queue both adopt a queue data structure, and the preset capacity is determined according to the size of the video memory in the environment of the training image generation model. The larger the video memory, the larger the size that can be set for the common feature queue and the difference feature queue. The feature vectors stored in the common feature queue and the difference feature queue are arranged in the order of construction. The common feature queue / difference feature vector constructed earlier is located at the head of the queue, and vice versa, the common feature queue / difference feature vector generated later is located at the tail of the queue. The common feature queue is used to save the common feature vector constructed in the first feature learning module. The common feature vector is expressed as ; The difference feature queue is used to store the difference feature vector constructed in the second feature learning module, and the difference feature queue The difference feature vector is expressed as The common feature queue and the difference feature queue are temporary feature queues constructed during model training and used to optimize model parameters, and the target image generation model obtained after training does not include the common feature queue and the difference feature queue.
[0048] According to one embodiment of the present invention, in step S32, if the synchronous optimization commonly used in the prior art is adopted without supervised optimization to optimize the first feature learning module and the second feature learning module, it will cause the model to be extremely unstable and even cause the model to crash. Therefore, the present invention proposes an asynchronous optimization method to update the first feature learning module and the second feature learning module. In summary, in each round of iteration, the first feature learning module is first updated once, and then the second feature learning module is updated a preset number of times to make the second feature learning module adapt to the updated changes of the first feature learning module. This asynchronous optimization strategy can realize the collaborative optimization of the first feature learning module and the second feature learning module, so that the model can converge stably.
[0049] According to one embodiment of the present invention, in step S32, each round of iteration includes: a first stage, including using a preset training set to perform a first retraining and a second retraining on the image generation model updated in the previous round in sequence, wherein, during the first retraining, a first feature learning module is used to obtain common features of images in the preset training set to construct a common feature vector, and based on the constructed common feature vector and the common feature queue, a first preset loss function is used to update the first feature learning module to obtain the image generation model after the first retraining, and the common feature queue is updated based on the constructed common feature vector; during the second retraining, the image generation model after the first retraining is trained using the preset training set and a second preset loss function is used to update the first feature learning module and the second retraining. Feature learning module; the second stage includes using a preset training set to train the image generation model trained in the first stage for a preset number of times, and each training is performed in turn with a third training and a fourth training, wherein, during the third training, the second feature learning module is used to obtain the differential features of the images in the preset training set to construct a differential feature vector, and based on the constructed differential feature vector and the differential feature queue, the third preset loss function is used to update the second feature learning module to obtain the image generation model after the third training, and the differential feature queue is updated based on the constructed differential feature vector; during the fourth training, the preset training set is used to train the image generation model after the third training and the second preset loss function is used to update the first feature learning module and the second feature learning module.
[0050] According to an embodiment of the present invention, in step S32, each round of iteration only updates the parameters in the first feature learning module and the second feature learning module, and does not change the parameters in the pre-trained image model, wherein in the first stage, updating the first feature learning module is to update the parameters in the first feature learning module. In the second stage, updating the second feature learning module is to update the parameters in the second feature learning module. .
[0051] In the existing technology, in order to ensure that the images generated by the target image generation model have common features, Euclidean distance or cosine similarity is often used to constrain the distance between features during the model training process. However, these methods enforce pixel-by-pixel constraints between feature vectors. Since the features constructed by the first feature learning module should show consistency in the overall semantic sense rather than pixel-by-pixel consistency, Euclidean distance or cosine similarity are not suitable for constraining the common feature vectors constructed in the first feature learning module.
[0052] According to an embodiment of the present invention, the present invention adopts a context loss calculation function as the first preset loss function to constrain the overall distribution consistency of the common feature vector in the spatial dimension. The context loss function is configured as:
[0053]
[0054] in, represents the context loss, Represents the number of common feature vectors stored in the common feature queue, represents the number of channels of the pre-trained image generation model, represents the mean calculation function, represents the covariance calculation function, Represents the common feature vector currently traversed, Represents the i-th common feature vector in the common feature queue.
[0055] According to one embodiment of the present invention, in the first stage, the common feature queue is updated in the following manner: when the number of common feature vectors in the common feature queue has not reached the preset capacity, the constructed common feature vector is directly inserted into the tail of the common feature queue; when the number of common feature vectors in the common feature queue has not reached the preset capacity, the common feature vector at the head of the common feature queue is popped out and the constructed common feature vector is inserted into the tail of the common feature queue.
[0056] In order to more clearly describe the process of updating the common feature queue, the following will be described in detail in combination with Example 1 and Example 2.
[0057] Example 1: The preset capacity of the common feature queue is , the common feature vector constructed is , the current common feature queue is and , that is, the current common feature queue has not reached the preset capacity, then the common feature vector is directly Insert the current common feature queue to the end to update the common feature queue to { }.
[0058] Example 2: The preset capacity of the common feature queue is , the common feature vector constructed is , if the current common feature queue is and , that is, the current common feature queue has not yet reached the preset capacity, and space needs to be reserved for the newly inserted common feature vector, so the common feature vector is popped out from the head of the common feature queue , and then insert from the end To update the common feature queue to .
[0059] According to one embodiment of the present invention, the multiple images in the preset training set have certain difference features on the basis of having the same common features, so the difference feature vector constructed by the second feature learning module will also show corresponding rules. Therefore, when the method constrains the difference feature vector, the orthogonality between different difference feature vectors is not strictly enforced and a certain angle is allowed, that is, a certain degree of overlap is allowed to ensure that there is partial similarity between the difference feature vectors. The present invention constructs a difference loss function based on the above analysis. The cosine threshold preset in the difference loss function is used to control the maximum angle constraint between the difference feature vectors. When the calculated value of the cosine of the angle between the two difference feature vectors is less than the cosine threshold, the cosine value of the two difference feature vectors is configured to 0, that is, partial similarity between the difference feature vectors is allowed.
[0060] According to one embodiment of the present invention, in the second stage, in order to ensure that the multiple images generated by the target image generation model have differences, a difference loss function is used as the second preset loss function to constrain the difference between the difference feature vectors in the newly constructed difference feature queue and the difference feature queue, wherein the difference loss function is configured as:
[0061]
[0062] in, represents the difference loss, Represents the number of difference feature vectors stored in the difference feature queue, represents the difference feature vector of the current traversal, represents the i-th difference feature vector in the difference feature queue, Represents the preset cosine threshold, represents a function that calculates the absolute value of the cosine of the angle between two vectors. Represents the cosine collocation function of two vectors.
[0063] According to one embodiment of the present invention, in the second stage, the difference feature queue is updated in the following manner: when the number of difference feature vectors in the difference feature queue has not reached the preset capacity, the constructed difference feature vector is directly inserted into the tail of the difference feature queue; when the number of difference feature vectors in the difference feature queue has not reached the preset capacity, the difference feature vector at the head of the difference feature queue is popped out and the constructed difference feature vector is inserted into the tail of the difference feature queue.
[0064] In order to more clearly describe the process of updating the difference feature queue, the following will be described in detail in combination with Example 3 and Example 4.
[0065] Example 3: The preset capacity of the difference feature queue is , the constructed difference feature vector is , the current difference feature queue is represented as and , that is, the current difference feature queue has not reached the preset capacity, then the difference feature vector is directly Insert the tail of the current difference feature queue to update the difference feature queue to { }.
[0066] Example 4: The preset capacity of the difference feature queue is , the constructed difference feature vector is , if the current difference feature queue is and , that is, the current difference feature queue has not reached the preset capacity, and space needs to be reserved for the newly inserted difference feature vector, so the difference feature vector is popped out from the head of the difference feature queue , insert from the end To update the difference feature queue to .
[0067] According to one embodiment of the present invention, in the first stage and the second stage, after each update of the first feature learning module or the second feature learning module, the second loss function is used to calculate the loss and simultaneously update the first feature learning module and the second feature learning module, wherein the third preset loss function is a potential diffusion model loss function, which is used to calculate the loss between the image generated by the image generation model and the image in the preset training set, and optimize the first feature learning module and the second feature learning module with the calculated loss, wherein the diffusion model loss function is configured as:
[0068]
[0069] in, represents the potential diffusion model loss, represents the time step, represents the initial latent variable, express Hidden variables at time steps, represents the conditional vector, represents noise, Represents the parameters of the image generation model, including the parameters of the pre-trained image generation model, the parameters of the first feature learning module, and the parameters of the second feature learning module. Represents the prediction noise of the pre-trained image generation model.
[0070] According to one embodiment of the present invention, the initial image generation model structure diagram is as follows: Figure 2As shown, the pre-trained image generation model is used to extract the features of the image in the input initial training set to construct a basic feature vector, the first feature learning module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the common features of the image and construct the common feature vector based on this and return the common feature vector to the pre-trained image generation model, in the model training process, the context loss function is used to calculate the context loss between the constructed common feature vector and the common feature vector in the common feature queue and update the first feature learning module accordingly, and insert the constructed common feature vector into the tail of the common feature queue; the second feature learning module is used to obtain the feature data transmitted between the network layers of the encoding module in the pre-trained image generation model to extract the difference features of the image and construct the difference feature vector based on this and return the difference feature vector to the pre-trained image generation model, in the model training process, the difference loss function is used to calculate the difference loss between the constructed difference feature vector and the difference feature vector in the difference feature queue and update the second feature learning module accordingly, and insert the constructed difference feature vector into the tail of the difference feature queue; in the model training process, the diffusion model loss is used to calculate the loss between the generated image and the preset training set image to update the first feature learning module and the second feature learning module.
[0071] According to one embodiment of the present invention, a method for generating a target image using an image generation model constructed based on an image generation model construction method proposed by the present invention is proposed, comprising: establishing a preset training set, the preset training set comprising multiple images and one or more texts describing the images, wherein the multiple images and the target image have certain differences on the basis of having common features, for example, assuming that the target image is an image of a white puppy, "white" and "puppy" are the discriminant features of the target image, and are also the common features of multiple images in the preset training set, so the images in the preset training set are multiple white puppy images and other features of these images except the common features may be different, such as the breed, action, environment of the puppy, etc., the one or more texts configured in the training set for describing the image may be texts describing image features, such as "white puppy", "white dog", etc., or may be irregular labels, such as "1", "a", etc., it should be understood that the present invention does not limit the text describing the image; constructing a target image generation model using an image generation model construction method proposed by the present invention based on the established preset training set; inputting the configured text in the preset training set into the constructed target image generation model to generate the target image.
[0072] In order to demonstrate the beneficial effects of the present invention, the inventors conducted image quality visualization experiments, downstream classification task accuracy experiments, and 3D multi-view image generation task experiments to compare the performance of the target image generation model constructed by the existing methods (WaveGAN, HAE, TextualInversion, DreamDistribution, DreamBooth) and the method proposed in the present invention (Dis²Booth) to demonstrate the beneficial effects of the present invention. The image quality quantitative evaluation experimental data results are shown in Figure 2. Figure 3 As shown in Figure 2, the image quality visual evaluation results are as follows: Figure 4 , Figure 5 As stated in Figure 3 The experimental data results include two performance indicators: FID and LPIPS. FID is an indicator that measures the difference between the generated image and the real image. The lower the FID value, the more similar the generated image is to the real image. LPIPS is a perceptual similarity indicator that measures the visual similarity of images. The lower the LPIPS value, the more similar the generated image is to the reference image. By analyzing Figure 3 Experimental data and observations Figure 4 , Figure 5 The images in the image can be obtained: Compared with the existing methods, the image generation model constructed by the target image model construction method proposed in the present invention has higher authenticity and diversity; the downstream classification task accuracy experimental results are as follows Figure 6 As shown, through Figure 6 The data analysis shows that the proposed method has higher accuracy than existing methods in performing downstream classification tasks. The image generation model constructed by the existing methods (WaveGAN, HAE, TextualInversion, DreamDistribution, DreamBooth) and the target image model construction method proposed in this invention (Dis²Booth) is transferred to the 3D multi-view image generation task. Figure 7 As shown, from the experimental results shown in the figure, it can be seen that the multi-view images generated by the image generation model constructed by the target image model construction method (Dis²Booth) proposed in the present invention also have higher diversity than the existing methods.
[0073] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.
[0074] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing an adjuster to implement various aspects of the present invention.
[0075] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.
[0076] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A method for constructing an image generation model, for constructing a target image generation model, characterized in that: The method comprises: Step S1, obtaining a preset training set, wherein the preset training set includes one or more texts describing an image and a corresponding plurality of images, wherein the plurality of images have some common features with the target image; Step S2: construct an initial image generation model, wherein the initial image generation model comprises a pre-trained image generation model, a first feature learning module, and a second feature learning module, wherein the pre-trained image generation model comprises an encoding module and a decoding module, wherein: The encoding module includes a plurality of network layers, which are used to extract features of the image and transmit feature data layer by layer in the network layer to generate basic feature vectors; The first feature learning module is used to obtain feature data transmitted between network layers of the encoding module in the pre-trained image generation model to extract common features of the image and construct a common feature vector based on this; The second feature learning module is used to obtain feature data transmitted between network layers of the encoding module in the pre-trained image generation model to extract the difference features of the image and construct a difference feature vector based on this; The decoding module is used to generate multiple target images by decoding the common feature vectors, the difference feature vectors and the basic feature vectors; Step S3: Perform multiple rounds of iterative training on the initial image generation model using a preset training set until convergence to obtain a target image generation model.
2. The method according to claim 1, characterized in that The step S3 comprises: Step S31, constructing a common feature queue and a difference feature queue, wherein the common feature queue is used to store the latest preset number of common feature vectors generated in the first feature learning module, and the difference feature queue is used to store the latest preset number of difference feature vectors generated in the second feature learning module; Step S32, using a preset training set to perform multiple rounds of iterative training on the initial image generation model, and during the training process, using a preset loss function to update the image generation model based on the constructed common feature vectors, difference feature vectors, common feature queues, and difference feature queues.
3. The method according to claim 2, characterized in that In step S32, each round of iteration includes: The first stage includes using a preset training set to perform a first retraining and a second retraining on the image generation model updated in the previous round in sequence, wherein, during the first retraining, a first feature learning module is used to obtain common features of images in the preset training set to construct a common feature vector, and based on the constructed common feature vector and the common feature queue, a first preset loss function is used to update the first feature learning module to obtain the image generation model after the first retraining, and the common feature queue is updated based on the constructed common feature vector; during the second retraining, the image generation model after the first retraining is trained using the preset training set and a second preset loss function is used to update the first feature learning module and the second feature learning module; The second stage includes using a preset training set to train the image generation model trained in the first stage for a preset number of times, and each training is performed in turn with a third training and a fourth training, wherein, during the third training, the second feature learning module is used to obtain the differential features of the images in the preset training set to construct a differential feature vector, and based on the constructed differential feature vector and the differential feature queue, the third preset loss function is used to update the second feature learning module to obtain the image generation model after the third training, and the differential feature queue is updated based on the constructed differential feature vector; during the fourth training, the image generation model after the third training is trained using the preset training set and the second preset loss function is used to update the first feature learning module and the second feature learning module.
4. The method according to claim 3, characterized in that The first preset loss function is configured as: in, represents the context loss, Represents the number of common feature vectors stored in the common feature queue, represents the number of channels of the pre-trained image generation model, represents the mean calculation function, represents the covariance calculation function, Represents the common feature vector currently traversed, Represents the i-th common feature vector in the common feature queue.
5. The method according to claim 3, characterized in that: The second preset loss function is configured as: in, represents the potential diffusion model loss, represents the time step, represents the initial latent variable, express The hidden variables of the time step, represents the conditional vector, represents noise, Represents the parameters of the image generation model, including the parameters of the pre-trained image generation model, the parameters of the first feature learning module, and the parameters of the second feature learning module. Represents the prediction noise of the pre-trained image generation model.
6. The method according to claim 3, characterized in that The third preset loss function is configured as: in, represents the difference loss, Represents the number of difference feature vectors stored in the difference feature queue, represents the difference feature vector of the current traversal, represents the i-th difference feature vector in the difference feature queue, Represents the preset cosine threshold, represents a function that calculates the absolute value of the cosine of the angle between two vectors, Represents the cosine collocation function of two vectors.
7. The method according to claim 3, characterized in that The common feature queue and the difference feature queue are updated in the following manner: Update the common feature queue as follows: When the number of common feature vectors in the common feature queue does not reach a preset capacity, the constructed common feature vector is directly inserted into the tail of the common feature queue; When the number of common feature vectors in the common feature queue has not reached a preset capacity, the common feature vector at the head of the common feature queue is popped out and the constructed common feature vector is inserted into the tail of the common feature queue; Update the difference feature queue as follows: When the number of difference feature vectors in the difference feature queue does not reach a preset capacity, the constructed difference feature vector is directly inserted into the tail of the difference feature queue; When the number of difference feature vectors in the difference feature queue has not reached the preset capacity, the difference feature vector at the head of the difference feature queue is popped out and the constructed difference feature vector is inserted into the tail of the difference feature queue.
8. The method according to any one of claims 1 to 7, characterized in that: The neural network architecture of the pre-trained image generation model is configured as U-Net or Diffusion Transformer.
9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by the regulator to implement the steps of the method according to any one of claims 1 to 8.
10. An electronic device, characterized in that: include: one or more regulators; as well as A memory, wherein the memory is used to store executable instructions; The one or more adjusters are configured to implement the steps of the method of any one of claims 1-8 by executing the executable instructions.