Figure generation method, apparatus and device, and computer readable storage medium

By introducing a combination of mixed expert model and backbone model in the literary and technological diffusion model, multiple sets of denoising model parameters are activated, and the problems of insufficient expression capabilities and low image quality of the existing literary and technological model are solved, which significantly improves the image generation quality and text Relevance.

CN118918414APending Publication Date: 2024-11-08ZHEJIANG TONGHUASHUN INTELLIGENT TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410961903.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing literary and biographic diffusion model has limited expression capabilities, poor image quality, and there is still room for improvement in the correlation between the generated image and the provided text.

Method used

Using pre-trained literary and biographic diffusion model, including text encoder and denoising model with expert adjustment module, multiple sets of denoising model parameters are activated at different time steps through the mixing expert model and backbone model, improving expression ability and compatibility with different Style image generation capability.

Benefits of technology

It significantly improves the image generation quality and text correlation of the literary diffusion model, ensures strong correlation between the generated images and text descriptions, and maintains the single-style image generation effect in the multi-style image training data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918414A_ABST
    Figure CN118918414A_ABST
Patent Text Reader

Abstract

The invention discloses a text map method, device and equipment and a computer readable storage medium, which are applied to the field of computers, and the method comprises the following steps: obtaining a pre-trained text map diffusion model, the text map diffusion model comprising a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module comprises a hybrid expert model and a trunk model; inputting the text prompt words into a text encoder to obtain text embedding; according to time embedding and text embedding, target models activated in different denoising time periods are determined by using a hybrid expert model; and in different de-noising time periods, de-noising processing is carried out on a noisy image self-generated by the de-noising model through the corresponding target model and the trunk model, and an image corresponding to the text cue word is generated. According to the invention, the expression ability of the text map diffusion model and the compatibility to different styles are improved, and the image generation effect of the text map diffusion model is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a method, device, equipment and computer-readable storage medium for text graphics. Background Art

[0002] In recent years, research on image generation algorithms has made significant progress. Among them, the denoising diffusion probability model (referred to as the diffusion model) has better training stability, sampling diversity and higher generation quality than existing methods, and has gradually become a popular image generation method. The diffusion model gradually applies noise to the image in the forward stage until the image is destroyed and becomes completely Gaussian noise, and then learns the process of restoring the original image from Gaussian noise in the reverse stage. In the inference stage, the diffusion model samples a random Gaussian noise image, and then generates the target image by performing multi-step denoising on the noise image. In each denoising process, the model predicts the noise that needs to be removed in the current step, and all time steps share a set of network parameters. In the process of predicting noise, the cross-attention mechanism can be used to inject text information into the network to guide the image generation process so that the generated image content conforms to the provided text description.

[0003] The existing text-image diffusion model has limited expressive power. The images generated using the text-image diffusion model are of poor quality, and there is still room for further improvement in the correlation between the generated images and the provided text. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a text-generated image method, device, equipment and computer-readable storage medium, which solves the problems that the Chinese text-generated image diffusion model in the prior art has limited expressive ability, poor quality of generated images, and there is still room for further improvement in the correlation between the generated image and the provided text.

[0005] In order to solve the above technical problems, the present invention provides a method for generating a cultural map, comprising:

[0006] Acquire a pre-trained text-graph diffusion model, wherein the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model;

[0007] Inputting the text prompt words into the text encoder to obtain text embedding;

[0008] Determine the target model activated in different denoising time periods using the hybrid expert model according to the time embedding and the text embedding;

[0009] In different denoising time periods, the noisy image generated by the denoising model is denoised by the corresponding target model and the backbone model to generate an image corresponding to the text prompt word.

[0010] Optionally, the pre-trained Vincent graph diffusion model further includes a priori model;

[0011] Correspondingly, after obtaining the text embedding, it also includes:

[0012] Inputting the text embedding into the prior model to obtain image embedding;

[0013] Accordingly, the method of determining the target model activated in different denoising time periods by using the hybrid expert model according to the time embedding and the text embedding includes:

[0014] According to the text embedding, the time embedding and the image embedding, a hybrid expert model in the text graph diffusion model is used to determine a target model activated in different denoising time periods.

[0015] Optionally, the training process of the prior model includes:

[0016] Get training images and corresponding text descriptions;

[0017] Encoding the text description into a training text embedding using a text encoder of the CLIP model;

[0018] Encoding the training image into a training image embedding using an image encoder of a CLIP model;

[0019] Inputting the training text embedding into the initial prior model to obtain a predicted image embedding;

[0020] The loss function is used to calculate the loss value between the predicted image embedding and the training image embedding. When a condition is met, the training ends and the prior model is obtained.

[0021] Optionally, the loss function is:

[0022] f θ represents the prior model; represents the predicted image embedding based on the initial prior model; z i ′ represents the training image embedding; z t ′ represents the training text embedding; L prior represents the loss function of the prior model; t represents the time step; T represents the total time step; Indicates expectation; q t Represents the true distribution of noisy image features at time t; Indicates that from q t A noisy image feature obtained by sampling.

[0023] Optionally, according to the text embedding, the time embedding and the image embedding, a hybrid expert model in the text graph diffusion model is used to determine a target model activated in different denoising time periods, including:

[0024] The gate control unit in the hybrid expert model is based on Determine the target model activated during different denoising time periods;

[0025] Among them, [;] represents the concatenation of vectors, W represents the mapping matrix; t represents the time step; T represents the total time step; z t represents text embedding; z i represents image embedding; Indicates that any model in the hybrid expert model is not activated; G(·) represents a gating unit.

[0026] Optionally, the loss function of the denoising model with the expert adjustment module during training is:

[0027]

[0028] Among them, Φ0 represents the backbone model parameters; Φ i represents the model parameters of a single branch in the hybrid expert model; L represents the training loss function of the denoising model with the expert adjustment module; x0 represents a clean image without noise; represents the standard distribution; ∈ is from the standard distribution The noise sampled from t represents a noisy image; t represents the time step; T represents the total time step; represents the expected value; q(x0) represents the true distribution of the image; z t ′ represents the training text embedding; z i ′ represents the training image embedding.

[0029] The present invention also provides a Wensheng map device, comprising:

[0030] A model acquisition module, used to acquire a pre-trained text-graph diffusion model, wherein the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model;

[0031] A first input module, used for inputting text prompt words into the text encoder to obtain text embedding;

[0032] A model activation module, used to determine the target model activated in different denoising time periods using the hybrid expert model according to the time embedding and the text embedding;

[0033] The image generation module is used to perform denoising processing on the noisy image generated by the denoising model through the corresponding target model and the backbone model in different denoising time periods to generate an image corresponding to the text prompt word.

[0034] Optionally, the pre-trained Vincent graph diffusion model further includes a priori model;

[0035] Correspondingly, it also includes:

[0036] A second input module, used for inputting the text embedding into the prior model to obtain image embedding;

[0037] Accordingly, the model activation module includes:

[0038] A model activation unit is used to determine a target model activated in different denoising time periods based on the text embedding, the time embedding and the image embedding using a hybrid expert model in the text graph diffusion model.

[0039] The present invention also provides a Wensheng map device, comprising:

[0040] Memory for storing computer programs;

[0041] A processor is used to implement the steps of the above-mentioned Vincent diagram method when executing the computer program.

[0042] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the steps of the above-mentioned Vincent diagram method are implemented.

[0043] It can be seen that the present invention obtains a pre-trained Wensheng graph diffusion model, which includes a text encoder and a denoising model with an expert adjustment module; the denoising model includes a hybrid expert model and a backbone model; the text prompt word is input into the text encoder to obtain text embedding; according to the time embedding and the text embedding, the hybrid expert model is used to determine the target model activated in different denoising time periods; in different denoising time periods, the corresponding target model and the backbone model are used to denoise the noisy image generated by the denoising model to generate the image corresponding to the text prompt word. This method improves the structure of the Wensheng graph diffusion model, and forms a denoising model with an expert adjustment module by adding a model branch based on the hybrid expert model on the basis of the backbone model. Different target models are activated at different time steps, and they are combined with the backbone model to form multiple sets of denoising model parameters, thereby improving the expressive ability of the Wensheng graph diffusion model and the compatibility with different styles, thereby improving the image generation effect of the Wensheng graph diffusion model; and when the training data contains images of different styles, and the images of multiple styles will affect each other, the present invention can also ensure the generation effect of the Wensheng graph model trained on a single style image.

[0044] In addition, the present invention also provides a cultural map device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flowchart of a method for producing a text map provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the overall architecture of a cultural graph diffusion model provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a specific structure of a denoising model provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the structure of a Wensheng graph device provided by an embodiment of the present invention;

[0050] Figure 5 A schematic structural diagram of a text image device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Please refer to Figure 1 , Figure 1 A flowchart of a method for a text map provided by an embodiment of the present invention. The method may include:

[0053] S101: Obtain a pre-trained text-graph diffusion model, where the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model includes a hybrid expert model and a backbone model.

[0054] Specifically, the overall architecture diagram of the cultural graph diffusion model in this embodiment can be referred to as Figure 2 , the specific structural diagram of the denoising model can be referred to Figure 3 The text-graph diffusion model in this embodiment includes a text encoder and a denoising model with an expert adjustment module, wherein the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model. Figure 3 As shown, the hybrid expert model includes various expert modules and gating units. The backbone model can be a U-Net (artificial intelligence image segmentation algorithm) backbone model. In this embodiment, the text encoder and the denoising model with the expert adjustment module can be trained separately to obtain a trained text image diffusion model. For example, model training can be divided into two stages: first training the prior model, and then training the denoising model. The prior model is in the form of a diffusion model, and the denoising model is also in the form of a diffusion model. The prior model here can also take other forms.

[0055] S102: Input the text prompt word into the text encoder to obtain text embedding.

[0056] This embodiment uses a text encoder to convert the text prompt words input by the user into text embeddings.

[0057] S103: Determine the target model activated in different denoising time periods using a hybrid expert model according to the time embedding and the text embedding.

[0058] This embodiment introduces time embedding based on the text embedding obtained by encoding text prompt words based on the text encoder, and uses the gating unit in the hybrid expert model to activate the corresponding target model in different denoising time periods. The Vincent graph diffusion model generates images by multi-step denoising of noisy images. During training, the Vincent graph diffusion model sets the total number of denoising steps to T. During inference, the current number of time steps is set to t. If the total denoising step T is 1000, the value of t will change from 999 to 0 with the number of denoising rounds. Time embedding uses sinusoidal embedding, inputs the time step value t, t is an integer, and the time embedding is a 2d-dimensional vector. The specific formula is as follows:

[0059]

[0060] Among them, PE t,i Represents the value of time embedding at the i-th dimension. The first d are generated by sin, and the last d are generated by cos.

[0061] S104: In different denoising time periods, the noisy image generated by the denoising model is denoised by using the corresponding target model and the backbone model to generate an image corresponding to the text prompt word.

[0062] Specifically, this embodiment enhances the model's expressiveness by adding a branch model based on the hybrid expert model in addition to the main model, activating different branch models at different time steps, and combining them with the main model to form multiple sets of model parameters. The image corresponding to the text prompt word is obtained based on the text image diffusion model.

[0063] The method for applying the text graph provided by the embodiment of the present invention is to obtain a pre-trained text graph diffusion model, the text graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model; the text prompt word is input into the text encoder to obtain text embedding; according to the time embedding and the text embedding, the hybrid expert model is used to determine the target model activated in different denoising time periods; in different denoising time periods, the noisy image generated by the denoising model is denoised by the corresponding target model and the backbone model to generate an image corresponding to the text prompt word. The method improves the structure of the text graph diffusion model, and forms a denoising model with an expert adjustment module by adding a branch based on the hybrid expert model on the basis of the backbone model. Different target models are activated at different time steps and combined with the backbone model to form multiple sets of denoising model parameters, thereby improving the expressiveness of the Vincent graph diffusion model and its compatibility with different styles, thereby improving the image generation effect of the Vincent graph diffusion model; and when the training data contains images of different styles and images of multiple styles affect each other, the present invention can also ensure the generation effect of the trained Vincent graph model on a single style image.

[0064] Based on the above embodiment, the above pre-trained Vincent graph diffusion model also includes a priori model. Accordingly, after obtaining the text embedding, the following steps may also be included: inputting the text embedding into the priori model to obtain the image embedding; accordingly, according to the time embedding and the text embedding, using the hybrid expert model to determine the target model activated in different denoising time periods, including: according to the text embedding, the time embedding and the image embedding, using the hybrid expert model in the Vincent graph diffusion model to determine the target model activated in different denoising time periods.

[0065] Specifically, the overall architecture diagram of the Wensheng graph diffusion model with the addition of the prior model can be referred to Figure 2 . In this embodiment, the text embedding obtained by encoding the text encoder is input into the denoising model and the prior model at the same time. The prior model can generate image embedding based on the text embedding, and the two are input into the denoising model with the expert adjustment module. The hybrid expert model in the denoising model with the expert adjustment module determines the target model used in each denoising time period based on the time embedding, the text embedding and the image embedding, and then jointly performs denoising on the noisy image generated by the denoising model with the backbone model to obtain the denoised image, that is, the image corresponding to the text prompt word.

[0066] This embodiment uses a priori models to generate image embedding, so that the hybrid expert model can be more accurate when determining the target model, resulting in better denoising effect, and the image generated based on the text prompt word has a strong correlation with the text prompt word.

[0067] Furthermore, the verification process of the above-mentioned prior model may specifically include the following steps:

[0068] Get training images and corresponding text descriptions;

[0069] Encode the text description into training text embedding using the text encoder of the CLIP model;

[0070] Encode the training images into training image embeddings using the image encoder of the CLIP model;

[0071] Input the training text embedding into the initial prior model to obtain the predicted image embedding;

[0072] The loss function is used to calculate the loss value between the predicted image embedding and the training image embedding. When the conditions are met, the training ends and the prior model is obtained.

[0073] Specifically, this embodiment uses training samples to train the prior model, wherein the training samples include training images and corresponding text descriptions, and the text descriptions and training images are encoded using the image encoder of the text encoder of the CLIP (CLIP model is a multimodal pre-trained neural network model) model to obtain training text embedding and training image embedding, respectively, and the prior model is trained using the training text embedding and training image embedding, and during the training process, the loss value calculated by the loss function is used to evaluate the training of the prior model. When the conditions are met, such as the loss value reaches a preset value, or the number of iterations reaches a preset value, the training is terminated to obtain the prior model.

[0074] Furthermore, the formula expression of the above loss function is:

[0075]

[0076] Among them, f θ represents the prior model; represents the predicted image embedding based on the initial prior model; z i ′ represents the training image embedding; z t ′ represents the training text embedding; L prior represents the loss function of the prior model; t represents the time step; T represents the total time step; Indicates expectation; q t Represents the true distribution of noisy image features at time t; Indicates that from q t A noisy image feature obtained by sampling.

[0077] After the prior model training is completed, the denoising model with the expert adjustment module can be trained. The existing denoising model is an unconditional denoising model. The parameter of the denoising model is Φ. The unconditional denoising model is used in the prior art. Its training process is as follows: sample a noise, use the noise and the clean image x0 without noise to sample the noisy image x t , according to the noisy image x t To predict the added noise ∈, and calculate the mean square error using the following loss function:

[0078]

[0079] Where t represents the time step; T represents the total time step, usually 1000; x0 represents a clean image without noise; represents the standard distribution; ∈ is from the standard distribution The noise sampled out.

[0080] The denoising model in this embodiment adds an additional control condition, namely, the text embedding z t and image embedding z i , to generate the corresponding image, so the denoising model in this embodiment is a conditional generation model, and its loss function calculation formula is:

[0081]

[0082] On the basis of the conditional generative model, by adding a branch network based on the hybrid expert model to the main model, different model branches are activated at different time steps, and they are combined with the main model to form multiple sets of model parameters to improve the model's expressiveness. Let the main model parameter be Φ0, and the parameter of a single branch network be Φ i , i = 1, 2, ... n, the structure of the branch model is a LoRA (Low-Rank Adaptation of Large Language Models, a low-rank adaptation technology for fine-tuning large language models) module, that is, Figure 3 Each expert module in is a LoRA module. The backbone model in this embodiment can characterize common features, and the models of each branch can characterize difference features. This method of this embodiment is conducive to reducing the learning difficulty of the common part, so it can increase the learning samples of the common part and improve the generalization ability.

[0083] Among them, the loss function calculation formula of the denoising model with expert adjustment module is:

[0084]

[0085] Among them, Φ0 represents the backbone model parameters; Φ irepresents the model parameters of a single branch in the hybrid expert model; L represents the training loss function of the denoising model with the expert adjustment module; x0 represents a clean image without noise; represents the standard distribution; ∈ is from the standard distribution The noise sampled from t represents a noisy image; t represents the time step; T represents the total time step; represents the expected value; q(x0) represents the true distribution of the image; z t ′ represents the training text embedding; z i ′ represents the training image embedding.

[0086] Furthermore, the above method of determining the target model activated in different denoising time periods based on text embedding, time embedding and image embedding using the hybrid expert model in the text graph diffusion model may specifically include the following steps:

[0087] The gate units in the mixture of experts model are Determine the target model activated during different denoising time periods;

[0088] Where [;] represents the concatenation of vectors; W represents the mapping matrix; t represents the time step; T represents the total time step; z t represents text embedding; z i represents image embedding; Indicates that any model in the hybrid expert model is not activated; G(·) represents a gating unit.

[0089] Specifically, the activation of the target model is controlled by the gate unit G(·), the input of the gate unit is (image embedding, text embedding, time embedding), and the output is the probability vector p of each branch model being selected. The order is predefined for different branch networks, and the value of the i-th position in the probability vector represents the probability pi of the i-th branch being selected.

[0090] From the above formula, we can see that

[0091] (1) When , i.e., the first 25% of the denoising steps, Indicates that the gating unit does not select any branch model, and the denoising is completely done by the backbone model Φ0. At this time, the model mainly generates the low-frequency part of the image, such as semantic information such as image layout. The low-frequency components of images of different styles with the same semantics are similar, so the shared backbone model Φ0 is used to model the common low-frequency information;

[0092] (2) When That is, corresponding to the middle 50% of the denoising steps, the model selects the branch model with the highest probability i = argmax(G(t, z t , z i)), using the parameter combination Φ0+Φ i To express images of different styles, where i = 1, 2, .. n-1;

[0093] (3) When That is, the last 25% of the denoising step, G(t, z t , z i )=[0,0,..0,1],then argmax(G(t,z t ,z i ))=n, indicating that an additional branch model Φ is forced to be selected n , which is specifically responsible for enhancing the model’s ability to reconstruct high-frequency information of images, such as local details of the image.

[0094] The following is an introduction to a Vincent graph device provided by an embodiment of the present invention. The Vincent graph device described below and the Vincent graph method described above can be referred to each other.

[0095] Please refer to Figure 4 , Figure 4 A schematic diagram of a structure of a Wensheng graph device provided by an embodiment of the present invention may include:

[0096] The model acquisition module 100 is used to acquire a pre-trained text-graph diffusion model, wherein the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with an expert adjustment module includes a hybrid expert model and a backbone model;

[0097] A first input module 200, for inputting a text prompt word into the text encoder to obtain a text embedding;

[0098] A model activation module 300, for determining a target model to be activated in different denoising time periods using the hybrid expert model according to the time embedding and the text embedding;

[0099] The image generation module 400 is used to perform denoising processing on the noisy image generated by the denoising model by the corresponding target model and the backbone model in different denoising time periods to generate an image corresponding to the text prompt word.

[0100] Based on the above embodiment, the pre-trained Vincent graph diffusion model further includes a priori model;

[0101] Correspondingly, the Wenshengtu device may further include:

[0102] A second input module, used for inputting the text embedding into the prior model to obtain image embedding;

[0103] Accordingly, the model activation module may include:

[0104] A model activation unit is used to determine a target model activated in different denoising time periods based on the text embedding, the time embedding and the image embedding using a hybrid expert model in the text graph diffusion model.

[0105] Based on the above embodiment, the training process of the prior model of the Wensheng graph device may include:

[0106] A training sample acquisition unit, used to acquire training images and corresponding text descriptions;

[0107] A first encoding unit, configured to encode the text description into a training text embedding using a text encoder of a CLIP model;

[0108] A second encoding unit, configured to encode the training image into a training image embedding using an image encoder of a CLIP model;

[0109] A prediction unit, configured to input the training text embedding into an initial prior model to obtain a predicted image embedding;

[0110] The loss calculation unit is used to calculate the loss value between the predicted image embedding and the training image embedding using a loss function. When a condition is met, the training ends and the prior model is obtained.

[0111] Based on the above embodiment, the loss function is:

[0112]

[0113] f θ represents the prior model; represents the predicted image embedding based on the initial prior model; z i ′ represents the training image embedding; z t ′ represents the training text embedding; L prior represents the loss function of the prior model; t represents the time step; T represents the total time step; Indicates expectation; q t Represents the true distribution of noisy image features at time t; Indicates that from q t A noisy image feature obtained by sampling.

[0114] Based on the above embodiment, the model activation unit may include:

[0115] The activation subunit is used for the gating unit in the hybrid expert model according to Determine the target model activated during different denoising time periods;

[0116] Where [;] represents the concatenation of vectors; W represents the mapping matrix; t represents the time step; T represents the total time step; z t represents text embedding; z i represents image embedding; Indicates that any model in the hybrid expert model is not activated; G(·) represents a gating unit.

[0117] Based on the above embodiment, the loss function of the denoising model with the expert adjustment module during the training process may be:

[0118]

[0119] Among them, Φ0 represents the backbone model parameters; Φ i represents the model parameters of a single branch in the hybrid expert model; L represents the training loss function of the denoising model with the expert adjustment module; x0 represents a clean image without noise; represents the standard distribution; ∈ is from the standard distribution The noise sampled from t represents a noisy image; t represents the time step; T represents the total time step; represents the expected value; q(x0) represents the true distribution of the image; z t ′ represents the training text embedding; z i ′ represents the training image embedding.

[0120] It should be noted that the order of the modules and units in the above-mentioned Wenshengtu device can be changed without affecting the logic.

[0121] The Wensheng graph device provided by the embodiment of the present invention is used to obtain a pre-trained Wensheng graph diffusion model through a model acquisition module 100, and the Wensheng graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model includes a hybrid expert model and a backbone model; a first input module 200 is used to input text prompt words into the text encoder to obtain text embedding; a model activation module 300 is used to determine the target model activated in different denoising time periods based on the time embedding and the text embedding using the hybrid expert model; an image generation module 400 is used to denoise the noisy image generated by the denoising model in different denoising time periods through the corresponding target model and the backbone model to generate an image corresponding to the text prompt word. The device improves the structure of the Wensheng graph diffusion model, and forms a denoising model with an expert adjustment module by adding a branch based on the hybrid expert model on the basis of the backbone model. Different target models are activated at different time steps and combined with the backbone model to form multiple sets of denoising model parameters, thereby improving the expressiveness of the Vincent graph diffusion model and its compatibility with different styles, thereby improving the image generation effect of the Vincent graph diffusion model; and when the training data contains images of different styles and images of multiple styles affect each other, the present invention can also ensure the generation effect of the trained Vincent graph model on a single style image.

[0122] The following is an introduction to a Vincent graph device provided by an embodiment of the present invention. The Vincent graph device described below and the Vincent graph method described above can be referred to each other.

[0123] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a Wensheng graph device provided by an embodiment of the present invention may include:

[0124] A memory 10, used for storing computer programs;

[0125] The processor 20 is used to execute a computer program to implement the above-mentioned Vincent graph method.

[0126] The memory 10 , the processor 20 , and the communication interface 31 all communicate with each other via the communication bus 32 .

[0127] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0128] Obtain a pre-trained text-graph diffusion model, which includes a text encoder and a denoising model with an expert adjustment module; the denoising model includes a hybrid expert model and a backbone model;

[0129] Input the text prompt words into the text encoder to obtain text embedding;

[0130] Based on time embedding and text embedding, a hybrid expert model is used to determine the target model activated in different denoising time periods;

[0131] In different denoising time periods, the noisy images generated by the denoising model are denoised through the corresponding target model and backbone model to generate images corresponding to the text prompt words.

[0132] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0133] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0134] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0135] The communication interface 31 may be an interface of a communication module, and is used to connect to other devices or systems.

[0136] Of course, it should be noted that Figure 5 The structure shown does not constitute a limitation on the Chinese image generation device in the embodiment of the present invention. In actual application, the Chinese image generation device may include Figure 5 More or fewer components than shown, or combinations of certain components.

[0137] The computer-readable storage medium provided by the embodiment of the present invention is introduced below. The computer-readable storage medium described below and the Vincent graph method described above can be referred to each other.

[0138] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned Vincent diagram method are implemented.

[0139] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0142] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0143] The above is a detailed introduction to a text graph method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A Vincent diagram method, characterized in that: include: Acquire a pre-trained text-graph diffusion model, wherein the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model; Inputting the text prompt words into the text encoder to obtain text embedding; Determine the target model activated in different denoising time periods using the hybrid expert model according to the time embedding and the text embedding; In different denoising time periods, the noisy image generated by the denoising model is denoised by the corresponding target model and the backbone model to generate an image corresponding to the text prompt word.

2. The method of claim 1, characterized in that: The pre-trained Vincent graph diffusion model also includes a priori model; Correspondingly, after obtaining the text embedding, it also includes: Inputting the text embedding into the prior model to obtain image embedding; Accordingly, the method of determining the target model activated in different denoising time periods by using the hybrid expert model according to the time embedding and the text embedding includes: According to the text embedding, the time embedding and the image embedding, a hybrid expert model in the text graph diffusion model is used to determine a target model activated in different denoising time periods.

3. The method of claim 2, characterized in that: The training process of the prior model includes: Get training images and corresponding text descriptions; Encoding the text description into a training text embedding using a text encoder of the CLIP model; Encoding the training image into a training image embedding using an image encoder of a CLIP model; Inputting the training text embedding into the initial prior model to obtain a predicted image embedding; The loss function is used to calculate the loss value between the predicted image embedding and the training image embedding. When a condition is met, the training ends and the prior model is obtained.

4. The method of claim 3, characterized in that: The loss function is f θ represents the prior model; represents the predicted image embedding based on the initial prior model; z i ′ represents the training image embedding; z t ′ represents the training text embedding; L prior represents the loss function of the prior model; t represents the time step; T represents the total time step; Indicates expectation; q t Represents the true distribution of noisy image features at time t; Indicates that from q t A noisy image feature obtained by sampling.

5. The method of claim 2, characterized in that: According to the text embedding, the time embedding and the image embedding, a hybrid expert model in the text graph diffusion model is used to determine a target model activated in different denoising time periods, including: The gate control unit in the hybrid expert model is based on Determine the target model activated during different denoising time periods; Where [;] represents the concatenation of vectors; W represents the mapping matrix; t represents the time step; T represents the total time step; z t represents text embedding; zi represents image embedding; Indicates that any model in the hybrid expert model is not activated; G(·) represents a gating unit.

6. The method of claim 1, wherein: The loss function of the denoising model with expert adjustment module during training is: Among them, Φ0 represents the backbone model parameters; Φ i represents the model parameters of a single branch in the hybrid expert model; L represents the training loss function of the denoising model with the expert adjustment module; x0 represents a clean image without noise; represents the standard distribution; ∈ is from the standard distribution The noise sampled from t represents a noisy image; t represents the time step; T represents the total time step; represents the expected value; q(x0) represents the true distribution of the image; z t ′ represents the training text embedding; z i ′ represents the training image embedding.

7. A Wensheng map device, characterized in that: include: A model acquisition module, used to acquire a pre-trained text-graph diffusion model, wherein the text-graph diffusion model includes a text encoder and a denoising model with an expert adjustment module; the denoising model with the expert adjustment module includes a hybrid expert model and a backbone model; A first input module, used for inputting text prompt words into the text encoder to obtain text embedding; A model activation module, used to determine the target model activated in different denoising time periods using the hybrid expert model according to the time embedding and the text embedding; The image generation module is used to perform denoising processing on the noisy image generated by the denoising model through the corresponding target model and the backbone model in different denoising time periods to generate an image corresponding to the text prompt word.

8. The Wensheng map device according to claim 7, characterized in that: The pre-trained Vincent graph diffusion model also includes a priori model; Accordingly, it also includes: A second input module, used for inputting the text embedding into the prior model to obtain image embedding; Accordingly, the model activation module includes: A model activation unit is used to determine a target model activated in different denoising time periods based on the text embedding, the time embedding and the image embedding using a hybrid expert model in the text graph diffusion model.

9. A Wensheng map device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the Vincent diagram method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the steps of the Vincent diagram method according to any one of claims 1 to 6 are implemented.

Citation Information

Cited By

  • Image generation method and device, storage medium and program product

    CN119941910A

  • Cross-modal learning preference optimization enhanced three-dimensional face generation method and system

    CN120510260A