Image generation processing method and device based on stable diffusion model, and related equipment

Abstract

The invention belongs to the technical field of financial science and technology, provides an image generation processing method and device and related equipment, and aims to solve the problem that an SD (Secure Digital) model cannot efficiently adapt to a new generation task through fine tuning in the traditional technology. The method comprises the following steps: acquiring a multi-modal pre-training model, encoding a text based on a preset first text encoder comprising a preset multi-modal pre-training model and a preset first low-rank adaptive layer to obtain text encoding information, and encoding the text encoding information based on a preset image generator comprising a preset stable diffusion model and a preset second low-rank adaptive layer to obtain text encoding information; according to the method, the image coding information is obtained, the target image is generated according to the image coding information, and the generated image corresponding to the text is obtained, so that the adaptive efficient fine tuning of the stable diffusion model on multi-theme or complex theme can be realized by enhancing the control capability of the stable diffusion model on text semantics; and the newly generated task demand of the business can be efficiently completed.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to an image generation and processing method, device, computer equipment and computer-readable storage medium based on a stable diffusion model. Background Art

[0002] Since the emergence of GAN (Generative Adversarial Networks, abbreviated as GAN, Generative Adversarial Network), the field of image generation has made significant progress. It is no longer a problem to generate the desired image through the GAN model, but GAN can only perform well on limited tasks, and there is still room for improvement in generalization. Until the emergence of the Diffusion model, the Diffusion model generated images with more diversity and higher quality than GAN, and the training process of the Diffusion model is also simpler than GAN, which is favored by a large number of researchers. The Stable Diffusion Model is developed on the basis of the traditional Diffusion Model (Denoising Diffusion Probabilistic Models, DDPMs). It is a new type of deep learning model that can realize text generation of pictures and can generate high-quality and diverse images while maintaining stability. For example, in the insurance field including but not limited to the financial field, in order to promote and publicize insurance business, it is necessary to generate promotional pictures about insurance knowledge based on text.

[0003] Since the Diffusion model itself is a multimodal model, multimodal image generation applications have been developed in a short time, including but not limited to text generation image, text generation video, etc., and the stable diffusion model can realize text generation image, etc., and can generate high-quality and diverse images while maintaining stability. However, the Stable Diffusion Model (abbreviated as SD model) is not omnipotent. Due to the limitations of its own algorithm, for example, the SD model has not been trained on the corresponding topic, or although it has been trained on several corresponding topics, it is often unable to complete the generation tasks of some topics due to insufficient matching ability. Therefore, when facing some new fields or specific fields within the scope of the original field, the original SD model is somewhat powerless, and retraining the SD model will consume a lot of manpower and material resources, and the cost is very high. Therefore, fine-tuning the SD model is the current mainstream method. The current open source fine-tuning methods include Dreambooth, Textual inversion, etc., but these fine-tuning methods only have good effects on a single topic, and fine-tuning for multiple topics and even specific fields needs to be explored. For example, in the insurance field, including but not limited to the financial field, in order to carry out business promotion and publicity, it is necessary to generate images containing insurance elements and health knowledge and medical knowledge based on text. If the original SD model is only adapted to generate images containing insurance knowledge, and now the SD model is adapted to images containing health knowledge and medical knowledge at the same time, it is difficult for the SD model to achieve the goal through traditional fine-tuning methods.

[0004] Therefore, how to fine-tune the SD model to adapt to corresponding new generation tasks including but not limited to multiple topics or specific fields has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides an image generation and processing method, apparatus, computer device and computer-readable storage medium based on a stable diffusion model, which can solve the technical problem that the SD model in traditional technology cannot efficiently adapt to new generation tasks through fine-tuning.

[0006] In the first aspect, the present application provides an image generation and processing method based on a stable diffusion model, including: determining the text required to generate a target image; encoding the text based on a preset first text encoder to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer; encoding the text encoding information based on a preset image generator to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer; generating a target image according to the image encoding information to obtain a generated image corresponding to the text.

[0007] In the second aspect, the present application provides an image generation and processing device based on a stable diffusion model, comprising: a first determination unit, used to determine the text required to generate a target image; a first encoding unit, used to encode the text based on a preset first text encoder to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer; a second encoding unit, used to encode the text encoding information based on a preset image generator to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer; a generation unit, used to generate a target image according to the image encoding information to obtain a generated image corresponding to the text.

[0008] In a third aspect, the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image generation and processing method based on the stable diffusion model are implemented.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the image generation processing method based on the stable diffusion model.

[0010] The present application provides an image generation processing method, device, computer equipment and computer-readable storage medium based on a stable diffusion model. The method adds corresponding low-rank adaptive layers to the text editor and diffusion model parts of the stable diffusion model, and trains the low-rank adaptive layers to support new generation tasks including but not limited to multiple topics or even specific fields, thereby increasing the text control ability of the stable diffusion model by means of the corresponding low-rank adaptive layers added to the text editor and diffusion model parts based on the stable diffusion model to explore the performance of the stable diffusion model on the new generation tasks, so that the stable diffusion model can support new generation tasks including but not limited to multiple topics and even specific fields, thereby enhancing the control ability of the stable diffusion model in text semantics, and realizing the adaptive and efficient fine-tuning of the stable diffusion model in performing multiple topics or complex topics, thereby helping to efficiently complete the new generation task requirements of the business. For example, in business scenarios including but not limited to finance, in order to promote financial business, through the above-mentioned scheme of the embodiment of the present application, on the basis of the original text-image generation model, new demand pictures containing special elements can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of a process flow of an image generation and processing method based on a stable diffusion model provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of a first sub-process of the image generation processing method based on the stable diffusion model provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of a training model flow of an image generation processing method based on a stable diffusion model provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of a second sub-process of the image generation and processing method based on the stable diffusion model provided in an embodiment of the present application;

[0016] Figure 5 A schematic block diagram of an image generation and processing device based on a stable diffusion model provided in an embodiment of the present application;

[0017] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] The embodiment of the present application provides an image generation and processing method based on a stable diffusion model, which can be applied to but not limited to computer devices such as smart phones, laptops, desktop computers, servers, smart billboards, and can be used to generate images based on text in fields including but not limited to financial technology.

[0021] Faced with the technical problem that the SD model in traditional technology cannot efficiently adapt to new generation tasks through fine-tuning, the inventor proposes an image generation processing method based on a stable diffusion model in the embodiment of the present application. The core idea of ​​the embodiment of the present application is: corresponding LORA layers are added to the text editor and diffusion model parts of the SD model respectively, and the LORA layer is trained separately with the help of an adversarial mechanism, and the LORA layer is set to support new generation tasks including but not limited to multiple topics and even specific fields. Therefore, on the basis of the original SD model, the text control ability of the SD model is increased with the help of the LORA layer to explore the performance of SD in new generation tasks. By further strengthening the control ability of the SD model in text semantics, the SD model can be adaptively and efficiently fine-tuned in performing multiple topics or complex topics, thereby helping to complete the new generation task requirements of the business.

[0022] Some embodiments of the present application are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0023] See also Figure 1 , Figure 1 The following is a flow chart of an image generation and processing method based on a stable diffusion model provided in an embodiment of the present application. Figure 1 As shown, the method includes but is not limited to the following steps S11-S14:

[0024] S11. Determine the text required to generate the target image.

[0025] Explanatory, in the case of generating an image based on text, the text required to generate the target image is first determined, and the text describes what kind of image is to be generated. For example, in business scenarios including but not limited to finance, in order to promote financial business, a large number of pictures containing special elements need to be generated. At this time, the special element is first described by text, and the text is the text required to generate the picture containing the special element.

[0026] S12. Encode the text based on a preset first text encoder to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-training model and a preset first low-rank adaptive layer.

[0027] Explanatory, a text encoder is pre-set, that is, a preset first text encoder, the preset first text encoder represents a model for encoding text, the preset first text encoder includes a preset multimodal pre-training model and a preset first low-rank adaptive layer, that is, the preset first text encoder is a text encoder based on the preset multimodal pre-training model, and the preset first text encoder adds a preset first low-rank adaptive layer on the basis of the preset multimodal pre-training model to fine-tune the preset multimodal pre-training model to adapt to the new generation task. Among them, the preset multimodal pre-training model represents a pre-training model of the corresponding modality between the cross-text and the image based on the text-image pair, and the preset multimodal pre-training model includes but is not limited to the pre-training model of contrastive text-image pairs (English is Contrastive Language-Image Pre-training, abbreviated as CLIP), CLIP embeds images and texts into the same semantic space by contrastive learning, so that the model can understand the semantic relationship between images and texts. The core idea of ​​the CLIP model is to pre-train a model that can understand images and texts at the same time by maximizing the consistency between image representations and their corresponding text descriptions. The preset first low-rank adaptive layer represents a low-rank adaptive layer. The "first" involved in the preset first low-rank adaptive layer is only used to distinguish different low-rank adaptive layers, and is not used to limit the low-rank adaptive layer. Other similar descriptions in this application are by analogy. Low-Rank adaptation (abbreviated as LoRA in English) includes but is not limited to LoRA itself and variants that improve the functions of LoRA in different ways, including but not limited to LoRA+, LoRA-fa, LoRA-drop, AdaLoRA, and Delta-LoRA.

[0028] Based on the preset first text encoder, the text is encoded, specifically, the text is first encoded through a preset multimodal pre-trained model, and then the encoded information is encoded through a preset first low-rank adaptive layer to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer, and the text encoding information encoded by the preset first low-rank adaptive layer has the fine-tuning function to adapt to the new generation task based on the implementation principle of the preset first low-rank adaptive layer itself, thereby increasing the text control ability of the SD model to explore the performance of SD on the new generation task.

[0029] S13. Based on a preset image generator, encode the text encoding information to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer.

[0030] Explanatory, a preset image generator, i.e., a preset image generator, is pre-set. The preset image generator represents a generator that generates an image according to the text encoding information corresponding to the text. The preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer, i.e., the preset image generator is a generator based on a preset stable diffusion model, and the preset image generator adds a preset second low-rank adaptive layer on the basis of the preset stable diffusion model to fine-tune the preset stable diffusion model to adapt to the new generation task. Among them, the preset second low-rank adaptive layer represents a low-rank adaptive layer, and the "second" involved in the preset second low-rank adaptive layer is only used to distinguish different low-rank adaptive layers, and is not used to limit the low-rank adaptive layer. Other similar descriptions in this application are analogous, and the preset second low-rank adaptive layer can refer to the description of the above-mentioned preset first low-rank adaptive layer. The preset stable diffusion model represents a stable diffusion model (Stable diffusion model in English, abbreviated as SD).

[0031] Based on the preset image generator, the text encoding information is encoded, specifically, the text encoding information is first encoded through a preset stable diffusion model, and then the encoded information is encoded through a preset second low-rank adaptive layer to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer, and the image encoding information encoded by the preset second low-rank adaptive layer has the fine-tuning function to adapt to new generation tasks based on the implementation principle of the preset second low-rank adaptive layer itself, thereby realizing the exploration of SD's performance on new generation tasks by increasing the text control ability of the SD model.

[0032] S14. Generate a target image according to the image coding information to obtain a generated image corresponding to the text.

[0033] Explanatory, according to the image coding information, and generating an image from the image coding information, a target image can be generated to obtain a generated image corresponding to the text. Exemplarily, in business scenarios including but not limited to finance, in order to promote financial services, through the above-mentioned scheme of the embodiment of the present application, on the basis of the original text-image generation model, the original text-image generation model can be fine-tuned to generate a new required image containing special elements, thereby helping to meet the promotion needs of financial services.

[0034] In the embodiment of the present application, the text required to generate the target image is determined, and the text is encoded based on a preset first text encoder including a preset multimodal pre-trained model and a preset first low-rank adaptive layer to obtain text encoding information, and then the text encoding information is encoded based on a preset image generator including a preset stable diffusion model and a preset second low-rank adaptive layer to obtain image encoding information, and the target image is generated according to the image encoding information to obtain the generated image corresponding to the text, thereby increasing the text control ability of the stable diffusion model by means of the corresponding low-rank adaptive layers added to the text editor based on the stable diffusion model and the diffusion model part to explore the performance of the stable diffusion model in the new generation task, so that the stable diffusion model can support but not limited to multiple topics and even new generation tasks in specific fields, thereby strengthening the control ability of the stable diffusion model in text semantics, and realizing the adaptive and efficient fine-tuning of the stable diffusion model in performing multiple topics or complex topics, thereby helping to efficiently complete the new generation task requirements of the business. For example, in business scenarios including but not limited to finance, in order to promote financial services, through the above-mentioned scheme of the embodiment of the present application, on the basis of the original text-image generation model, new demand pictures containing special elements can be generated.

[0035] In one embodiment, see Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of a first sub-process of the image generation processing method based on the stable diffusion model provided in an embodiment of the present application. Figure 3 A schematic diagram of a training model flow chart of an image generation processing method based on a stable diffusion model provided in an embodiment of the present application. Figure 2 and Figure 3 As shown, based on the preset first text encoder, encoding the text to obtain text encoding information also includes:

[0036] S21, determining a preset training sample, wherein the preset training sample includes a preset training text;

[0037] S22, encoding the preset training text based on a preset first text encoder to obtain first training text encoding information;

[0038] S23, encoding the first training text encoding information based on a preset image generator to obtain training image encoding information;

[0039] S24, generating a target training image according to the training image encoding information, and obtaining a training generated image corresponding to the preset training text;

[0040] S25. According to the preset training samples and the training generated images, train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer.

[0041] Explanatory, as described above, the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer, and the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer, that is, by adding a preset first low-rank adaptive layer on the basis of the preset multimodal pre-trained model, and adding a preset second low-rank adaptive layer on the basis of the preset stable diffusion model, the preset multimodal pre-trained model is fine-tuned to adapt to the new generation task, and the preset stable diffusion model is fine-tuned to adapt to the new generation task, thereby increasing the text control ability of the SD model to explore the performance of SD on new generation tasks.

[0042] Based on the above, when training the model, first determine the preset training samples, the preset training samples include the preset training text, then based on the preset first text encoder, encode the preset training text to obtain the first training text encoding information, then based on the preset image generator, encode the first training text encoding information to obtain the training image encoding information, then generate the target training image according to the training image encoding information, and obtain the training generated image corresponding to the preset training text, finally, according to the preset training samples and the training generated image, train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, so that on the basis of the original text-image generation model, the original original text-image generation model increases the text control ability to explore the performance of SD on the new generation task.

[0043] In an embodiment of the present application, a preset first low-rank adaptive layer is added on the basis of a preset multimodal pre-trained model, and a preset second low-rank adaptive layer is added on the basis of a preset stable diffusion model, so as to fine-tune the preset multimodal pre-trained model to adapt to the new generation task, and the preset stable diffusion model is fine-tuned to adapt to the new generation task, and the newly added preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained, thereby increasing the text control ability of the SD model to explore the performance of SD on the new generation task, so that the stable diffusion model can support new generation tasks including but not limited to multiple topics and even specific fields, and realize the adaptive and efficient fine-tuning of the stable diffusion model in performing multiple topics or complex topics, thereby helping to efficiently complete the new generation task requirements of the business.

[0044] In one embodiment, before training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training sample and the training generated image, the method further includes:

[0045] Freeze the parameters respectively included in the preset multimodal pre-trained model and the preset stable diffusion model.

[0046] Explanatoryly, according to the preset training samples and training generated images, before training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the parameters contained in the preset multimodal pre-trained model and the preset stable diffusion model are frozen, so that the preset multimodal pre-trained model and the preset stable diffusion model can no longer be trained or changed, and only the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained. Only by adding and training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the adaptive and efficient fine-tuning of the stable diffusion model on multiple topics or complex topics is achieved, and the original good encoding capabilities of the preset multimodal pre-trained model and the preset stable diffusion model can be retained.

[0047] In the embodiment of the present application, by freezing the parameters contained in the preset multimodal pre-trained model and the preset stable diffusion model, when training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, not only can the good performance of the already trained preset multimodal pre-trained model and the preset stable diffusion model be retained, but also because only the preset first low-rank adaptive layer and the preset second low-rank adaptive layer need to be trained, the training efficiency of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be improved compared to training all parameters, thereby achieving high efficiency of adaptive fine-tuning of the stable diffusion model on multiple topics or complex topics, which in turn helps to efficiently complete the new generated task requirements of the business.

[0048] In one embodiment, the preset training sample includes a preset training image corresponding to the preset training text; and training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training sample and the training generated image includes:

[0049] Calculating a first loss value corresponding to an image loss between the training generated image and the preset training image;

[0050] According to the first loss value, the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained.

[0051] Explanatoryly, a first loss value corresponding to the image loss between the training generated image and the preset training image is calculated, and based on the first loss value, a preset first low-rank adaptive layer and a preset second low-rank adaptive layer are trained, wherein the loss function corresponding to the image loss between the training generated image and the preset training image can adopt an LDM loss function including but not limited to the LDM loss function, and the LDM loss function represents the loss function of the latent space diffusion model (Latent Diffusion Models, abbreviated as LDM).

[0052] In an embodiment of the present application, by calculating a first loss value corresponding to the image loss between a training generated image and a preset training image, a preset first low-rank adaptive layer and a preset second low-rank adaptive layer are trained according to the first loss value, and with the help of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the stable diffusion model can be enabled to support new generation tasks including but not limited to multiple topics and even specific fields, and the adaptive and efficient fine-tuning of the stable diffusion model on multiple topics or complex topics can be achieved, thereby helping to efficiently complete the new generation task requirements of the business.

[0053] In one embodiment, see Figure 4 , Figure 4 This is a schematic diagram of the second sub-flow of the image generation processing method based on the stable diffusion model provided in the embodiment of the present application. Figure 4 As shown, in this embodiment, according to the preset training sample and the training generated image, training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer includes:

[0054] S41, encoding the training generated image based on a preset multimodal pre-trained image encoder included in a preset discriminator model to obtain generated image encoding information;

[0055] S42, encoding the preset training text based on the preset second text encoder included in the preset discriminator model to obtain second training text encoding information;

[0056] S43, based on a preset image discriminator, calculating an adversarial loss value corresponding to the adversarial loss between the generated image encoding information and the second training text encoding information;

[0057] S44. According to the adversarial loss value, train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer.

[0058] Explanatory, please continue to see Figure 3 and Figure 4 ,like Figure 3 and Figure 4As shown, a discriminator model is preset, that is, a preset discriminator model. The preset discriminator model uses an adversarial network to discriminate the generated target training image. The preset discriminator model includes a preset multimodal pre-trained image encoder, a preset second text encoder, and a preset image discriminator. The preset multimodal pre-trained image encoder is used to encode the generated target training image to obtain a multi-dimensional image encoding feature. The preset multimodal pre-trained image encoder is an image encoder including but not limited to a CLIP-based image encoder. The preset second text encoder is used to encode the preset training text. The preset second text encoder is a text encoder including but not limited to a CLIP-based text encoder. The preset image discriminator The image discriminator can adopt but is not limited to ResNet50. ResNet50 is a deep convolutional neural network with 50 layers, which is mainly used for image recognition and classification tasks. The preset image discriminator adopts the adversarial principle in the generative adversarial principle to identify and classify the target training images, and the outputs of the preset multimodal pre-trained image encoder and the preset second text encoder are respectively input into the image discriminator, and the image discriminator identifies the target training image based on the above two and calculates the adversarial loss, and feeds back the adversarial loss to the preset first low-rank adaptive layer and the preset second low-rank adaptive layer to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer.

[0059] According to the setting of the above-mentioned preset discriminator model, based on the preset multimodal pre-trained image encoder included in the preset discriminator model, the training generated image is encoded to obtain generated image encoding information, and based on the preset second text encoder included in the preset discriminator model, the preset training text is encoded to obtain second training text encoding information, and then based on the image discriminator, the adversarial loss value corresponding to the adversarial loss between the generated image encoding information and the second training text encoding information is calculated to distinguish the authenticity of the target training image and obtain the adversarial loss value, and then according to the adversarial loss value, the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained.

[0060] In the embodiment of the present application, an image discriminator based on an adversarial network is used to distinguish the authenticity of the generated target training images, and the corresponding adversarial loss is used to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer. The preset first text encoder including the preset first low-rank adaptive layer is based on the preset multimodal pre-training model, and the preset image generator including the preset second low-rank adaptive layer is gradually optimized based on the preset stable diffusion model to generate more realistic, closer to reality, and more diverse target training images, and the text control ability of the stable diffusion model is further increased to explore the performance of the stable diffusion model in new generation tasks, so that the stable diffusion model can support new generation tasks including but not limited to multiple topics and even specific fields, and realize the adaptive and efficient fine-tuning of the stable diffusion model in performing multiple topics or complex topics, thereby helping to efficiently complete the new generation task requirements of the business.

[0061] In one embodiment, based on a preset multimodal pre-trained image encoder included in a preset discriminator model, before encoding the training generated image, the method further includes:

[0062] Freeze the parameters contained in the preset discriminator model.

[0063] Explanatory, since the purpose of the embodiment of the present application is to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the preset discriminator does not need to be trained. Before encoding the trained generated image based on the preset multimodal pre-trained image encoder contained in the preset discriminator model, the parameters contained in the preset discriminator model are frozen, so that the preset discriminator model can no longer be trained. The preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained only with the help of the preset discriminator model to improve the efficiency and effect of training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, and then based on the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the adaptive and efficient fine-tuning of the stable diffusion model on multiple topics or complex topics is achieved.

[0064] In an embodiment of the present application, by freezing the parameters included in the preset discriminator model, the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained only with the help of the preset discriminator model. When training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the training efficiency of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be improved, thereby achieving high efficiency in adaptive fine-tuning of the stable diffusion model on multiple topics or complex topics, and thus helping to efficiently complete the new generation task requirements of the business.

[0065] In one embodiment, based on a preset multimodal pre-trained image encoder included in a preset discriminator model, encoding the training generated image to obtain the generated image encoding information further includes:

[0066] Based on a preset intermediate layer included in the preset image generator, outputting corresponding training image coding features from the preset intermediate layer;

[0067] The training image encoding features are used as generated training images to obtain training generated images.

[0068] Explanatory, based on several preset intermediate layers included in the preset image generator, the corresponding training image coding features are output from each preset intermediate layer. Since the preset image generator includes several preset intermediate layers, training image coding features of different dimensions are output, and the training image coding features are used as the generated training image to obtain a training generated image. The training generated image includes the training image coding features output by the preset intermediate layer and the target training image generated by the preset image generator. The training generated image is input into the preset discriminator model, and the training image coding features of different dimensions are encoded based on the preset multimodal pre-trained image encoder included in the preset discriminator model. Then, the authenticity of the training generated image is identified according to the preset training samples, and adversarial Loss supervised image generation is used to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer.

[0069] In an embodiment of the present application, a preset intermediate layer included in a preset image generator is used to output corresponding training image coding features from the preset intermediate layer, and the training image coding features are used as generated training images to obtain training generated images. Since the target training image is also generated based on the training image coding features, the training image coding features are used as the generated training images to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer together with the target training image. The preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be adapted to training generated images of different dimensions, and the robustness and generalization ability of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be further improved, thereby increasing the text control ability of the SD model to explore the performance of SD in new generation tasks, thereby achieving adaptive fine-tuning of the stable diffusion model in performing multiple topics or complex topics, which is helpful to efficiently complete the new generation task requirements of the business.

[0070] In one embodiment, based on a preset first text encoder, encoding the preset training text to obtain first training text encoding information includes:

[0071] Based on a preset first text encoder, encoding the preset training text to obtain initial training text encoding information;

[0072] Random Gaussian noise is added to the initial training text encoding information to obtain first training text encoding information.

[0073] Explanatoryly, Gaussian noise, also known as white noise, is a random noise that obeys a normal distribution.

[0074] Based on a preset first text encoder, the preset training text is encoded to obtain initial training text encoding information, and then random Gaussian noise is added to the initial training text encoding information to expand the initial training text encoding information to obtain first training text encoding information, and the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained according to the first training text encoding information, and the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are forced to learn features that are robust to small changes corresponding to the random Gaussian noise contained in the first training text encoding information.

[0075] In an embodiment of the present application, random Gaussian noise is added to the training text encoding information corresponding to the preset training text to obtain the first training text encoding information, and the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained according to the first training text encoding information. The robustness and generalization ability of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be improved, thereby achieving adaptive fine-tuning of the stable diffusion model on multiple topics or complex topics, which helps to efficiently complete the new generation task requirements of the business.

[0076] In one embodiment, determining a preset training sample includes:

[0077] Determine n preset training topics, and determine m pairs of training text images corresponding to each of the preset training topics and corresponding preset training texts to obtain preset training samples, wherein n and m are natural numbers.

[0078] Explanatory, n preset training topics are determined, and m pairs of preset training pictures and corresponding preset training texts corresponding to each preset training topic are determined to form training text image pairs, and preset training samples are obtained. For example, n subtopics included in the preset topic are determined, 5-10 training sample pictures are prepared for each subtopic, and corresponding text descriptions are edited for the pictures, and the images and corresponding texts are combined into training text image pairs to obtain n subtopics included in the preset topic and corresponding training text image pairs, and then the training text image pairs are used as preset training samples to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, so that the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can adapt to the n subtopics included in the preset topic, thereby increasing the text control ability of the SD model on the n subtopics included in the preset topic to explore the performance of SD on the new generation tasks corresponding to the n subtopics included in the preset topic.

[0079] In the embodiment of the present application, by using n preset training topics and training text image pairs corresponding to each preset training topic as preset training samples to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer, the robustness and generalization ability of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer can be further improved, thereby increasing the text control ability of the SD model to explore the performance of SD on new generation tasks, thereby achieving adaptive fine-tuning of the stable diffusion model in performing multiple topics or complex topics, which helps to efficiently complete the new generation task requirements of the business.

[0080] It should be noted that the image generation and processing methods based on the stable diffusion model described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but they are all within the protection scope required by this application.

[0081] See also Figure 5 , Figure 5 The following is a schematic block diagram of an image generation processing device based on a stable diffusion model provided in an embodiment of the present application. Corresponding to the above-mentioned image generation processing method based on a stable diffusion model, an embodiment of the present application also provides an image generation processing device based on a stable diffusion model. Figure 5 As shown, the image generation processing device based on the stable diffusion model includes a unit for executing the above-mentioned image generation processing method based on the stable diffusion model. The image generation processing device based on the stable diffusion model can be configured in a computer device. Specifically, please refer to Figure 5 The image generation processing device 50 based on the stable diffusion model includes a first determining unit 51, a first encoding unit 52, a second encoding unit 53 and a generating unit 54.

[0082] Wherein, the first determining unit 51 is used to determine the text required to generate the target image;

[0083] A first encoding unit 52 is used to encode the text based on a preset first text encoder to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer;

[0084] A second encoding unit 53 is used to encode the text encoding information based on a preset image generator to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer;

[0085] The generating unit 54 is used to generate a target image according to the image coding information, and obtain a generated image corresponding to the text.

[0086] In one embodiment, the image generation processing device 50 based on the stable diffusion model further includes:

[0087] A second determining unit, configured to determine a preset training sample, wherein the preset training sample includes a preset training text;

[0088] A third encoding unit, configured to encode the preset training text based on a preset first text encoder to obtain first training text encoding information;

[0089] a fourth encoding unit, configured to encode the first training text encoding information based on a preset image generator to obtain training image encoding information;

[0090] A second generating unit, configured to generate a target training image according to the training image encoding information, and obtain a training generated image corresponding to the preset training text;

[0091] A training unit is used to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training samples and the training generated image.

[0092] In one embodiment, the image generation processing device 50 based on the stable diffusion model further includes:

[0093] The first freezing unit is used to freeze the parameters respectively included in the preset multimodal pre-training model and the preset stable diffusion model.

[0094] In one embodiment, the training unit comprises:

[0095] A first encoding subunit, configured to encode the training generated image based on a preset multimodal pre-trained image encoder included in a preset discriminator model to obtain generated image encoding information;

[0096] A second encoding subunit is used to encode the preset training text based on a preset second text encoder included in the preset discriminator model to obtain second training text encoding information;

[0097] A calculation subunit, configured to calculate, based on a preset image discriminator, an adversarial loss value corresponding to the adversarial loss between the generated image encoding information and the second training text encoding information;

[0098] The first training subunit is used to train the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the adversarial loss value.

[0099] In one embodiment, the training unit further includes:

[0100] The first freezing subunit is used to freeze the parameters included in the preset discriminator model.

[0101] In one embodiment, the training unit further includes:

[0102] An output subunit, configured to output corresponding training image coding features from a preset intermediate layer based on the preset intermediate layer included in the preset image generator;

[0103] The first determination subunit is used to use the training image encoding features as a generated training image to obtain a training generated image.

[0104] In one embodiment, the third encoding unit includes:

[0105] A third encoding subunit is used to encode the preset training text based on a preset first text encoder to obtain initial training text encoding information;

[0106] The adding subunit is used to add random Gaussian noise to the initial training text encoding information to obtain first training text encoding information.

[0107] It should be noted that technicians in the relevant field can clearly understand that the specific implementation process of the above-mentioned image generation and processing device based on the stable diffusion model and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0108] Meanwhile, the division and connection method of each unit in the above-mentioned image generation and processing device based on the stable diffusion model is only used for illustration. In other embodiments, the image generation and processing device based on the stable diffusion model can be divided into different units as needed, and each unit in the image generation and processing device based on the stable diffusion model can also be connected in a different order and method to complete all or part of the functions of the above-mentioned image generation and processing device based on the stable diffusion model.

[0109] The image generation processing device based on the stable diffusion model can be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0110] See also Figure 6 , Figure 6 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a desktop computer or a server, or may be a component or part in other devices.

[0111] See also Figure 6The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0112] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the above-mentioned image generation processing method based on the stable diffusion model.

[0113] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0114] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned image generation processing method based on the stable diffusion model.

[0115] The network interface 505 is used to communicate with other devices over the network. Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 6 The embodiments shown are consistent and will not be described again here.

[0116] In which, the processor 502 is used to run a computer program 5032 stored in the memory to implement the following steps: determine the text required to generate a target image; based on a preset first text encoder, encode the text to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer; based on a preset image generator, encode the text encoding information to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer; generate a target image according to the image encoding information to obtain a generated image corresponding to the text.

[0117] In one embodiment, before encoding the text based on a preset first text encoder to obtain text encoding information, the processor 502 further implements the following steps:

[0118] Determining a preset training sample, wherein the preset training sample includes a preset training text;

[0119] Based on a preset first text encoder, encoding the preset training text to obtain first training text encoding information;

[0120] Based on a preset image generator, encoding the first training text encoding information to obtain training image encoding information;

[0121] Generate a target training image according to the training image encoding information, and obtain a training generated image corresponding to the preset training text;

[0122] The preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained according to the preset training samples and the training generated images.

[0123] In one embodiment, before implementing the training of the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training sample and the training generated image, the processor 502 further implements the following steps:

[0124] Freeze the parameters respectively included in the preset multimodal pre-trained model and the preset stable diffusion model.

[0125] In one embodiment, when the processor 502 implements training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training sample and the training generated image, the processor 502 specifically implements the following steps:

[0126] Based on a preset multimodal pre-trained image encoder included in a preset discriminator model, encoding the training generated image to obtain generated image encoding information;

[0127] Based on a preset second text encoder included in a preset discriminator model, encoding the preset training text to obtain second training text encoding information;

[0128] Based on a preset image discriminator, calculating an adversarial loss value corresponding to the adversarial loss between the generated image encoding information and the second training text encoding information;

[0129] According to the adversarial loss value, the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained.

[0130] In one embodiment, the processor 502 further implements the following steps before implementing the preset multimodal pre-trained image encoder included in the preset discriminator model to encode the training generated image:

[0131] Freeze the parameters contained in the preset discriminator model.

[0132] In one embodiment, the processor 502 further implements the following steps before implementing a preset multimodal pre-trained image encoder included in a preset discriminator model to encode the training generated image to obtain the generated image encoding information:

[0133] Based on a preset intermediate layer included in the preset image generator, outputting corresponding training image coding features from the preset intermediate layer;

[0134] The training image encoding features are used as generated training images to obtain training generated images.

[0135] In one embodiment, when the processor 502 encodes the preset training text based on the preset first text encoder to obtain the first training text encoding information, the processor 502 specifically implements the following steps:

[0136] Based on a preset first text encoder, encoding the preset training text to obtain initial training text encoding information;

[0137] Random Gaussian noise is added to the initial training text encoding information to obtain first training text encoding information.

[0138] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0139] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0140] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0141] A computer program product, when running on a computer, enables the computer to execute the steps of the image generation processing method based on the stable diffusion model described in the above embodiments.

[0142] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0147] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0149] The software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An image generation and processing method based on a stable diffusion model, characterized in that: include: Determine the text required to generate the target image; Based on a preset first text encoder, encoding the text to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer; Based on a preset image generator, the text encoding information is encoded to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer; A target image is generated according to the image encoding information to obtain a generated image corresponding to the text.

2. The image generation processing method based on the stable diffusion model according to claim 1 is characterized in that: Based on the preset first text encoder, encoding the text to obtain text encoding information, the method further includes: Determining a preset training sample, wherein the preset training sample includes a preset training text; Based on a preset first text encoder, encoding the preset training text to obtain first training text encoding information; Based on a preset image generator, encoding the first training text encoding information to obtain training image encoding information; Generate a target training image according to the training image encoding information, and obtain a training generated image corresponding to the preset training text; The preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained according to the preset training samples and the training generated images.

3. The image generation processing method based on the stable diffusion model according to claim 2 is characterized in that: Before training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer according to the preset training sample and the training generated image, the method further includes: Freeze the parameters respectively included in the preset multimodal pre-trained model and the preset stable diffusion model.

4. The image generation processing method based on the stable diffusion model according to claim 2 or 3, characterized in that: According to the preset training sample and the training generated image, training the preset first low-rank adaptive layer and the preset second low-rank adaptive layer includes: Based on a preset multimodal pre-trained image encoder included in a preset discriminator model, encoding the training generated image to obtain generated image encoding information; Based on a preset second text encoder included in a preset discriminator model, encoding the preset training text to obtain second training text encoding information; Based on a preset image discriminator, calculating an adversarial loss value corresponding to the adversarial loss between the generated image encoding information and the second training text encoding information; According to the adversarial loss value, the preset first low-rank adaptive layer and the preset second low-rank adaptive layer are trained.

5. The image generation processing method based on the stable diffusion model according to claim 4 is characterized in that: Based on the preset multimodal pre-trained image encoder included in the preset discriminator model, before encoding the training generated image, it also includes: Freeze the parameters contained in the preset discriminator model.

6. The image generation processing method based on the stable diffusion model according to claim 4 is characterized in that: Based on the preset multimodal pre-trained image encoder included in the preset discriminator model, encoding the training generated image to obtain the generated image encoding information, it also includes: Based on a preset intermediate layer included in the preset image generator, outputting corresponding training image coding features from the preset intermediate layer; The training image encoding features are used as generated training images to obtain training generated images.

7. The image generation and processing method based on the stable diffusion model according to claim 2 or 3, characterized in that: Based on a preset first text encoder, encoding the preset training text to obtain first training text encoding information includes: Based on a preset first text encoder, encoding the preset training text to obtain initial training text encoding information; Random Gaussian noise is added to the initial training text encoding information to obtain first training text encoding information.

8. An image generation and processing device based on a stable diffusion model, characterized in that: include: A first determining unit, used to determine the text required to generate a target image; A first encoding unit, configured to encode the text based on a preset first text encoder to obtain text encoding information, wherein the preset first text encoder includes a preset multimodal pre-trained model and a preset first low-rank adaptive layer; A second encoding unit, configured to encode the text encoding information based on a preset image generator to obtain image encoding information, wherein the preset image generator includes a preset stable diffusion model and a preset second low-rank adaptive layer; A generating unit is used to generate a target image according to the image coding information to obtain a generated image corresponding to the text.

9. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images