Picture generation model processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the field of artificial intelligence, and relates to a picture generation model processing method, which comprises the following steps: acquiring a training picture and a training prompt text corresponding to the training picture; obtaining a preset training guidance mode, wherein the training guidance mode comprises classifier-free guidance and CLIP guidance; alternately using the training guide mode to train an initial picture generation model to obtain a picture generation model; and obtaining a prompt text, and inputting the prompt text into the picture generation model to obtain a target picture. The invention further provides a picture generation model processing device, computer equipment and a storage medium. According to the invention, the picture generation accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, computer equipment and storage medium for processing an image generation model. Background Art

[0002] Existing image generation technologies can generate images based on natural language under the guidance of text, and achieve diversity and semantic relevance to a certain extent. However, when faced with complex text prompts, existing image generation technologies often find it difficult to simultaneously generate images that meet both global consistency and high fidelity of details. For generation tasks that require precise details, such as light and shadow effects and object textures, the model is insufficient in expressiveness, and the generated images may lack natural detail processing and layering, and have poor quality. For complex text prompts, existing image generation technologies also have difficulty in accurately capturing semantics, all of which lead to low accuracy in image generation. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for processing an image generation model to solve the problem of poor image generation quality.

[0004] In order to solve the above technical problems, the present application embodiment provides a method for processing an image generation model, which adopts the following technical solution:

[0005] Get the training images and their corresponding training prompt texts;

[0006] Obtaining a preset training guidance mode, wherein the training guidance mode includes a non-classifier guidance and a CLIP guidance;

[0007] Alternately using the training guidance method to train the initial image generation model to obtain an image generation model;

[0008] Obtain a prompt text, and input the prompt text into the image generation model to obtain a target image.

[0009] In order to solve the above technical problems, the embodiment of the present application further provides a picture generation model processing device, which adopts the following technical solution:

[0010] A training acquisition module is used to acquire training images and their corresponding training prompt texts;

[0011] A mode acquisition module, used to acquire a preset training guidance mode, wherein the training guidance mode includes a classifier-free guidance and a CLIP guidance;

[0012] A model training module, used to alternately use the training guidance method to train the initial image generation model to obtain an image generation model;

[0013] The image generation module is used to obtain the prompt text and input the prompt text into the image generation model to obtain the target image.

[0014] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0015] Get the training images and their corresponding training prompt texts;

[0016] Obtaining a preset training guidance mode, wherein the training guidance mode includes a non-classifier guidance and a CLIP guidance;

[0017] Alternately using the training guidance method to train the initial image generation model to obtain an image generation model;

[0018] Obtain a prompt text, and input the prompt text into the image generation model to obtain a target image.

[0019] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0020] Get the training images and their corresponding training prompt texts;

[0021] Obtaining a preset training guidance mode, wherein the training guidance mode includes a non-classifier guidance and a CLIP guidance;

[0022] Alternately using the training guidance method to train the initial image generation model to obtain an image generation model;

[0023] Obtain a prompt text, and input the prompt text into the image generation model to obtain a target image.

[0024] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: obtaining training samples and their corresponding training prompt texts provides a rich semantic context for the model, ensuring the accuracy of image generation; in training, alternating between classifier-free guidance and CLIP guidance enhances the self-learning ability of the model, improves the flexibility of the image generation process, and ensures that the generated images achieve higher fidelity in diversity and detail; the model training process also combines the advantages of different guidance strategies, making the generated images more consistent and realistic in complex scenes; when applied, high-quality target images can be obtained through simple text prompts, which simplifies the image generation process, expands the application scenarios of the generation model, and improves the accuracy of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0027] Figure 2 is a flowchart of an embodiment of a method for processing a picture generation model according to the present application;

[0028] Figure 3 is a structural schematic diagram of an embodiment of a picture generation model processing device according to the present application;

[0029] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0031] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0034] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0035] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0036] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0037] It should be noted that the image generation model processing method provided in the embodiment of the present application is generally executed by a server, and accordingly, the image generation model processing device is generally set in the server.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0039] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for processing a picture generation model according to the present application. The method for processing a picture generation model comprises the following steps:

[0040] Step S201, obtaining a training image and its corresponding training prompt text.

[0041] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can communicate with the terminal device through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0042] Specifically, the server obtains the training image and its corresponding training prompt text. The training prompt text is a text description of the training image, which provides the model with semantic information that needs to be followed when generating images.

[0043] The server can obtain multiple training images and their corresponding training prompt texts to ensure that the model can learn from diverse samples and improve its generation ability.

[0044] Step S202, obtaining a preset training guidance mode, where the training guidance mode includes non-classifier guidance and CLIP guidance.

[0045] Specifically, the guidance methods used in the model training process are obtained, including classifier-free guidance and CLIP guidance. These two methods provide different learning strategies, allowing the model to generate and optimize images based on its own knowledge without an independent classifier, while also taking advantage of the CLIP model to guide the classifier and improve the consistency between images and texts.

[0046] Classifier-free guidance is a form of guidance in the diffusion model that does not rely on a separate classifier. Classifier guidance requires training a separate classifier to guide image generation based on text cues. Classifier-free guidance implements classifier-free conditional guidance by replacing text captions with empty sequences during training. This allows the diffusion model to fully leverage its own knowledge during guidance without relying on external classifiers. Second, it simplifies the process of handling unpredictable information and avoids relying on classifiers for complex conditional guidance. Classifier-free guidance increases the flexibility of the diffusion model, enabling it to generate high-quality images consistent with text cues without labels or classifiers.

[0047] CLI (Contrasting Language-Image Pre-training) guidance is a technique for matching images and texts, which is used to improve the image generation quality of the diffusion model. The CLI model consists of an image encoder and a text encoder, which is trained on a large-scale dataset through contrastive learning and can match images with corresponding text descriptions. In the diffusion model, CLI guidance is used for classifier guidance, that is, the CLI model is used to calculate the similarity between the image and the text prompt during the generation process, and the image generation is guided by gradients to make it closer to the text description. By introducing a noise-trained CLI model (i.e., a noisy CLI model), CLI guidance can generate more robust and credible images in noisy and complex environments.

[0048] Step S203, alternately using training guidance methods to train the initial image generation model to obtain an image generation model.

[0049] Specifically, two training guidance methods are used alternately to ensure that the model is fully trained under different guidance strategies, so that it can flexibly adapt to complex text prompts and improve the diversity and detail fidelity of generated images.

[0050] During model training, the model parameters are adjusted by comparing the generated images with the training images to optimize the image generation quality.

[0051] In the present application, the initial image generation model can be constructed based on the diffusion model.

[0052] Furthermore, the above-mentioned step S203 may include: inputting the training picture and its corresponding training prompt text into the initial picture generation model, and generating a first picture according to the training prompt text through the initial picture generation model without using an independent classifier; calculating a first loss according to the first picture and the training picture; inputting the training picture and its corresponding training prompt text into the initial picture generation model, and guiding the classifier through CLIP to obtain a second picture generated by the initial picture generation model; calculating a second loss according to the second picture and the training picture; calculating a joint loss according to the first loss and the second loss, and adjusting the initial picture generation model according to the joint loss until the joint loss is less than a preset loss threshold to obtain a picture generation model.

[0053] Specifically, the acquired training image and its corresponding training prompt text are input into the initial image generation model for non-classifier guided training. The model generates the first image according to the training prompt text without relying on an independent classifier. Non-classifier guidance strengthens the self-learning ability of the model, enabling it to understand and generate images that conform to the text description in the absence of external guidance.

[0054] After the first image is generated, it is compared with the training image to calculate the first loss. The quality of model generation is evaluated by comparing the difference between the generated first image and the real training image.

[0055] The same training images and training prompt text are input into the initial generative model, and the CLIP guidance method is used to utilize the powerful semantic understanding ability of the CLIP model to ensure that the generated second image is more visually consistent with the text prompt.

[0056] Compare the second image with the training image and calculate the second loss.

[0057] The first loss and the second loss are combined, for example, weighted calculation is performed on the first loss and the second loss to calculate the joint loss. The joint loss is used to adjust the initial generation model until it is reduced to less than a preset loss threshold. Through this iterative optimization process, the model gradually improves its generation ability and obtains an accurate image generation model.

[0058] In this embodiment, by alternately using classifier-free guidance and CLIP guidance, the learning ability of the model is enhanced, and the overall improvement of the quality of generated images is achieved; the first image obtains loss feedback through the intrinsic self-learning mechanism, which prompts the model to be continuously optimized; the second image generated through CLIP guidance further ensures the consistency between the image and the text prompt at the semantic level; the joint loss integrates the two guidance strategies, so that the model achieves a balance between diversity and detail fidelity, can generate higher quality images, improve its performance under complex text prompts, and improve the accuracy of image generation.

[0059] Furthermore, the above-mentioned step of generating the first image according to the training prompt text through the initial image generation model without using an independent classifier may include: removing the text content of the training prompt text to obtain the first prompt text, and the first prompt text includes empty text; generating the first image according to the first prompt text through the initial image generation model without using an independent classifier.

[0060] Specifically, the obtained training prompt text is subjected to content elimination to generate the first prompt text. The elimination process can be to remove certain specific descriptions, modifiers, or directly convert the text into empty text, so that the generative model can learn how to generate images without clear prompts or too little information. In this way, the model will train its self-generation ability and gradually adapt to generating images with reasonable structure and details in the absence of detailed information.

[0061] After receiving the first prompt text, the first picture is generated according to the first prompt text without the guidance of an independent classifier, so that the model can rely on its own knowledge base and generation mechanism to construct an image related to the first prompt text in the process of autonomous learning. This generation method emphasizes the self-conditioning ability of the model and ensures that it can remain creative in a diverse prompt environment.

[0062] In this embodiment, by eliminating the content in the training prompt text, the initial image generation model is prompted to conduct self-learning and generation in the case of minimal prompts or empty prompts, thereby improving the model's self-conditioning ability, enabling it to generate images with reasonable structure and details when there is insufficient information; effectively improving its adaptability and creativity in diverse situations, reducing dependence on specific prompt texts, thereby improving the flexibility and effectiveness of image generation, and enabling the model to generate higher quality images when faced with complex or ambiguous prompts.

[0063] Furthermore, the above-mentioned step of guiding the classifier through CLIP to obtain the second image generated by the initial image generation model may include: obtaining the second image generated by the initial image generation model; encoding the second image through the CLIP model to obtain an image code, and encoding the training prompt text to obtain a text code; calculating the dot product of the image code and the text code; generating gradient information according to the dot product, and instructing the initial image generation model to continue generating the second image through the gradient information and the training image.

[0064] Specifically, the initial image generation model can be built based on the diffusion model. Noise is introduced during the training process of the diffusion model to generate and optimize noisy images. The diffusion model gradually removes noise through the diffusion process, generates images multiple times, and makes the generated images gradually approach the semantics of the text prompt.

[0065] A second image generated by the initial image generation model is obtained, and the image is encoded using the CLIP model to generate an image code. At the same time, the training prompt text is encoded to obtain a text code.

[0066] The dot product of the above-mentioned image encoding and text encoding is calculated. The dot product can reflect the similarity between the second image and the training prompt text. According to the dot product value, gradient information is further generated, and the gradient information indicates the adjustment direction of the model when generating the image. In this way, the initial image generation model can be adjusted according to the gradient information and the original training image, and continue to optimize the generated second image.

[0067] In this embodiment, by introducing the encoding and dot product calculation of the CLIP model, classifier guidance is implemented for the initial image generation model; by encoding and comparing the generated second image and the training prompt text, the semantic correlation between the generated image and the text can be accurately captured, the image generation process can be improved, the generated image can be made more in line with the requirements of complex prompts, and the model's ability to accurately capture details and semantics can be improved; through gradient information, the adjustment of the model in the generation process becomes more flexible and efficient, thereby improving the overall expressiveness and diversity of the generated image.

[0068] Furthermore, the above-mentioned step of generating gradient information according to the dot product and instructing the initial image generation model to continue to generate the second image through the gradient information and the training image may include: calculating the gradient information according to the dot product, wherein the dot product is negatively correlated with the gradient information; when denoising is performed according to the training image through the initial image generation model, adjusting the pixel value of the generated image according to the gradient information to obtain the second image.

[0069] Specifically, the gradient information is generated according to the dot product value. The dot product is negatively correlated with the gradient information. The higher the dot product value, the higher the similarity between the generated image and the text prompt, and the smaller the adjustment of the model; conversely, if the dot product value is low, it means that the generated image and the prompt text are poorly matched, and the model will make more significant adjustments to improve the generation quality.

[0070] In the process of denoising through the initial image generation model, the gradient information is used to adjust the pixel values ​​of the generated image, thereby improving the quality of the generated image so that it is more in line with the standards of the training image in terms of details and overall structure. By dynamically adjusting the pixel values, the model can be gradually optimized and ultimately obtain a higher quality second image.

[0071] In this embodiment, a negative correlation design is performed on the gradient information of the dot product calculation, which effectively guides the adjustment strategy of the image generation model during the optimization process; when the matching degree between the generated image and the text prompt is low, the adjustment force is increased to make the generated image gradually approach the target, thereby improving the details and overall quality of the generated image and making the generation process more adaptable and flexible; combined with the denoising process, the model can reduce the interference of noise in the generation stage, enhance the clarity and realism of the image, and improve the performance of the generated image under complex semantics.

[0072] Step S204, obtaining a prompt text, and inputting the prompt text into a picture generation model to obtain a target picture.

[0073] Specifically, in the model application stage, new prompt text is obtained and input into the image generation model, and the model generates the target image based on the prompt text.

[0074] In this embodiment, training samples and their corresponding training prompt texts are obtained to provide a rich semantic context for the model, ensuring the accuracy of image generation; in training, classifier-free guidance and CLIP guidance are used alternately to enhance the self-learning ability of the model, improve the flexibility of the image generation process, and ensure that the generated images achieve higher fidelity in diversity and details; the model training process also combines the advantages of different guidance strategies, making the generated images more consistent and realistic in complex scenes; when applied, high-quality target images can be obtained through simple text prompts, which simplifies the image generation process, expands the application scenarios of the generation model, and improves the accuracy of image generation.

[0075] Furthermore, after the above step S203, the method may further include: obtaining an image editing training set, the image editing training set including a second training image and its corresponding image description text; and fine-tuning the image generation model according to the image editing training set.

[0076] Specifically, a specialized image editing training set is obtained. The training set consists of second training pictures and their corresponding image description texts, with the purpose of providing the model with diverse training materials and enhancing its performance in specific editing tasks. The second training pictures usually contain a variety of styles, content, and details, which can help the model better understand and handle different types of image generation and editing tasks.

[0077] The model is fine-tuned using the image editing training set to further optimize the model's ability to understand complex text descriptions, allowing it to more accurately generate images that meet user needs. During the fine-tuning process, the model will adjust its internal parameters to improve the generation effect by comparing the differences between the generated image and the real image. In this way, the model can not only achieve higher accuracy and detail when generating images, but also flexibly adapt to different image editing needs and achieve realistic editing of existing images.

[0078] In this embodiment, fine-tuning is performed through the image editing training set to enhance the flexibility and adaptability of the image generation model; fine-tuning allows the model to further learn based on a variety of images and corresponding texts, and improves the details and style adaptability of its generated images, so that the model can effectively capture the subtle differences in image descriptions and achieve higher generation quality when faced with complex and changing user needs; fine-tuning also promotes the model's understanding of different styles and effects, so that it can generate more accurate and expressive images in practical applications.

[0079] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned training images and their corresponding training prompt texts, the above-mentioned training images and their corresponding training prompt texts can also be stored in a node of a blockchain.

[0080] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0081] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0082] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0083] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0084] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0085] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for processing a picture generation model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0086] like Figure 3 As shown, the image generation model processing device 300 described in this embodiment includes: a training acquisition module 301, a method acquisition module 302, a model training module 303 and an image generation module 304, wherein:

[0087] The training acquisition module 301 is used to acquire training pictures and their corresponding training prompt texts.

[0088] The mode acquisition module 302 is used to acquire a preset training guidance mode, where the training guidance mode includes non-classifier guidance and CLIP guidance.

[0089] The model training module 303 is used to train the initial image generation model by alternately using the training guidance method to obtain the image generation model.

[0090] The image generation module 304 is used to obtain the prompt text and input the prompt text into the image generation model to obtain the target image.

[0091] In this embodiment, training samples and their corresponding training prompt texts are obtained to provide a rich semantic context for the model, ensuring the accuracy of image generation; in training, classifier-free guidance and CLIP guidance are used alternately to enhance the self-learning ability of the model, improve the flexibility of the image generation process, and ensure that the generated images achieve higher fidelity in diversity and details; the model training process also combines the advantages of different guidance strategies, making the generated images more consistent and realistic in complex scenes; when applied, high-quality target images can be obtained through simple text prompts, which simplifies the image generation process, expands the application scenarios of the generation model, and improves the accuracy of image generation.

[0092] In some optional implementations of this embodiment, the model training module 203 may also be used to:

[0093] The training image and its corresponding training prompt text are input into the initial image generation model, and the first image is generated according to the training prompt text by the initial image generation model without using an independent classifier.

[0094] Calculate the first loss based on the first image and the training images.

[0095] The training image and its corresponding training prompt text are input into the initial image generation model, and the classifier is guided by CLIP to obtain the second image generated by the initial image generation model.

[0096] A second loss is calculated based on the second image and the training image.

[0097] A joint loss is calculated according to the first loss and the second loss, and the initial image generation model is adjusted according to the joint loss until the joint loss is less than a preset loss threshold, thereby obtaining the image generation model.

[0098] In some optional implementations of this embodiment, the model training module 203 may also be used to:

[0099] The text content of the training prompt text is removed to obtain a first prompt text, wherein the first prompt text includes empty text.

[0100] Without using an independent classifier, a first image is generated according to the first prompt text by using an initial image generation model.

[0101] In some optional implementations of this embodiment, the model training module 203 may also be used to:

[0102] Get the second image generated by the initial image generation model.

[0103] The second image is encoded by the CL IP model to obtain an image encoding, and the training prompt text is encoded to obtain a text encoding.

[0104] Compute the dot product of the image encoding and the text encoding.

[0105] Gradient information is generated according to the dot product, and the initial image generation model is instructed to continue to generate the second image through the gradient information and the training image.

[0106] In some optional implementations of this embodiment, the model training module 203 may also be used to:

[0107] The gradient information is calculated based on the dot product, where the dot product is negatively correlated with the gradient information.

[0108] When denoising is performed based on the training image through the initial image generation model, the pixel values ​​of the generated image are adjusted based on the gradient information to obtain a second image.

[0109] In some optional implementations of this embodiment, the image generation model processing device 300 may further include:

[0110] The editing acquisition module is used to acquire an image editing training set, where the image editing training set includes a second training picture and its corresponding image description text.

[0111] The model fine-tuning module is used to fine-tune the image generation model based on the image editing training set.

[0112] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0113] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0114] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0115] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image generation model processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0116] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the image generation model processing method.

[0117] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0118] The computer device provided in this embodiment can execute the above-mentioned picture generation model processing method. Here, the picture generation model processing method can be the picture generation model processing method of each of the above-mentioned embodiments.

[0119] In this embodiment, training samples and their corresponding training prompt texts are obtained to provide the model with rich semantic context and ensure the accuracy of image generation. During training, classifier-free guidance and CLIP guidance are used alternately to enhance the self-learning ability of the model, improve the flexibility of the image generation process, and ensure that the generated images achieve higher fidelity in diversity and detail. The model training process also combines the advantages of different guidance strategies to make the generated images more consistent and realistic in complex scenes. When applied, high-quality target images can be obtained through simple text prompts, which simplifies the image generation process, expands the application scenarios of the generation model, and improves the accuracy of image generation.

[0120] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image generation model processing method as described above.

[0121] In this embodiment, training samples and their corresponding training prompt texts are obtained to provide the model with rich semantic context and ensure the accuracy of image generation. During training, classifier-free guidance and CLIP guidance are used alternately to enhance the self-learning ability of the model, improve the flexibility of the image generation process, and ensure that the generated images achieve higher fidelity in diversity and detail. The model training process also combines the advantages of different guidance strategies to make the generated images more consistent and realistic in complex scenes. When applied, high-quality target images can be obtained through simple text prompts, which simplifies the image generation process, expands the application scenarios of the generation model, and improves the accuracy of image generation.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0123] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for processing an image generation model, characterized in that: The steps include: Get the training images and their corresponding training prompt texts; Obtaining a preset training guidance mode, wherein the training guidance mode includes a non-classifier guidance and a CLIP guidance; Alternately using the training guidance method to train the initial image generation model to obtain an image generation model; Obtain a prompt text, and input the prompt text into the image generation model to obtain a target image.

2. The image generation model processing method according to claim 1, characterized in that: The step of alternately using the training guidance method to train the initial picture generation model to obtain the picture generation model comprises: Inputting the training picture and its corresponding training prompt text into an initial picture generation model, and generating a first picture according to the training prompt text by the initial picture generation model without using an independent classifier; Calculate a first loss based on the first image and the training image; Inputting the training picture and its corresponding training prompt text into the initial picture generation model, and guiding the classifier through CLIP guidance to obtain a second picture generated by the initial picture generation model; Calculate a second loss based on the second image and the training image; A joint loss is calculated according to the first loss and the second loss, and the initial image generation model is adjusted according to the joint loss until the joint loss is less than a preset loss threshold, thereby obtaining an image generation model.

3. The image generation model processing method according to claim 2, characterized in that: The step of generating a first picture according to the training prompt text by using the initial picture generation model without using an independent classifier comprises: Eliminate text content from the training prompt text to obtain a first prompt text, where the first prompt text includes empty text; Without using an independent classifier, a first picture is generated according to the first prompt text by using the initial picture generation model.

4. The image generation model processing method according to claim 2, characterized in that: The step of obtaining the second image generated by the initial image generation model by performing classifier guidance through CLIP guidance comprises: Obtain a second image generated by the initial image generation model; Encoding the second image through the CLIP model to obtain an image code, and encoding the training prompt text to obtain a text code; Calculating the dot product of the image code and the text code; Gradient information is generated according to the dot product, and the initial image generation model is instructed to continue to generate the second image through the gradient information and the training image.

5. The image generation model processing method according to claim 4, characterized in that: The step of generating gradient information according to the dot product and instructing the initial picture generation model to continue to generate the second picture through the gradient information and the training picture includes: Calculating gradient information according to the dot product, wherein the dot product is negatively correlated with the gradient information; When denoising is performed based on the training image using the initial image generation model, the pixel values ​​of the generated image are adjusted based on the gradient information to obtain a second image.

6. The image generation model processing method according to claim 1, characterized in that: After the step of alternately using the training guidance method to train the initial picture generation model to obtain the picture generation model, the method further includes: Acquire an image editing training set, wherein the image editing training set includes a second training picture and its corresponding image description text; The picture generation model is fine-tuned according to the image editing training set.

7. A picture generation model processing device, characterized in that: include: A training acquisition module is used to acquire training images and their corresponding training prompt texts; A mode acquisition module, used to acquire a preset training guidance mode, wherein the training guidance mode includes a classifier-free guidance and a CLIP guidance; A model training module, used to alternately use the training guidance method to train the initial image generation model to obtain an image generation model; The image generation module is used to obtain the prompt text and input the prompt text into the image generation model to obtain the target image.

8. The image generation model processing device according to claim 7, characterized in that: The model training module includes: A first training submodule, configured to input the training picture and its corresponding training prompt text into an initial picture generation model, and generate a first picture according to the training prompt text through the initial picture generation model without using an independent classifier; A first calculation submodule, used to calculate a first loss according to the first image and the training image; A second training submodule is used to input the training picture and its corresponding training prompt text into the initial picture generation model, and perform classifier guidance through CLIP guidance to obtain a second picture generated by the initial picture generation model; A second calculation submodule, used to calculate a second loss according to the second image and the training image; A model adjustment submodule is used to calculate a joint loss according to the first loss and the second loss, and adjust the initial image generation model according to the joint loss until the joint loss is less than a preset loss threshold, so as to obtain an image generation model.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the image generation model processing method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the image generation model processing method according to any one of claims 1 to 6 are implemented.

Images