Method, device, medium and equipment for training image synthesis model of subject image

Abstract

The application provides a training method, system, medium and equipment of an image synthesis model of a theme image, the method comprising: obtaining a theme image input by a user, adding different degrees of noise to the theme image to obtain a low-resolution image and a high-resolution image, the noise of the low-resolution image being more than that of the high-resolution image; taking the low-resolution image and a text prompt word as input of a low-resolution text-to-image model to generate an intermediate image, taking the intermediate image and the high-resolution image as input of a super-resolution diffusion model, fine-tuning the super-resolution diffusion model according to a comparison result, and obtaining the image synthesis model of the theme image. The method is used to train a model based on a small amount of training data, simplifies the difficulty of model training, and can ensure the high quality and diversity of the synthesized image.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, medium, and device for an image synthesis model of a subject image. Background Technology

[0002] The mainstream text-to-image generation technology currently uses the diffusion model, an image generation model based on Markov chains that can gradually recover a clear image from Gaussian noise. The text-to-image diffusion model typically involves two steps: first, generating a low-resolution image from the text, and then using a super-resolution diffusion model to improve image quality, resulting in high-quality and diverse images. Currently, the diffusion model's primary task is generating high-quality and diverse images; however, there is significant market demand for customized images, such as cloning one's own image onto a new background. Technically, this customized image creation is called image compositing. The advantages of image compositing include the ability to create new images, enhance their visual effects, and improve their quality and performance.

[0003] Understandably, image synthesis techniques aim to clone a given subject into a new background, seamlessly integrating the subject into the background. However, current image synthesis techniques rely on a large number of high-quality images as training samples, thus consuming significant computational resources. Furthermore, the realism and naturalness of the images are affected by the quality of the samples. Therefore, there is an urgent need to provide a new image synthesis technique to address these issues. Summary of the Invention

[0004] The purpose of this invention is to provide a training method, system, medium, and device for an image synthesis model of a subject image, which can be used to train the model based on a small amount of training data, simplify the model training difficulty, and ensure the high quality and diversity of the synthesized images.

[0005] In a first aspect, embodiments of the present invention provide a training method for an image synthesis model of a subject image. The method includes: acquiring a subject image input by a user; adding noise of different degrees to the subject image to obtain a low-resolution image and a high-resolution image, wherein the noise in the low-resolution image is greater than the noise in the high-resolution image; using the low-resolution image and a text prompt as input to a low-resolution text-to-image model to generate an intermediate image, wherein the text-to-image model is encoded with a unique identifier referencing the subject image; using the intermediate image and the high-resolution image as input to a super-resolution diffusion model, and fine-tuning the super-resolution diffusion model based on the comparison results to obtain an image synthesis model of the subject image.

[0006] The beneficial effects of the training method for the image synthesis model of the subject image provided in this embodiment of the invention are as follows: the training process of the model requires a small number of subject images to fine-tune the text-to-image model and the super-resolution diffusion model. Usually, three to five subject images are sufficient. Moreover, the training process of the image synthesis model can be compatible with processing multiple types of images, such as cars, animals, food, etc., thus simplifying the training difficulty of the model and ensuring the high quality and diversity of the synthesized images.

[0007] In one possible implementation, the low-resolution image and text prompts are used as input to a low-resolution text-to-image model to generate intermediate images. This includes: using the low-resolution image and text prompts as input to the low-resolution text-to-image model, and using N frames of images of the same category as the subject image as input to generate intermediate images. The advantages of this embodiment are that using N frames of images of the same category as the subject image as input is equivalent to using a class-specific prior preservation loss to prevent overfitting or language drift in the text-to-image model, preserving the model's semantic knowledge of the subject category. Furthermore, the model trained in this way can generate more diverse and stylized images during the customization process.

[0008] In other possible implementations, using the low-resolution image and text prompts as input to a low-resolution text-to-image model to generate an intermediate image includes: using the low-resolution image as a base image, performing a restoration operation on the low-resolution image using the text prompts to generate the intermediate image.

[0009] Using the intermediate image and the high-resolution image as input to the super-resolution diffusion model, and fine-tuning the super-resolution diffusion model based on the comparison results to obtain an image synthesis model for the subject image, the process includes: comparing the intermediate image with the high-resolution image to obtain a loss function, inputting the loss function into the super-resolution diffusion model, iteratively reducing the loss value to obtain an image synthesis model for the subject image.

[0010] In another possible implementation, the method further includes: inputting a source image into the image synthesis model; obtaining the output result of the image synthesis model, wherein the output result is a synthesized image obtained by image fusion of the source image and the sample background image. This method can utilize artificial intelligence technology for image synthesis, providing the ability to synthesize user-defined images into multiple scenes. Based on user needs, it selects the most matching image elements from a vast amount of image information sources to synthesize a result image that meets user expectations. It can also perform conversion and fusion between different image domains, realizing image style transfer, enhancement, and repair functions. It can be widely used in computer vision, pattern recognition, and other fields to solve practical problems.

[0011] In other possible implementations, the super-resolution diffusion model is used to calculate a denoising distribution based on the current noise level and category, and to sample a new image from it, obtaining an image that satisfies the training data distribution after multiple iterations.

[0012] Secondly, embodiments of the present invention also provide a training apparatus for an image synthesis model of a subject image. This training apparatus includes modules / units that execute the methods of any possible implementation of the first aspect described above. These modules / units can be implemented in hardware or by hardware executing corresponding software.

[0013] Thirdly, the present invention also provides a computer-readable storage medium comprising a program that, when run on a terminal device, causes the terminal device to perform any of the possible implementations of the first aspect described above.

[0014] Fourthly, embodiments of the present invention also provide a computer program product that, when the program product is run on a terminal device, causes the terminal device to execute any of the possible implementation methods of the first aspect described above.

[0015] For the beneficial effects of the second to fourth aspects mentioned above, please refer to the description in the first aspect mentioned above. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of an image synthesis model structure provided in an embodiment of the present invention;

[0018] Figure 2 A schematic diagram illustrating the training method of an image synthesis model for a subject image provided in an embodiment of the present invention;

[0019] Figure 3 A schematic diagram illustrating a training example architecture for an image synthesis model of a subject image provided in an embodiment of the present invention;

[0020] Figure 4 A schematic diagram illustrating the forward inference process of the trained super-resolution diffusion model provided in this embodiment of the invention.

[0021] Figure 5 This is a schematic diagram of the image synthesis effect provided in an embodiment of the present invention;

[0022] Figure 6 A schematic diagram of a training device for an image synthesis model of a subject image provided in an embodiment of the present invention;

[0023] Figure 7 This is a schematic diagram of a terminal device structure provided in an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention relate to artificial intelligence (AI) and machine learning (ML) technologies, and are implemented based on deep learning networks and machine learning in artificial intelligence.

[0025] With the research and advancement of artificial intelligence technology, AI is being studied and applied in various fields, such as smart homes, intelligent customer service, virtual assistants, smart speakers, intelligent marketing, autonomous driving, robotics, and smart healthcare. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0026] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0027] In the description of embodiments of the present invention, the terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be a limitation of the invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of the present invention, “at least one” and “one or more” refer to one or more (including two). The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0028] References to "one embodiment" or "some embodiments" as used in this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the invention. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized. The term "connection" includes both direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0029] In this embodiment of the invention, the words "exemplarily" or "for example" are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or implementation described as "exemplarily" or "for example" in this embodiment of the invention should not be construed as being more preferred or advantageous than other embodiments or implementations. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0030] The image synthesis technology provided by this invention can achieve excellent results with a small amount of training data, generating images of specific themes based on text prompts. The training technique for the proposed image synthesis model utilizes natural language processing, deep learning, image recognition, and multimodal technologies. This technique leverages a pre-trained text-to-image conversion model and a category-specific prior preservation loss to improve image quality and diversity. The key features of this invention are that it requires only a small number of theme images to fine-tune the text-to-image model and the super-resolution diffusion model; typically three to five theme images are sufficient. Furthermore, the training process of this image synthesis model is compatible with processing multiple image categories, such as cars, animals, and food, thus simplifying model training while ensuring high quality and diversity of synthesized images.

[0031] The main structure of the training method for the image synthesis model of the subject image provided in this embodiment of the invention is as follows: Figure 1 As shown, the model mainly consists of two parts: a low-resolution text-to-image model and a super-resolution diffusion model. The low-resolution text-to-image model generates a low-resolution image of the subject based on text prompts and a unique identifier. The super-resolution diffusion model generates a high-resolution image based on the low-resolution image and a high-resolution subject image, thereby improving the fidelity of details.

[0032] This image synthesis model can be integrated into terminal devices such as smartphones, tablets, or portable personal computers. It utilizes artificial intelligence technology for image synthesis, enabling the synthesis of user-defined images into multiple scenes. It solves the problem of selecting the most matching image elements from a vast amount of image information sources according to user needs, synthesizing the result image that meets the user's expectations, and performing conversion and fusion between different image domains. It also realizes functions such as image style transfer, enhancement, and repair, and can be widely used in computer vision, pattern recognition, and other fields to solve practical problems.

[0033] This invention provides a method for training an image synthesis model for a subject image, such as... Figure 2 As shown, the process of this method can be executed by a terminal device, and the method includes the following steps:

[0034] S201, acquire the subject image input by the user, add different levels of noise to the subject image to obtain a low-resolution image and a high-resolution image, wherein the noise in the low-resolution image is greater than the noise in the high-resolution image.

[0035] S202, the low-resolution image and text prompt are used as input to a low-resolution text-to-image model to generate an intermediate image, wherein the text-to-image model is encoded with a unique identifier that references the topic image.

[0036] In one possible implementation, the low-resolution image and text prompts are used as input to a low-resolution text-to-image model, and N frames of images belonging to the same category as the topic image are used as input to generate intermediate images. This approach, using N frames of images belonging to the same category as the topic image as input, is equivalent to using a class-specific prior preservation loss to prevent overfitting or language drift in the text-to-image model, preserving the model's semantic knowledge of the topic category. Furthermore, the model trained in this way can generate more diverse and stylized images during the customization process.

[0037] S203, the intermediate image and the high-resolution image are used as inputs to the super-resolution diffusion model, and the super-resolution diffusion model is fine-tuned according to the comparison results to obtain the image synthesis model of the subject image.

[0038] Combination Figure 3For example, a user uploads several photos of face cream. Noise is added to these photos, and adjacent noisy images are selected. The low-resolution images are used as the base images, and the text prompts are used to restore the low-resolution images, generating intermediate images. The text prompts, for example, are the words "sks" from "A photo of sks," where "sks" is a meaningless English word. Using these prompts for image generation, the high-resolution and low-resolution images are compared to obtain a loss function. This loss function is fed into a diffusion model, and the loss value is iteratively reduced, allowing the model to learn well from the user-provided photos of face cream. Furthermore, to help the model recognize face cream, a number of other face cream photos can be fed into the model for training. After training, the model can generate more diverse and stylized face cream images during image customization, such as images of face cream from different angles and directions.

[0039] For example, a photo of a face cream. Figure 4 As shown, the forward inference process using the trained super-resolution diffusion model includes: First, starting with an image composed entirely of Gaussian noise, which can be obtained through random sampling; then, iteratively denoising to obtain a high-quality sample by predicting the noise filtered out in each step. In each step, the diffusion model calculates the denoising distribution based on the current noise level and category conditions, and samples a new image from it. Finally, after multiple iterations, an image satisfying the training data distribution is obtained. It can be seen that the training process of the above model consists of two steps: The first step is to fine-tune the low-resolution text-to-image model, using topic images and text prompts as input, while employing a class-specific prior preservation loss to prevent overfitting or language drift, preserving the model's semantic knowledge of the topic category; the second step is to fine-tune the super-resolution diffusion model, using low-resolution and high-resolution topic image pairs as input, enabling the model to retain important details of the topic.

[0040] Furthermore, after a user inputs a source image into the image synthesis model, the user can obtain the output result of the image synthesis model. The output result is a synthesized image obtained by fusing the source image with a sample background image. The output synthesized image can modify the original image, such as changing the position of the subject, changing the subject's attributes such as color and shape, and modifying the subject's pose, expression, material, and other semantic modifications. For example, a user uploads a subject image to the workbench. Figure 5 (a) in the image is a photo input by the user into the image synthesis model. After using this subject image to train the model, the training result is as follows: Figure 5As shown in (b) above. In summary, this invention utilizes artificial intelligence technology for image synthesis, providing the ability to synthesize user-defined images into multiple scenes. It solves the problem of selecting the most matching image elements from a vast amount of image information sources according to user needs, synthesizing the result image that meets user expectations, and performing conversion and fusion between different image domains to achieve image style transfer, enhancement, and repair functions. It can be widely used in computer vision, pattern recognition, and other fields to solve practical problems.

[0041] In some embodiments of the present invention, a training apparatus for an image synthesis model of a subject image is disclosed, such as... Figure 6 As shown, the device is used to implement the methods described in the above training method embodiments, and includes: an acquisition unit 601, a first-level training unit 602, and a second-level training unit 603. The acquisition unit 601 is used to acquire a user-inputted subject image, add different levels of noise to the subject image to obtain a low-resolution image and a high-resolution image, wherein the low-resolution image has more noise than the high-resolution image. The first-level training unit 602 is used to take the low-resolution image and text prompts as input to a low-resolution text-to-image model to generate an intermediate image, wherein the text-to-image model is encoded with a unique identifier referencing the subject image. The second-level training unit 603 is used to take the intermediate image and the high-resolution image as input to a super-resolution diffusion model, fine-tune the super-resolution diffusion model based on the comparison results, and obtain an image synthesis model for the subject image.

[0042] The first-level training unit 602 uses the low-resolution image and text prompts as input to the low-resolution text-to-image model to generate an intermediate image. Specifically, it uses the low-resolution image as the base image and the text prompts to perform a restoration operation on the low-resolution image to generate an intermediate image.

[0043] The secondary training unit 603 takes the intermediate image and the high-resolution image as input to the super-resolution diffusion model, and fine-tunes the super-resolution diffusion model according to the comparison results to obtain an image synthesis model for the subject image. Specifically, it compares the intermediate image with the high-resolution image to obtain a loss function, inputs the loss function into the super-resolution diffusion model, and iteratively reduces the loss value to obtain an image synthesis model for the subject image.

[0044] All relevant content of each step involved in the above training method embodiment can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0045] The training device for the image synthesis model of the aforementioned subject image can be executed by a chip or chip module. Regarding the various devices and products described in the above embodiments, each module / unit can be a software module / unit, a hardware module / unit, or a combination of both.

[0046] In other embodiments of the present invention, a terminal device is disclosed, such as... Figure 7 As shown, the terminal device can integrate the aforementioned image synthesis model and its hardware components include: one or more processors 701; a memory 702; a display 703; one or more application programs (not shown); and one or more computer programs 704. These components can be connected via one or more communication buses 705. The one or more computer programs 704 are stored in the memory 702 and configured to be executed by the one or more processors 701. The one or more computer programs 704 include instructions.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in the above-described method embodiments. Specific beneficial effects can be found in the above-described method embodiments.

[0048] The present invention also provides a computer program product that, when executed by a computer, implements the method described in the above-described method embodiments. Specific beneficial effects can be found in the above-described method embodiments.

[0049] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0050] In the various embodiments of this invention, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0051] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0052] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of the present invention should be covered within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A training method for an image synthesis model of a subject image, characterized in that, include: A user-input theme image is obtained, and different levels of noise are added to the theme image to obtain a low-resolution image and a high-resolution image, wherein the low-resolution image has more noise than the high-resolution image. The low-resolution image and text prompts are used as input to a low-resolution text-to-image model to generate an intermediate image. The text-to-image model is encoded with a unique identifier that references the subject image. The intermediate image and the high-resolution image are used as inputs to the super-resolution diffusion model. The super-resolution diffusion model is then fine-tuned based on the comparison results to obtain an image synthesis model for the subject image. Using the intermediate image and the high-resolution image as input to the super-resolution diffusion model, and fine-tuning the super-resolution diffusion model based on the comparison results, an image synthesis model for the subject image is obtained, including: The intermediate image is compared with the high-resolution image to obtain a loss function. The loss function is then fed into the super-resolution diffusion model, and the loss value is iteratively reduced to obtain an image synthesis model for the subject image. Input a source image into the image synthesis model; obtain the output result of the image synthesis model, wherein the output result is a synthesized image obtained by fusing the source image with the sample background image.

2. The method according to claim 1, characterized in that, The low-resolution image and text prompts are used as input to a low-resolution text-to-image model to generate an intermediate image, including: The low-resolution image and text prompt are used as input to a low-resolution text-to-image model, and N frames of images of the same category as the subject image are used as input to generate an intermediate image.

3. The method according to claim 1, characterized in that, The low-resolution image and text prompts are used as input to a low-resolution text-to-image model to generate an intermediate image, including: Using the low-resolution image as the base image, the text prompts are used to restore the low-resolution image and generate an intermediate image.

4. The method according to any one of claims 1 to 3, characterized in that, The super-resolution diffusion model is used to calculate the denoising distribution based on the current noise level and category, and to sample a new image from it. After multiple iterations, an image that satisfies the training data distribution is obtained.

5. A training device for an image synthesis model of a subject image, characterized in that, include: The acquisition unit is used to acquire a subject image input by the user, add different levels of noise to the subject image to obtain a low-resolution image and a high-resolution image, wherein the noise in the low-resolution image is greater than the noise in the high-resolution image; A first-level training unit is used to take the low-resolution image and text prompts as input to a low-resolution text-to-image model to generate an intermediate image. The text-to-image model is encoded with a unique identifier that references the topic image. The secondary training unit is used to take the intermediate image and the high-resolution image as input to the super-resolution diffusion model, and fine-tune the super-resolution diffusion model according to the comparison results to obtain the image synthesis model of the subject image; The secondary training unit takes the intermediate image and the high-resolution image as input to the super-resolution diffusion model, and fine-tunes the super-resolution diffusion model based on the comparison results to obtain an image synthesis model for the subject image, specifically used for: The intermediate image is compared with the high-resolution image to obtain a loss function. The loss function is then fed into the super-resolution diffusion model, and the loss value is iteratively reduced to obtain an image synthesis model for the subject image. Input a source image into the image synthesis model; obtain the output result of the image synthesis model, wherein the output result is a synthesized image obtained by fusing the source image with the sample background image.

6. The apparatus according to claim 5, characterized in that, The first-level training unit takes the low-resolution image and text prompts as input to the low-resolution text-to-image model to generate intermediate images, specifically for: The low-resolution image and text prompt are used as input to a low-resolution text-to-image model, and N frames of images of the same category as the subject image are used as input to generate an intermediate image.

7. The apparatus according to claim 5, characterized in that, The first-level training unit takes the low-resolution image and text prompts as input to the low-resolution text-to-image model to generate intermediate images, specifically for: Using the low-resolution image as the base image, the text prompts are used to restore the low-resolution image and generate an intermediate image.

8. A computer-readable storage medium storing a program therein, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

9. A terminal device, characterized in that, The device includes a memory and a processor, wherein the memory stores a program that can run on the processor, and when the program is executed by the processor, causes the terminal device to implement the method as described in any one of claims 1 to 4.

Images