Zero-sample custom image generation method, control device, storage medium and system

Abstract

The invention discloses a zero-sample custom image generation method, a control device, a storage medium and a system, and belongs to the technical field of image generation. The method comprises the steps of obtaining an input text sample set used for customizing images, generating at least one image consistent with an identity described by the input text sample for each input text sample in the input text sample set, constructing a training data set based on the generated images, and adopting a parallel processing architecture to process the input text sample set and the training data set. And training a preset text-to-image diffusion model, and deploying the trained text-to-image diffusion model so as to obtain a corresponding self-defined image according to the text input by the user. Through the parallel processing architecture, the preset text-to-image diffusion model is trained, so that the trained model can accurately output the image consistent with the input text information.

Description

Technical Field

[0001] The present invention relates to the field of image generation technology, and in particular to a zero-sample custom image generation method, a control device, a storage medium and a system. Background Art

[0002] Existing image generation models can generate high-quality images based on the text prompts people input. However, the defect of this type of model is that it is impossible to control the various details of image generation in detail, that is, the generated images cannot guarantee the consistency of the generated images. In order to overcome the problems of the above models, the industry has taken some measures to try to enhance the model's control over image generation. However, this method needs to be adapted to specific fields and requires a large amount of training data to train the model, which results in a waste of resources.

[0003] Currently, there is a lack of an image generation method that can both enhance the model's consistent control over image generation and avoid wasting resources by using a large amount of training data. Summary of the invention

[0004] The present invention provides a zero-sample custom image generation method, a control device, a storage medium and a system, which are used to solve the defect in the prior art that there is a lack of an image generation method that can both enhance the consistency control of the model on image generation and avoid the waste of resources caused by using a large amount of training data.

[0005] On the one hand, the present invention provides a zero-sample custom image generation method based on a diffusion model, comprising: obtaining an input text sample set for a custom image; for each input text sample in the input text sample set, generating at least one image consistent with the identity described by the input text sample; based on the generated images, constructing a training data set, and adopting a parallel processing architecture to train a preset text-to-image diffusion model; and deploying the trained text-to-image diffusion model to obtain a corresponding custom image according to the user input text.

[0006] Optionally, for each input text sample in the input text sample set, generating at least one image consistent with the identity described by the input text sample includes: generating a text prompt for describing an image grid according to each input text sample; generating a corresponding image grid according to the generated text prompt, wherein the image grid is a rectangular network including a plurality of panels, each panel corresponding to an image with the same identity; and screening out, in the generated image grid, the at least one image consistent with the identity described by the input text sample.

[0007] Optionally, the image corresponding to each panel further includes different feature information, and the feature information includes character feature information, environment feature information and style feature information.

[0008] Optionally, screening out, in the generated image grid, at least one image that is consistent with the identity described by the input text sample, comprises: using an image encoder to convert an image corresponding to each panel in the generated image grid into an image embedding vector; using a text encoder to convert the text prompt into a text embedding vector; and screening out the at least one image that is consistent with the identity described by the input text sample by determining whether the image embedding vector is consistent with information indicating the identity in the text embedding vector.

[0009] Optionally, the parallel processing architecture is used to train the preset text-to-image diffusion model, including: inputting the training data set into the preset image-to-image model, and the output reference image is used as the first frame of video image; at the same time, inputting the training data set into the preset text-to-image diffusion model, and the output image is used as the second frame of video image; exchanging information between the first frame of video image and the second frame of video image; calculating the loss between the first frame of video image and the second frame of video image to establish an identity mapping between the first frame of video image and the second frame of video image; and repeatedly training the preset text-to-image diffusion model, and when the loss between the output second frame of video image and the first frame of video image is less than a preset threshold, the trained text-to-image diffusion model is obtained.

[0010] Optionally, the calculating the loss between the first frame of video image and the second frame of video image includes: calculating identity loss using a feature extractor and cosine similarity; calculating style loss using a style transfer model; calculating content loss using a content feature extractor; and calculating a weighted sum of the identity loss, the style loss and the content loss to obtain the loss between the first frame of video image and the second frame of video image.

[0011] Optionally, the deployment of a trained text-to-image diffusion model for obtaining a corresponding customized image based on user input text includes: obtaining user input text, and generating at least one image consistent with the identity of the user input text based on the user input text; inputting the generated at least one image consistent with the identity of the user input text prompt into a preset image-to-image model, and outputting a reference image as a first frame of video image; and at the same time, inputting the user input text and the reference image into the trained text-to-image diffusion model to obtain a corresponding customized image.

[0012] On the other hand, the present invention also provides a control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the zero-sample custom image generation method.

[0013] On the other hand, the present invention further provides a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute the zero-sample custom image generation method.

[0014] On the other hand, the present invention further provides a zero-sample custom image generation system based on a diffusion model, wherein the zero-sample custom image generation system comprises a user end and a cloud server end, and the cloud server end comprises the above-mentioned control device.

[0015] The zero-sample custom image generation method, control device, storage medium and system provided by the present invention obtain an input text sample set for custom images; for each input text sample in the input text sample set, generate at least one image consistent with the identity described by the input text sample; based on the generated images, construct a training data set, and adopt a parallel processing architecture to train a preset text-to-image diffusion model; and deploy the trained text-to-image diffusion model to obtain a corresponding custom image according to the user input text, thereby solving the problem that the prior art is difficult to control the consistency of image generation or wastes a large amount of training data resources, and achieving the beneficial effect of generating multiple images with consistent identities through zero-sample learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the flow charts of the method for generating a zero-sample custom image based on a diffusion model provided by an embodiment of the present invention; Figure 2 This is the second flow chart of the method for generating a zero-sample custom image based on a diffusion model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] As mentioned earlier, when artists and designers need to create images in different scenes based on specific concepts, such as "identity-preserving generation", it is often difficult to achieve. To solve this problem, existing technologies try to enhance the control ability of the model by adding additional conditional mechanisms or fine-tuning. For example, some models achieve fine-grained control over image structure by introducing spatial conditional inputs such as sketches, depth maps, or segmentation masks. Others fine-tune the model to enable it to generate consistent results based on a small number of reference images. However, these methods are either limited to specific fields or require a large amount of training data or computing resources.

[0020] To solve this problem, the present invention proposes a zero-sample custom image generation method based on a diffusion model. Figure 1-Figure 2 The present invention is described in detail.

[0021] Figure 1 This is one of the flow charts of the zero-sample custom image generation method based on the diffusion model provided in an embodiment of the present invention.

[0022] like Figure 1 As shown, the zero-sample custom image generation method based on the diffusion model provided by the embodiment of the present invention can be executed by a controller, and the method can include the following steps: Step S100: Obtain an input text sample set for customizing an image.

[0023] To illustrate with an example, for each input text sample in the input text sample set, such as "a little boy posing in a victory pose, generate two pictures with different styles that are consistent with the identity of the example picture (e.g., the sample picture)".

[0024] Step S200: for each input text sample in the input text sample set, generate at least one image consistent with the identity described by the input text sample.

[0025] Preferably, the step S200 may include: S210, generating a text prompt for describing an image grid according to each input text sample. S220, generating a corresponding image grid according to the generated text prompt, wherein the image grid is a rectangular network including a plurality of panels, each panel corresponding to an image with the same identity. S230, screening out at least one image having the same identity as described by the input text sample in the generated image grid.

[0026] In the embodiment of the present invention, the image corresponding to each panel may also include different feature information, and the feature information may include character feature information, environment feature information, style feature information, and the like.

[0027] For example, the feature information may include character, expression, posture, lighting conditions, etc. For example, the image corresponding to each panel may be the same person, but in different environments or showing different styles.

[0028] Following the above example, an existing tool for generating text prompts describing an image grid (e.g., a GPT-4o model) can be used to generate text prompts for describing an image grid. For example: an image containing 4 panels, each panel shows the same little boy, but with different expressions and postures (e.g., victory posture, frustrated posture, smiling expression, angry expression, etc.). Configure and use an existing tool for generating an image grid based on text prompts (e.g., a FLUX1.0DEV model, which includes multiple Transformer layers for learning the relationship between text and images). Using this tool, a corresponding image grid can be generated, for example: the same subject (e.g., a little boy) is displayed on each panel, but with different expressions, postures, lighting conditions, etc. (e.g., victory posture, frustrated posture, smiling expression, angry expression, etc.). Configure and use an existing tool for filtering the generated image grid (e.g., a Gemini-1.5 model), and filter out at least one image in the generated image grid that is consistent with the identity described by the input text sample (e.g., the same little boy).

[0029] Further preferably, the S230 may also include: S231, using an image encoder to convert the image corresponding to each panel in the generated image grid into an image embedding vector. S232, using a text encoder to convert the text prompt into a text embedding vector. S233, by determining whether the image embedding vector is consistent with the information indicating the identity in the text embedding vector, to filter out the at least one image that is consistent with the identity described by the input text sample.

[0030] Continuing with the above example, the Gemini-1.5 model, which includes an image encoder and a text encoder, can be used to convert images and texts into corresponding embedding vectors. Based on the image embedding vector and the text embedding vector, determine whether the image is consistent with the information in the text prompt, retain images with consistent identities, and discard images that do not meet the requirements. Using this tool, determine whether the images in the generated image grid meet the requirements of identity consistency (for example, the same little boy), and filter out images with consistent identities.

[0031] Step S300: construct a training data set based on the generated images, and use a parallel processing architecture to train a preset text-to-image diffusion model.

[0032] Preferably, the S300 may also include: S310, inputting the training data set into the preset image-to-image model, and the output reference image is used as the first frame of video image. S320, inputting the training data set into the preset text-to-image diffusion model, and the output image is used as the second frame of video image. S330, exchanging information between the first frame of video image and the second frame of video image. S340, calculating the loss between the first frame of video image and the second frame of video image to establish an identity mapping between the first frame of video image and the second frame of video image. And S350, repeatedly training the preset text-to-image diffusion model, and when the loss between the output second frame of video image and the first frame of video image is less than a preset threshold, the trained text-to-image diffusion model is obtained.

[0033] For example, the training data set is input into a preset image-to-image model, and the output reference image is used as the first frame of video image, which can be consistent with the preset image information. Then the training data set is input into the preset FLUX1.0DEV model, and information is exchanged between the first frame of video image and the second frame of video image to calculate the loss between the first frame of video image and the second frame of video image, so as to establish an identity mapping between the first frame of video image and the second frame of video image. The preset text-to-image diffusion model is repeatedly trained, and when the loss between the output second frame of video image and the first frame of video image is less than the preset threshold, the trained text-to-image diffusion model is obtained.

[0034] Further preferably, in step S340, the image generation method may further include: S341, calculating identity loss using a feature extractor and cosine similarity.

[0035] To illustrate, for example, the calculated identity loss can be expressed by the following formula: L_{identity}=1-cos(φ(x_1),φ(x_2))(1) Among them, φ is the feature extractor, x_1 and x_2 are two frames of images respectively.

[0036] S342. Calculate style loss using a style transfer model (e.g., a convolutional neural network (CNN) model: VGG16).

[0037] Take an example to illustrate that when using it, remove the fully connected layer of the neural network (CNN) model and only keep the convolution layer and pooling layer to extract the feature map. Use the style feature extractor φ_style to extract the feature maps of two frames of images x_1 and x_2, respectively, denoted as φ_style(x_1) and φ_style(x_2).

[0038] The feature maps of x_1 and x_2 may be converted to the same size and shape using at least upsampling, cropping, and scaling.

[0039] Continuing with the above example, the calculated style loss can be expressed as follows: L_{style}=||φ_style(x_1)-φ_style(x_2)||^2(2) Among them, φ_style is the style feature extractor.

[0040] S343. Calculate content loss using a content feature extractor.

[0041] To illustrate, for example, the calculated content loss can be expressed by the following formula: L_{content}=||φ_content(x_1)-φ_content(x_2)||^2 (3) Among them, φ_content is the content feature extractor.

[0042] S344. Calculate a weighted sum of the identity loss, the style loss, and the content loss to obtain a loss between the first frame of video image and the second frame of video image.

[0043] To illustrate with an example, for example, the weighted sum of the identity loss, the style loss, and the content loss can be expressed by the following formula: L=L_{identity}+L_{style}+L_{content} (4) Where: L is the total loss, L_{identity} is the identity loss, L_{style} is the style loss, and L_{content} is the content loss.

[0044] Step S400: deploy the trained text-to-image diffusion model to obtain a corresponding customized image according to the user input text.

[0045] Preferably, step S400 may also include: S410, obtaining user input text, and generating at least one image consistent with the identity of the user input text according to the user input text. S420, inputting the generated at least one image consistent with the identity of the user input text prompt into a preset image model, and outputting a reference image as the first frame of video image; at the same time. S430, inputting the user input text and the reference image into the trained text-to-image diffusion model to obtain a corresponding customized image.

[0046] For example, the text input by the user can be "a little boy posing in a victory pose", and based on the text, 4 images with the same identity as the input text are generated. The image and text are input into the preset image-to-image model, and the output reference image is used as the first frame image. At the same time, the text input by the user and the reference image are input into the trained text-to-image diffusion model to obtain the corresponding "a little boy posing in a victory pose" image.

[0047] Accordingly, when image generation is required, the embodiment of the present invention obtains an input text sample set for customizing an image; for each input text sample in the input text sample set, generates at least one image consistent with the identity described by the input text sample; based on the generated images, constructs a training data set, and adopts a parallel processing architecture to train a preset text-to-image diffusion model; and deploys the trained text-to-image diffusion model to obtain a corresponding customized image based on the user input text, thereby solving the problem that the prior art is difficult to control the consistency of image generation or wastes a large amount of training data resources, so that the present invention can generate a variety of images with consistent identities and varied styles through zero-sample learning, thereby being applied to various identity-preserving generation tasks.

[0048] Furthermore, the present invention can generate images with consistent identities based on reference images without any additional training data or fine-tuning. It can perform various edits and transformations on images, such as style changes, lighting adjustments, depth control, etc., based on the user's text prompts and reference images. It can also automatically generate training data without manually collecting data by leveraging existing pre-trained text-to-image diffusion models, large language models, and visual language models. It can also be applied to various identity-preserving generation tasks, such as character generation, camera control, lighting adjustment, and asset customization.

[0049] Based on the same general inventive concept, the present invention also protects a control device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the zero-sample custom image generation method.

[0050] Based on the same general inventive concept, the present invention also protects a machine-readable storage medium, on which instructions are stored, and the instructions enable a machine to execute the above-mentioned zero-sample custom image generation method.

[0051] Based on the same general inventive concept, the present invention also protects a zero-sample custom image generation system based on a diffusion model, wherein the zero-sample custom image generation system includes a user end and a cloud server end, and the cloud server end includes the above-mentioned control device.

[0052] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0056] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0057] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0058] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0059] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0060] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A zero-shot custom image generation method based on a diffusion model, characterized in that: The zero-sample custom image generation method comprises: Get a sample set of input text for custom images; For each input text sample in the input text sample set, generating at least one image consistent with the identity described by the input text sample; Based on the generated images, a training dataset is constructed and a parallel processing architecture is used to train the preset text-to-image diffusion model. Deploy the trained text-to-image diffusion model to obtain the corresponding customized image based on the user input text.

2. The zero-sample custom image generation method according to claim 1, characterized in that: The step of generating, for each input text sample in the input text sample set, at least one image consistent with the identity described by the input text sample comprises: Based on each input text sample, generating a text prompt for describing the image grid; Generating a corresponding image grid according to the generated text prompt, wherein the image grid is a rectangular network including a plurality of panels, each panel corresponding to an image with the same identity; and In the generated image grid, at least one image that is consistent with the identity described by the input text sample is screened out.

3. The zero-sample custom image generation method according to claim 2, characterized in that: The image corresponding to each panel also includes different feature information. The feature information includes character feature information, environment feature information and style feature information.

4. The zero-sample custom image generation method according to claim 2, characterized in that: The step of selecting at least one image consistent with the identity described by the input text sample from the generated image grid comprises: Using an image encoder, convert the image corresponding to each panel in the generated image grid into an image embedding vector; Converting the text prompt into a text embedding vector using a text encoder; and By determining whether the information indicating the identity in the image embedding vector is consistent with that in the text embedding vector, the at least one image consistent with the identity described by the input text sample is screened out.

5. The zero-sample custom image generation method according to claim 1, characterized in that: The parallel processing architecture is used to train the preset text-to-image diffusion model, including: The training data set is input into a preset image into the image model, and the output reference image is used as the first frame of the video image; at the same time, Inputting the training data set into the preset text-to-image diffusion model, and using the output image as the second frame of video image; Exchanging information between the first frame of video image and the second frame of video image; Calculating a loss between the first frame of video image and the second frame of video image to establish an identity mapping between the first frame of video image and the second frame of video image; The preset text-to-image diffusion model is repeatedly trained, and when the loss between the output second frame of video image and the first frame of video image is less than a preset threshold, the trained text-to-image diffusion model is obtained.

6. The zero-sample custom image generation method according to claim 5, characterized in that: The calculating the loss between the first frame of video image and the second frame of video image comprises: Using feature extractor and cosine similarity, calculate identity loss; Use the style transfer model to calculate the style loss; Using the content feature extractor, calculate the content loss; A weighted sum of the identity loss, the style loss, and the content loss is calculated to obtain a loss between the first frame of video image and the second frame of video image.

7. The zero-sample custom image generation method according to claim 5, characterized in that: The deployed trained text-to-image diffusion model is used to obtain a corresponding customized image according to the user input text, including: Acquire user input text, and generate at least one image consistent with the identity of the user input text according to the user input text; Input the generated at least one image that is consistent with the identity of the user input text prompt into the preset image model, and output the reference image as the first frame of video image; at the same time, The user input text and the reference image are input into the trained text into the image diffusion model to obtain a corresponding customized image.

8. A control device, characterized in that: The control device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the zero-sample custom image generation method according to any one of claims 1 to 7.

9. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which enable the machine to execute the zero-sample custom image generation method according to any one of claims 1-7.

10. A zero-shot custom image generation system based on a diffusion model, characterized in that: The zero-sample custom image generation system includes a user end and a cloud server end. The cloud server side includes the control device described in claim 8.

Images