An image generation method, electronic device, and medium

Abstract

This invention discloses an image generation method, electronic device, and medium. The method includes the following steps: obtaining original text, a first concept, and a second concept with the opposite meaning to the first concept; encoding the original text, the first concept, and the second concept to obtain an original text vector, a first concept vector, and a second concept vector; subtracting the first concept vector from the second concept vector to obtain a concept rendering vector; scaling the concept rendering vector and adding it to the original text vector to obtain a rendered text vector; concatenating a learnable prefix to the original text to obtain a target text; encoding the target text to obtain a target text vector; obtaining the optimal target text that is closest to the rendered text by maximizing the cosine similarity between the rendered text vector and the target text vector; and inputting the original text and the optimal target text into a text-to-image model to obtain the original image and the target image.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to an image generation method, electronic device, and medium. Background Technology

[0002] Text-to-image (TPE) models (such as Stablediffusion) have garnered widespread attention due to their outstanding generative capabilities. These models can create stunning works of art based on user-input text. Applications range from scene design and animation creation to 3D scene generation. However, because the generated image and text have a one-to-one relationship, it's difficult for users to further customize or transfer the style of their generated images. Therefore, enabling users to customize the style of their creations has become a pressing issue.

[0003] Existing work (such as diffusion inversion or Dreambooth) can enable user-defined style modifications; however, they require fine-tuning the entire network to preserve entities from the user's original text for reuse in style transfer tasks. This fine-tuning is time-consuming and consumes additional training resources. Summary of the Invention

[0004] To address the shortcomings of existing technologies, embodiments of the present invention provide an image generation method, an electronic device, and a medium.

[0005] In a first aspect, embodiments of the present invention provide an image generation method, the method comprising the following steps:

[0006] Obtain the original text, the first concept, and the second concept with the opposite meaning to the first concept; encode the original text, the first concept, and the second concept to obtain the original text vector, the first concept vector, and the second concept vector;

[0007] Subtract the first concept vector from the second concept vector to obtain the concept rendering vector; scale the concept rendering vector and add it to the original text vector to obtain the rendered text vector.

[0008] Learnable prefixes are concatenated before the original text to obtain the target text; the target text is then encoded to obtain the target text vector.

[0009] By maximizing the cosine similarity between the rendered text vector and the target text vector, the optimal target text that is most similar to the rendered text is obtained;

[0010] The original text and the optimal target text are input into the text-to-image model respectively to obtain the original image and the target image.

[0011] Secondly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described image generation method.

[0012] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described image generation method.

[0013] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described image generation method.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] This invention provides an image generation method that generates original and target images by inputting the original text and the optimal target text obtained after concept rendering into a text-based image model. This method requires no prior knowledge and can automatically generate target images. The images generated by this invention retain the semantic features of the original image to the greatest extent possible and improve generation efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart of the image generation method provided in an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of the structure of the image generation method provided in the embodiment of the present invention;

[0019] Figure 3 Example diagram of the generated image provided in the embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0023] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides an image generation method, which specifically includes the following steps:

[0024] Step S1: Obtain the original text, the first concept, and the second concept that has the opposite meaning to the first concept; encode the original text, the first concept, and the second concept to obtain the original text vector, the first concept vector, and the second concept vector.

[0025] Furthermore, in this example, the original text is defined as a sentence of length L, denoted as p. t = [p1, p2, ..., p L ].

[0026] For example, suppose we assume the first concept representing the concept of "woman". r + The first concept is "woman," and the second concept is "man."

[0027] Furthermore, given an encoder f(·), the original text, the first concept, and the second concept are encoded to obtain the original text vector f(p). t ), first concept vector f(r) + ), second concept vector f(r) - ).

[0028] Step S2: Subtract the first concept vector from the second concept vector to obtain the concept rendering vector; scale the concept rendering vector and add it to the original text vector to obtain the rendered text vector.

[0029] Furthermore, in vector space, the analogical relationship between vectors can be represented as the additivity and subtraction property between words with similar meanings, such as f('Queen') - f('woman') + f('man') = f('king'). In this example, by using the analogical property of vector space to subtract the first concept vector and the second concept vector, we obtain the concept rendering vector r, as expressed below:

[0030] r = f(r)+ )-f(r - )

[0031] The concept rendering vector is scaled and then added to the original text vector to obtain the rendered text vector, as shown in the following expression:

[0032] f(p r )=f(p t )+λ·r

[0033] In the formula, f(p) r ) represents the rendered text vector, and λ represents the scaling ratio.

[0034] It should be noted that this example utilizes the analogical properties of vector space to perform conceptual rendering on the high-dimensional representation of the original text, thereby achieving semantic transfer from the original text to the target concept.

[0035] Step S3: Concatenate a learnable prefix before the original text to obtain the target text; encode the target text to obtain the target text vector.

[0036] Specifically, for each original text pt, a learnable prefix k is added to it in the text space, denoted as [v1,...,v...]. i ,..,v k ]; where i represents the i-th position, i∈k, v i ∈V, where V is a predefined vocabulary. For example, in CLIP's text space, V is a vocabulary of length 49408.

[0037] Furthermore, the expression for the target text pa is as follows:

[0038] p a = [v1, ..., v i ,..,v k p1, ..., p L ]

[0039] Encode the target text to obtain the target text vector f(p) a ).

[0040] Step S4: By maximizing the cosine similarity between the rendered text vector and the target text vector, the optimal target text that is most similar to the rendered text is obtained.

[0041] Specifically, the cosine similarity between the rendered text vector and the target text vector is defined as the learning objective. By maximizing the cosine similarity between these two vectors, the learnable target text is optimized. The learning objective is:

[0042]

[0043] Furthermore, step S4 also includes:

[0044] For each learnable word v i Set the gradient and update the learnable prefix k in the target text using PGD (Projected Gradient Descent). Search in the discrete text space to optimize and obtain the optimal target text pa that satisfies the learning objective.

[0045] Step S5: Input the original text and the optimal target text into the text-to-image model respectively to obtain the original image and the target image.

[0046] Specifically, the original text pt and the optimal target text pa are input into the stable diffusion model, which is represented as M(·) to obtain the original image M(p). t ) and target image M(p a The original image M(p) t ) and target image M(p a They are semantically similar. The image M(p) generated from the optimal target text is... a It contains specific concepts.

[0047] Example 1

[0048] Figure 3 (A) in the image is the first example image generated. Figure 3 In (A), the original text is "A photo of a chocolate cake" and the concept is "strawberry". After concept rendering, the optimal target text is "gergeous strawafantastic A photo of a chocolate cake". The original text and the optimal target are input into the text-generated image model to obtain the original image of the chocolate cake and the target image of the chocolate cake with strawberries.

[0049] Figure 3 (B) in the image is the first example image generated. Figure 3 In (B), the original text is “A photo of a king” and the concept is “woman”. After concept rendering, the optimal target text is “glumeryfemalasex A photo of a king”. The original text and the optimal target are input into the text-based image model to obtain the original photo of the king and the target image of the king with female features.

[0050] like Figure 4 As shown, this application provides an electronic device including a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, they implement the method as described in any of the first aspects above.

[0051] The system also includes a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0052] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0053] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor 102, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0054] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can also be implemented in other ways. The method and system embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0055] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0056] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by processor 102, the computer program implements the methods described in any of the first aspects above. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory 101 (ROM), random access memory 101 (RAM), magnetic disks, or optical disks.

[0057] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0058] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for generating an image, characterized in that, The method includes the following steps: Obtain the original text, the first concept, and the second concept with the opposite meaning to the first concept; encode the original text, the first concept, and the second concept to obtain the original text vector, the first concept vector, and the second concept vector; Subtract the first concept vector from the second concept vector to obtain the concept rendering vector; scale the concept rendering vector and add it to the original text vector to obtain the rendered text vector. Learnable prefixes are concatenated before the original text to obtain the target text; the target text is then encoded to obtain the target text vector. By maximizing the cosine similarity between the rendered text vector and the target text vector, the optimal target text that is most similar to the rendered text is obtained; The original text and the optimal target text are input into the text-to-image model respectively to obtain the original image and the target image; Among them, by concatenating learnable prefixes before the original text, the target text includes: Learnable prefixes , recorded as Where i represents the i-th position, i∈k, , It is a predefined vocabulary; The target text is obtained by prepending a learnable prefix to the original text; the expression is as follows: ； In the formula, Indicates the target text. L represents the original text; The process of obtaining the optimal target text that is closest to the rendered text also includes: Learnable prefix Every learnable word Set gradients and learnable prefixes in the target text. Gradient descent updates are performed to optimize and obtain the optimal target text vector that maximizes the cosine similarity between the rendered text vector and the target text vector. .

2. The image generation method according to claim 1, characterized in that, The concept rendering vector is scaled and then added to the original text vector to obtain the rendered text vector, which includes: ； In the formula, This indicates the text vector to be rendered. Represents the original text vector. Indicates the scaling ratio. This represents the concept rendering vector.

3. The image generation method according to claim 1, characterized in that, By maximizing the cosine similarity between the rendered text vector and the target text vector, the optimal target text that is most similar to the rendered text is obtained, including: The cosine similarity between the rendered text vector and the target text vector is defined as the learning objective. By maximizing the cosine similarity between these two vectors, the learnable target text is optimized. The learning objective is: ； In the formula, This indicates the text vector to be rendered. This represents the target text vector.

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the image generation method according to any one of claims 1-3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image generation method as described in any one of claims 1-3.

6. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the image generation method according to any one of claims 1-3.

Images