Image generation method and device and storage medium

Abstract

The invention relates to an image generation method and device and a storage medium, and the method comprises the steps: obtaining a style reference image and a description text, and carrying out the semantic erasing of the style reference image, and obtaining the style reference image after semantic erasing; and generating a target image based on the style reference image after semantic erasing and the description text. The semantic erasing processing is performed on the style reference image, so that the semantics of the style reference image does not cover the semantics of the description text, the semantic leakage problem is avoided, and the influence on the generation of the target image is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device and storage medium. Background Art

[0002] In the field of computer vision, style transfer is an important type of image editing task. Style transfer refers to the transfer of image style, that is, transferring the style of an image with artistic characteristics to a natural image, so that the original natural image retains the original content while having a unique artistic style.

[0003] In the related art, the style transfer method based on style loss only transfers the color and texture of the style reference image, and has poor generalization. The style transfer method based on stable diffusion has the problem of semantic leakage, that is, the semantics of the style reference image will cover the semantics of the description text, affecting the generation of the stylized image. Summary of the invention

[0004] In order to solve the above-mentioned style transfer method based on style loss, only the color and texture of the style reference image are transferred, and the generalization is poor. The style transfer method based on stable diffusion has the problem of semantic leakage, that is, the semantics of the style reference image will cover the semantics of the description text, affecting the technical problem of the generation of the stylized image. The embodiment of the present application provides an image generation method, device, electronic device and storage medium. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a method for generating an image is first provided, the method comprising:

[0006] Acquire a style reference image and a description text, and perform semantic erasure processing on the style reference image to obtain the style reference image after semantic erasure;

[0007] A target image is generated based on the semantically erased style reference image and the description text.

[0008] In an optional implementation, performing semantic erasure processing on the style reference image to obtain the semantically erased style reference image includes:

[0009] Performing cropping processing on the style reference image to obtain an image block in the style reference image;

[0010] Performing an inversion process on the image block to obtain the inverted image block;

[0011] The inverted image block is determined as the style reference image after semantic erasure.

[0012] In an optional implementation, performing semantic erasure processing on the style reference image to obtain the semantically erased style reference image includes:

[0013] Shuffling the style reference image to obtain an image with shuffled content;

[0014] The image with disrupted contents is determined as the style reference image after semantic erasure.

[0015] In an optional implementation, shuffling the style reference image to obtain an image with shuffled content includes:

[0016] Acquire a first noise matrix and a second noise matrix, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image;

[0017] According to the first noise matrix and the second noise matrix, a relative coordinate matrix is ​​formed;

[0018] For any relative coordinate in the relative coordinate matrix, searching the style reference image for a pixel corresponding to the relative coordinate;

[0019] The pixels are placed at corresponding positions of the relative coordinates in the relative coordinate matrix to obtain an image with scrambled contents.

[0020] In an optional implementation, the acquiring the first noise matrix and the second noise matrix includes:

[0021] Obtaining a randomly generated first noise matrix and a second noise matrix;

[0022] The first element in the first noise matrix and the second element in the second noise matrix are both within a preset value range.

[0023] In an optional implementation, forming a relative coordinate matrix according to the first noise matrix and the second noise matrix includes:

[0024] For any first element in the first noise matrix, determining a first position of the first element in the first noise matrix;

[0025] According to the first position, searching the second noise matrix for a second element corresponding to the first element;

[0026] Combine the first element and the second element to obtain a first relative coordinate, and place the first relative coordinate at the first position to obtain a relative coordinate matrix;

[0027] or,

[0028] For any second element in the second noise matrix, determining a second position of the second element in the second noise matrix;

[0029] According to the second position, searching the first noise matrix for a first element corresponding to the second element;

[0030] The first element and the second element are combined to obtain a second relative coordinate, and the second relative coordinate is placed at the second position to obtain a relative coordinate matrix.

[0031] In an optional implementation, searching the style reference image for pixels corresponding to the relative coordinates includes:

[0032] According to the size of the style reference image, the relative coordinates are converted to obtain image coordinates;

[0033] The pixel corresponding to the image coordinate is searched in the style reference image, and the pixel corresponding to the image coordinate is determined as the pixel corresponding to the relative coordinate.

[0034] In an optional implementation, generating a target image based on the semantically erased style reference image and the description text includes:

[0035] Vectorizing the style reference image after semantic erasure to obtain an image embedding vector;

[0036] The description text is vectorized to obtain a text embedding vector, and a target image is generated according to the image embedding vector and the text embedding vector.

[0037] In an optional implementation, generating a target image according to the image embedding vector and the text embedding vector includes:

[0038] Assigning corresponding weights to the image embedding vectors;

[0039] The weighted image embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0040] In an optional implementation, generating a target image according to the image embedding vector and the text embedding vector includes:

[0041] Obtaining a preset learnable embedding vector, wherein the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector;

[0042] Generate a style embedding vector according to the preset learnable embedding vector and the image embedding vector;

[0043] A target image is generated according to the style embedding vector and the text embedding vector.

[0044] In an optional implementation, generating a style embedding vector according to the preset learnable embedding vector and the image embedding vector includes:

[0045] Inputting the preset learnable embedding vector and the image embedding vector into a pre-trained attention model to obtain a style embedding vector;

[0046] The pre-trained attention model performs a cross-attention operation on the preset learnable embedding vector and the image embedding vector to generate a style embedding vector;

[0047] The step of generating a target image according to the style embedding vector and the text embedding vector comprises:

[0048] The style embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0049] In an optional implementation, before executing the step of acquiring the style reference image and the description text, the method further includes:

[0050] Acquire a sample style reference image and a sample description text, and perform semantic erasure processing on the sample style reference image to obtain the sample style reference image after semantic erasure;

[0051] Vectorizing the sample style reference image after semantic erasure to obtain a sample image embedding vector;

[0052] Vectorizing the sample description text to obtain a sample text embedding vector;

[0053] Inputting the sample image embedding vector and the preset learnable embedding vector into the attention model to obtain a sample style embedding vector;

[0054] Inputting the sample style embedding vector and the sample text embedding vector into a noise prediction model to obtain a predicted image;

[0055] Determining a loss value between the predicted image and a style image corresponding to the sample style reference image;

[0056] Training the attention model and the noise prediction model according to the loss value;

[0057] When the loss value converges, the training is stopped to obtain a pre-trained attention model and a pre-trained noise prediction model.

[0058] In an optional implementation, performing semantic erasure processing on the sample style reference image to obtain the semantically erased sample style reference image includes:

[0059] Performing cropping processing on the sample style reference image to obtain a sample image block in the sample style reference image;

[0060] Performing an inversion process on the sample image block to obtain the inverted sample image block;

[0061] The inverted sample image block is determined as the sample style reference image after semantic erasure.

[0062] In an optional implementation, performing semantic erasure processing on the sample style reference image to obtain the semantically erased sample style reference image includes:

[0063] Shuffling the sample style reference image to obtain a sample image with shuffled content;

[0064] The sample image with disrupted content is determined as the sample style reference image after semantic erasure.

[0065] In an optional implementation, shuffling the sample style reference image to obtain a sample image with shuffled content includes:

[0066] Obtaining a first sample noise matrix and a second sample noise matrix, wherein the size of the first sample noise matrix and the size of the second sample noise matrix are both the same as the size of the sample style reference image;

[0067] According to the first sample noise matrix and the second sample noise matrix, a sample relative coordinate matrix is ​​formed;

[0068] For any sample relative coordinate in the sample relative coordinate matrix, searching for a sample pixel corresponding to the sample relative coordinate from the sample style reference image;

[0069] The sample pixel is placed at a corresponding sample position of the sample relative coordinate in the sample relative coordinate matrix to obtain a sample image with disrupted content.

[0070] In an optional implementation, the acquiring the first sample noise matrix and the second sample noise matrix includes:

[0071] Obtaining a randomly generated first sample noise matrix and a second sample noise matrix;

[0072] The first sample elements in the first sample noise matrix and the second sample elements in the second sample noise matrix are both within a preset value range.

[0073] In an optional implementation, forming a sample relative coordinate matrix according to the first sample noise matrix and the second sample noise matrix includes:

[0074] For any first sample element in the first sample noise matrix, determining a first sample position of the first sample element in the first sample noise matrix;

[0075] According to the first sample position, searching the second sample noise matrix for a second sample element corresponding to the first sample element;

[0076] Combine the first sample element and the second sample element to obtain a first sample relative coordinate;

[0077] Put the first sample relative coordinates at the first sample position to obtain a sample relative coordinate matrix;

[0078] or,

[0079] For any second sample element in the second sample noise matrix, determining a second sample position of the second sample element in the second sample noise matrix;

[0080] According to the second sample position, searching the first sample element corresponding to the second sample element from the first sample noise matrix;

[0081] Combine the first sample element and the second sample element to obtain a second sample relative coordinate;

[0082] The second sample relative coordinates are placed on the second sample position to obtain a sample relative coordinate matrix.

[0083] In an optional implementation, searching the sample style reference image for sample pixels corresponding to the sample relative coordinates includes:

[0084] According to the size of the sample style reference image, the sample relative coordinates are converted to obtain sample image coordinates;

[0085] Searching for sample pixels corresponding to the sample image coordinates from the sample style reference image;

[0086] The sample pixel corresponding to the sample image coordinate is determined as the sample pixel corresponding to the sample relative coordinate.

[0087] In a second aspect of the embodiments of the present application, an image generating device is further provided, the device comprising:

[0088] Image and text acquisition module, used to obtain style reference images and description text;

[0089] An image processing module, used for performing semantic erasure processing on the style reference image to obtain the style reference image after semantic erasure;

[0090] The image generation module is used to generate a target image based on the style reference image after semantic erasure and the description text.

[0091] In an optional implementation, the image processing module is specifically used to:

[0092] Performing cropping processing on the style reference image to obtain an image block in the style reference image;

[0093] Performing an inversion process on the image block to obtain the inverted image block;

[0094] The inverted image block is determined as the style reference image after semantic erasure.

[0095] In an optional implementation, the image processing module specifically includes:

[0096] An image shuffling submodule, used for shuffling the style reference image to obtain an image with shuffled content;

[0097] The image determination submodule is used to determine the image with disrupted content as the style reference image after semantic erasure.

[0098] In an optional implementation, the image processing submodule specifically includes:

[0099] A matrix acquisition unit, used to acquire a first noise matrix and a second noise matrix, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image;

[0100] A matrix forming unit, used for forming a relative coordinate matrix according to the first noise matrix and the second noise matrix;

[0101] A pixel search unit, configured to search, for any relative coordinate in the relative coordinate matrix, a pixel corresponding to the relative coordinate from the style reference image;

[0102] The pixel processing unit is used to place the pixel at a corresponding position of the relative coordinate in the relative coordinate matrix to obtain an image with scrambled content.

[0103] In an optional implementation, the matrix acquisition unit is used to:

[0104] Obtaining a randomly generated first noise matrix and a second noise matrix;

[0105] The first element in the first noise matrix and the second element in the second noise matrix are both within a preset value range.

[0106] In an optional implementation, the matrix composition unit is specifically used for:

[0107] For any first element in the first noise matrix, determining a first position of the first element in the first noise matrix;

[0108] According to the first position, searching the second noise matrix for a second element corresponding to the first element;

[0109] Combine the first element and the second element to obtain a first relative coordinate, and place the first relative coordinate at the first position to obtain a relative coordinate matrix;

[0110] or,

[0111] For any second element in the second noise matrix, determining a second position of the second element in the second noise matrix;

[0112] According to the second position, searching the first noise matrix for a first element corresponding to the second element;

[0113] The first element and the second element are combined to obtain a second relative coordinate, and the second relative coordinate is placed at the second position to obtain a relative coordinate matrix.

[0114] In an optional implementation, the pixel search unit is specifically used to:

[0115] According to the size of the style reference image, the relative coordinates are converted to obtain image coordinates;

[0116] The pixel corresponding to the image coordinate is searched in the style reference image, and the pixel corresponding to the image coordinate is determined as the pixel corresponding to the relative coordinate.

[0117] In an optional implementation, the image generation module specifically includes:

[0118] An image vectorization processing submodule is used to perform vectorization processing on the style reference image after semantic erasure to obtain an image embedding vector;

[0119] A text vectorization processing submodule, used to perform vectorization processing on the description text to obtain a text embedding vector;

[0120] The image generation submodule is used to generate a target image according to the image embedding vector and the text embedding vector.

[0121] In an optional implementation, the image generation submodule is specifically used for:

[0122] Assigning corresponding weights to the image embedding vectors;

[0123] The weighted image embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0124] In an optional implementation, the image generation module specifically includes:

[0125] A vector acquisition unit, configured to acquire a preset learnable embedding vector, wherein the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector;

[0126] A vector generating unit, configured to generate a style embedding vector according to the preset learnable embedding vector and the image embedding vector;

[0127] An image generating unit is used to generate a target image according to the style embedding vector and the text embedding vector.

[0128] In an optional implementation, the vector generation unit is specifically used to:

[0129] Inputting the preset learnable embedding vector and the image embedding vector into a pre-trained attention model to obtain a style embedding vector;

[0130] The pre-trained attention model performs a cross-attention operation on the preset learnable embedding vector and the image embedding vector to generate a style embedding vector;

[0131] The image generation unit is specifically used for:

[0132] The style embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0133] In an optional embodiment, the device further comprises:

[0134] A sample style reference image and sample description text acquisition module is used to acquire a sample style reference image and a sample description text;

[0135] A sample style reference image processing module, used for performing semantic erasure processing on the sample style reference image to obtain the sample style reference image after semantic erasure;

[0136] An image vectorization processing module, used for performing vectorization processing on the sample style reference image after semantic erasure to obtain a sample image embedding vector;

[0137] A text vectorization processing module, used to perform vectorization processing on the sample description text to obtain a sample text embedding vector;

[0138] A sample style embedding vector generation module, used for inputting the sample image embedding vector and a preset learnable embedding vector into an attention model to obtain a sample style embedding vector;

[0139] An image prediction module, used for inputting the sample style embedding vector and the sample text embedding vector into a noise prediction model to obtain a predicted image;

[0140] A loss value determination module, used to determine a loss value between the predicted image and a style image corresponding to the sample style reference image;

[0141] A model training module is used to train the attention model and the noise prediction model according to the loss value; when the loss value converges, the training is stopped to obtain a pre-trained attention model and a pre-trained noise prediction model.

[0142] In an optional implementation, the sample style reference image processing module is specifically used to:

[0143] Performing cropping processing on the sample style reference image to obtain a sample image block in the sample style reference image;

[0144] Performing an inversion process on the sample image block to obtain the inverted sample image block;

[0145] The inverted sample image block is determined as the sample style reference image after semantic erasure.

[0146] In an optional implementation, the sample style reference image processing module specifically includes:

[0147] A sample style reference image shuffling submodule, used for shuffling the sample style reference image to obtain a sample image with shuffled content;

[0148] The sample style reference image determination submodule is used to determine the sample image with disrupted content as the sample style reference image after semantic erasure.

[0149] In an optional implementation, the sample style reference image shuffling submodule specifically includes:

[0150] A noise matrix acquisition unit, used to acquire a first sample noise matrix and a second sample noise matrix, wherein the size of the first sample noise matrix and the size of the second sample noise matrix are both the same as the size of the sample style reference image;

[0151] A coordinate matrix forming unit, used for forming a sample relative coordinate matrix according to the first sample noise matrix and the second sample noise matrix;

[0152] A sample pixel search unit, used for searching, for any sample relative coordinate in the sample relative coordinate matrix, a sample pixel corresponding to the sample relative coordinate from the sample style reference image;

[0153] The sample pixel processing unit is used to place the sample pixel at the corresponding sample position of the sample relative coordinate in the sample relative coordinate matrix to obtain a sample image with disrupted content.

[0154] In an optional implementation manner, the noise matrix acquisition unit is specifically used to:

[0155] Obtaining a randomly generated first sample noise matrix and a second sample noise matrix;

[0156] The first sample elements in the first sample noise matrix and the second sample elements in the second sample noise matrix are both within a preset value range.

[0157] In an optional implementation, the coordinate matrix composition unit is specifically used for:

[0158] For any first sample element in the first sample noise matrix, determining a first sample position of the first sample element in the first sample noise matrix;

[0159] According to the first sample position, searching the second sample noise matrix for a second sample element corresponding to the first sample element;

[0160] Combine the first sample element and the second sample element to obtain a first sample relative coordinate;

[0161] Put the first sample relative coordinates at the first sample position to obtain a sample relative coordinate matrix;

[0162] or,

[0163] For any second sample element in the second sample noise matrix, determining a second sample position of the second sample element in the second sample noise matrix;

[0164] According to the second sample position, searching the first sample element corresponding to the second sample element from the first sample noise matrix;

[0165] Combine the first sample element and the second sample element to obtain a second sample relative coordinate;

[0166] The second sample relative coordinates are placed on the second sample position to obtain a sample relative coordinate matrix.

[0167] In an optional implementation, the sample pixel search unit is specifically configured to:

[0168] According to the size of the sample style reference image, the sample relative coordinates are converted to obtain sample image coordinates;

[0169] Searching for sample pixels corresponding to the sample image coordinates from the sample style reference image;

[0170] The sample pixel corresponding to the sample image coordinate is determined as the sample pixel corresponding to the sample relative coordinate.

[0171] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0172] Memory, used to store computer programs;

[0173] The processor is used to implement any image generation method described in the first aspect when executing the program stored in the memory.

[0174] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes any image generation method described in the first aspect above.

[0175] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image generation methods.

[0176] The technical solution provided in the embodiment of the present application obtains a style reference image and a description text, and performs semantic erasure processing on the style reference image to obtain a style reference image after semantic erasure, and generates a target image based on the style reference image after semantic erasure and the description text.

[0177] By performing semantic erasure processing on the style reference image, a semantically erased style reference image is obtained, and then a target image is generated based on the semantically erased style reference image and the description text. By performing semantic erasure processing on the style reference image, the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and avoiding affecting the generation of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0178] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0179] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0180] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0181] Figure 1 A schematic diagram of an implementation flow of an image generation method shown in an embodiment of the present application;

[0182] Figure 2 A schematic diagram of a style reference image shown in an embodiment of the present application;

[0183] Figure 3 A schematic diagram of a stylized image shown in an embodiment of the present application;

[0184] Figure 4 It is a schematic diagram of an implementation process of another image generation method shown in an embodiment of the present application;

[0185] Figure 5 A schematic diagram of a target image shown in an embodiment of the present application;

[0186] Figure 6 It is a schematic diagram of an implementation process of another image generation method shown in an embodiment of the present application;

[0187] Figure 7 It is a schematic diagram of an implementation process of another image generation method shown in an embodiment of the present application;

[0188] Figure 8A schematic diagram of an implementation process of an image shuffling method shown in an embodiment of the present application;

[0189] Fig. 9 A schematic diagram of a comparison between a style reference image and an image with disrupted contents shown in an embodiment of the present application;

[0190] Fig.10 A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present application;

[0191] Fig.11 is a schematic structural diagram of an image generating device shown in an embodiment of the present application;

[0192] Fig.12 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0193] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0194] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0195] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0196] S101, obtaining a style reference image and description text, and performing semantic erasure processing on the style reference image to obtain a semantically erased style reference image.

[0197] In an embodiment of the present application, a style reference image may be obtained, and the style reference image is used to provide a corresponding style reference so as to facilitate the transfer of the style in the style reference image. Figure 2 The style reference image shown can be used to transfer the style, where the style includes but is not limited to color, texture, composition, artistic style, etc.

[0198] In addition to obtaining the style reference image, the description text can also be obtained. The description text is used to provide the corresponding content, so as to generate an image of the corresponding content, and the image transfers the style in the style reference image. For example, if the description text is "A dog", an image with content related to "dog" can be generated, and the image transfers the style in the style reference image.

[0199] For style reference images, there are corresponding semantics, such as Figure 2 The style reference image shown in the figure has a semantic representation of "lizard". When its semantics is strong, it will cover the semantics of the description text, causing semantic leakage and affecting the generation of stylized images. For example, the description text is "A man". Figure 2 The style reference image and description text shown in the figure generate a stylized image. The semantics of the stylized image should theoretically represent "people", but its semantic representation is still "lizard", such as Figure 3 As shown, the reason is that the semantics of the description text does not work and is Figure 2 The semantic coverage of the style reference image is shown.

[0200] To this end, in the embodiment of the present application, semantic erasure processing is performed on the style reference image to obtain a semantically erased style reference image, thereby weakening the semantics of the style reference image to prevent it from covering the semantics of the description text, causing semantic leakage problems, and affecting the generation of the stylized image (i.e., the subsequent target image). The technical means for semantic erasure processing will be described in detail later.

[0201] S102, generating a target image based on the semantically erased style reference image and the description text.

[0202] In the embodiment of the present application, for the semantically erased style reference image and the above-mentioned description text, a target image can be generated according to the semantically erased style reference image and the description text.

[0203] By performing semantic erasure processing on the style reference image in this way, the semantics of the style reference image does not cover the semantics of the description text, thereby avoiding the semantic leakage problem and avoiding affecting the generation of the target image.

[0204] like Figure 4 FIG. 1 is a schematic diagram of an implementation flow of another image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0205] S401, obtaining a style reference image and description text, and performing semantic erasure processing on the style reference image to obtain a semantically erased style reference image.

[0206] In the embodiment of the present application, this step is similar to the above-mentioned step S101, and the embodiment of the present application will not be described one by one here.

[0207] S402, performing vectorization processing on the semantically erased style reference image to obtain an image embedding vector.

[0208] In an embodiment of the present application, the style reference image after semantic erasure can be vectorized to obtain an image embedding vector, i.e., image embedding. An image encoder can be used to encode the style reference image after semantic erasure to obtain an image embedding vector, thereby completing the vectorization of the style reference image after semantic erasure.

[0209] It should be noted that the image encoder can be any common image encoder, including but not limited to CLIP (pre-trained model based on contrastive text-image pairs), Dino (unlabeled knowledge distillation), etc., and the embodiments of the present application are not limited to this.

[0210] S403, vectorizing the description text to obtain a text embedding vector, and generating a target image according to the image embedding vector and the text embedding vector.

[0211] In the embodiment of the present application, for the description text, the description text can be vectorized to obtain a text embedding vector, i.e., text embedding. Among them, a text encoder can be used to encode the description text to obtain a text embedding vector, thereby completing the vectorization of the description text.

[0212] It should be noted that the text encoder, i.e., text encoder, can be any common text encoder, including but not limited to CLIP (a pre-trained model based on contrasting text-image pairs), and the embodiments of the present application are not limited to this.

[0213] For the image embedding vector and the text embedding vector, a target image can be generated according to the image embedding vector and the text embedding vector, where the target image is a stylized image. Due to the semantic erasure processing of the style reference image, the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the semantic leakage problem and avoiding affecting the generation of the target image.

[0214] For example, the description text is "Adog". Based on the style reference image and the description text, a target image is generated. Here, the style reference image is semantically erased so that the semantics of the style reference image does not cover the semantics of the description text, thereby avoiding the semantic leakage problem and avoiding affecting the generation of the target image. The generated target image is as follows: Figure 5 As shown, its semantic representation is "dog", which is the same as the semantics of the description text representation, but the target image transfers the style of the style reference image.

[0215] Through the above description of the technical solution provided in the embodiment of the present application, a style reference image and a description text are obtained, and the style reference image is semantically erased to obtain a style reference image after semantic erasure, the style reference image after semantic erasure is vectorized to obtain an image embedding vector, the description text is vectorized to obtain a text embedding vector, and a target image is generated based on the image embedding vector and the text embedding vector.

[0216] By performing semantic erasure processing on the style reference image, a semantically erased style reference image is obtained, and then the semantically erased style reference image and the description text are vectorized to obtain an image embedding vector and a text embedding vector. According to the image embedding vector and the text embedding vector, a target image is generated. In this way, the semantic erasure processing is performed on the style reference image, so that the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and avoiding affecting the generation of the target image.

[0217] In addition, in the embodiment of the present application, the semantic erasure processing of the style reference image is essentially to randomly crop and reverse the style reference image to erase its semantics. Figure 6 FIG. 1 is a schematic diagram of an implementation flow of another image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0218] S601, obtaining a style reference image and description text, and performing cropping processing on the style reference image to obtain an image block in the style reference image.

[0219] S602, performing inversion processing on the image block to obtain an inverted image block.

[0220] S603: Determine the inverted image block as the style reference image after semantic erasure.

[0221] In the embodiment of the present application, a style reference image and a description text may be obtained, wherein the style reference image needs to be subjected to semantic erasure processing, and the semantic erasure processing specifically refers to cropping and inversion.

[0222] Specifically, the style reference image is cropped to obtain an image block in the style reference image, the image block is inverted to obtain an inverted image block, and the inverted image block is determined as the semantically erased style reference image. In this way, the style reference image is cropped, inverted, etc. to erase its semantics.

[0223] It should be noted that, for the style reference image, random cropping can be performed, so that image blocks in the style reference image can be randomly obtained. In addition, for the image block, a mirror inversion process in the horizontal direction or a mirror inversion process in the vertical direction can be performed, and of course other inversion processes can also be performed, which is not limited in the embodiments of the present application.

[0224] S604, performing vectorization processing on the semantically erased style reference image to obtain an image embedding vector.

[0225] In the embodiment of the present application, this step is similar to the above-mentioned step S402, and the embodiment of the present application will not be described one by one here.

[0226] S605, obtaining a preset learnable embedding vector, where the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector.

[0227] S606, generating a style embedding vector according to a preset learnable embedding vector and an image embedding vector.

[0228] In an embodiment of the present application, a learnable embedding vector, namely, learning embedding, is pre-set, and the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector, but the length of the preset learnable embedding vector may be different from the length of the image embedding vector.

[0229] For example, a 4xn learning embedding is pre-set, 4 is the length of the learning embedding, which can be set to 4, but is not limited to 4, and n is the dimension of the learning embedding, which remains the same as the dimension of the image embedding.

[0230] Based on this, a preset learnable embedding vector can be obtained, and a style embedding vector can be generated according to the preset learnable embedding vector and the image embedding vector. A pre-trained attention model is provided, which includes N attention blocks.

[0231] Thus, the preset learnable embedding vector and the image embedding vector can be input into the pre-trained attention model to obtain the style embedding vector. The pre-trained attention model performs a cross-attention operation on the preset learnable embedding vector and the image embedding vector to generate a style embedding vector, namely, style embedding.

[0232] It should be noted that randomly cropping and reversing the style reference image to erase its semantics can erase the semantics of the style reference image to a certain extent. The style reference image after semantic erasure may still partially retain the semantics of the style reference image. For this reason, for an image encoder, such as CLIP, which is a pre-trained model based on contrasting text-image pairs, it is used to encode the style reference image after semantic erasure to obtain an image embedding vector. The image embedding vector will introduce the semantics of the style reference image that may partially remain in the style reference image after semantic erasure. For this reason, a learnable embedding vector is pre-set. In the pre-trained attention model, a cross-attention operation is performed on the preset learnable embedding vector and the image embedding vector to learn the style features of the style reference image, thereby generating a style embedding vector. In this way, the semantics of the style reference image carried in the image embedding vector can be erased twice, so that the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the semantic leakage problem and avoiding affecting the generation of the target image.

[0233] S607, vectorizing the description text to obtain a text embedding vector, and generating a target image according to the style embedding vector and the text embedding vector.

[0234] In an embodiment of the present application, for the description text, the description text can be vectorized to obtain a text embedding vector, i.e., text embedding. Among them, a text encoder can be used to encode the description text to obtain a text embedding vector, thereby completing the vectorization of the description text.

[0235] For the style embedding vector and the text embedding vector, a target image can be generated according to the style embedding vector and the text embedding vector. Among them, a pre-trained noise prediction model is provided, which can be used to generate an image.

[0236] Based on this, the style embedding vector and the text embedding vector are input into the pre-trained noise prediction model to obtain the target image. The pre-trained noise prediction model includes but is not limited to Unet, and the pre-trained noise prediction model can perform cross-attention operations on the style embedding vector and the text embedding vector, so that the generated target image can not only transfer the color, texture, composition, artistic style, etc. in the style reference image, but also well depict the content of the description text.

[0237] By performing semantic erasure processing on the style reference image, a semantically erased style reference image is obtained, and then the semantically erased style reference image and the description text are respectively vectorized to obtain an image embedding vector and a text embedding vector. A style embedding vector is generated based on the preset learnable embedding vector and the image embedding vector, and a target image is generated based on the style embedding vector and the text embedding vector. By performing semantic erasure processing on the style reference image in this way, the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and avoiding affecting the generation of the target image.

[0238] In addition, a cross-attention operation is performed on the preset learnable embedding vector and the image embedding vector to learn the style features of the style reference image, thereby generating a style embedding vector. This can secondarily erase the semantics of the style reference image carried in the image embedding vector, so that the semantics of the style reference image does not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and affecting the generation of the target image. The cross-attention operation is performed on the style embedding vector and the text embedding vector, so that the generated target image can not only transfer the color, texture, composition, artistic style, etc. in the style reference image, but also well portray the content of the description text.

[0239] In addition, in the embodiment of the present application, the semantic erasure processing is performed on the style reference image, which is essentially shuffling the style reference image to erase its semantics. Figure 7 FIG. 1 is a schematic diagram of an implementation flow of another image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0240] S701, obtaining a style reference image and description text, and shuffling the style reference image to obtain an image with shuffled content.

[0241] S702: Determine the image with disrupted contents as the style reference image after semantic erasure.

[0242] In the embodiment of the present application, a style reference image and a description text may be obtained, wherein the style reference image needs to be subjected to semantic erasure processing, and the semantic erasure processing specifically refers to shuffle.

[0243] Specifically, the style reference image is shuffled to obtain a content-shuffled image, and the content-shuffled image is determined as the style reference image after semantic erasure. The size of the content-shuffled image is the same as that of the style reference image.

[0244] Among them, the shuffling process of the style reference image can be specifically referred to as follows Figure 8 As shown in the method. Figure 8FIG. 1 is a schematic diagram of an implementation flow of an image shuffling method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0245] S801, obtaining a first noise matrix and a second noise matrix, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image.

[0246] In an embodiment of the present application, a first noise matrix and a second noise matrix are obtained, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image.

[0247] For example, for a style reference image I, its size is HxW, then a first noise matrix X and a second noise matrix Y are obtained, and the sizes of the first noise matrix X and the second noise matrix Y are both HxW, which are the same as the size of the style reference image.

[0248] The first noise matrix and the second noise matrix may be randomly generated, and the value range of the elements in the first noise matrix and the second noise matrix is ​​limited to, for example, between 0 and 1, and the size is the same as that of the style reference image.

[0249] Based on this, a randomly generated first noise matrix and a second noise matrix are obtained, the first element in the first noise matrix and the second element in the second noise matrix are both in a preset numerical range, for example, between 0 and 1, and the size of the first noise matrix and the size of the second noise matrix are the same as the size of the style reference image.

[0250] S802: compose a relative coordinate matrix according to the first noise matrix and the second noise matrix.

[0251] In the embodiment of the present application, for the first noise matrix and the second noise matrix, a relative coordinate matrix can be formed according to the first noise matrix and the second noise matrix.

[0252] For example, for the first noise matrix X and the second noise matrix Y, a relative coordinate matrix C of (H, W, 2) can be formed, which means that for each position in the style reference image I, there is a corresponding relative coordinate (x, y). Since the value range of the elements in the first noise matrix and the second noise matrix is ​​between 0 and 1, the value range of the relative coordinate matrix C is also between 0 and 1, so the relative coordinate matrix C contains relative coordinates.

[0253] Specifically, for any first element in the first noise matrix, determine the first position of the first element in the first noise matrix; according to the first position, find the second element corresponding to the first element from the second noise matrix; combine the first element with the second element to obtain a first relative coordinate, put the first relative coordinate at the first position, and obtain a relative coordinate matrix.

[0254] Alternatively, for any second element in the second noise matrix, determine the second position of the second element in the second noise matrix; according to the second position, find the first element corresponding to the second element from the first noise matrix; combine the first element with the second element to obtain a second relative coordinate, and place the second relative coordinate at the second position to obtain a relative coordinate matrix.

[0255] For example, for the first element 0.5 in the first noise matrix, determine the first position (1, 1) of the first element 0.5 in the first noise matrix, and the first position (1, 1) represents the first row and the first column. At this time, find the second element 0.6 corresponding to the first position (1, 1) from the second noise matrix as the second element corresponding to the first element 0.5, combine the first element 0.5 with the second element 0.6 to obtain the first relative coordinates (0.5, 0.6), and put the first relative coordinates on the first position (1, 1). In this way, perform the above steps for each first element in the first noise matrix to finally obtain a relative coordinate matrix.

[0256] S803 , for any relative coordinate in the relative coordinate matrix, searching for a pixel corresponding to the relative coordinate from the style reference image.

[0257] S804, placing the pixels at corresponding positions of the relative coordinates in the relative coordinate matrix to obtain an image with scrambled contents.

[0258] In the embodiment of the present application, for any relative coordinate in the relative coordinate matrix, the pixel corresponding to the relative coordinate is searched from the style reference image, and the pixel is placed at the corresponding position of the relative coordinate in the relative coordinate matrix to obtain an image with disrupted content, for example, Fig. 9 Schematic diagram of the comparison between the style reference image and the image with shuffled content.

[0259] Since the relative coordinate matrix contains relative coordinates, they need to be converted into absolute coordinates before the pixels corresponding to the relative coordinates can be found from the style reference image. Based on this, the relative coordinates are converted according to the size of the style reference image to obtain the image coordinates (i.e., absolute coordinates), and the pixels corresponding to the image coordinates are found from the style reference image, and the pixels corresponding to the image coordinates are determined as the pixels corresponding to the relative coordinates.

[0260] For example, for the relative coordinates (x, y), according to the size HxW of the style reference image, the relative coordinates are transformed to obtain the image coordinates, and the conversion method is (x*H, y*W). Then, the pixels corresponding to the image coordinates are found from the style reference image, and the pixels corresponding to the image coordinates are determined as the pixels corresponding to the relative coordinates. The pixels are placed at the corresponding positions of the relative coordinates in the relative coordinate matrix, and finally an image with scrambled content is obtained.

[0261] S703, performing vectorization processing on the semantically erased style reference image to obtain an image embedding vector, and assigning a corresponding weight to the image embedding vector.

[0262] In an embodiment of the present application, the style reference image after semantic erasure can be vectorized to obtain an image embedding vector, i.e., image embedding. An image encoder can be used to encode the style reference image after semantic erasure to obtain an image embedding vector, thereby completing the vectorization of the style reference image after semantic erasure.

[0263] In addition, for the image embedding vector, a corresponding weight can be assigned to the image embedding vector to further weaken the semantics of the style reference image. The image embedding vector is generally composed of N-dimensional features, and for each dimensional feature, a corresponding weight can be assigned, and the sum of all weights is, for example, 1.

[0264] S704, vectorizing the description text to obtain a text embedding vector.

[0265] In the embodiment of the present application, for the description text, the description text can be vectorized to obtain a text embedding vector, i.e., text embedding. Among them, a text encoder can be used to encode the description text to obtain a text embedding vector, thereby completing the vectorization of the description text.

[0266] It should be noted that the text encoder, i.e., text encoder, can be any common text encoder, including but not limited to CLIP (a pre-trained model based on contrasting text-image pairs), and the embodiments of the present application are not limited to this.

[0267] S705, inputting the weighted image embedding vector and text embedding vector into a pre-trained noise prediction model to obtain a target image.

[0268] In an embodiment of the present application, the weighted image embedding vector and the text embedding vector can be input into a pre-trained noise prediction model to obtain a target image.

[0269] It should be noted that the pre-trained noise prediction model includes but is not limited to Unet, and the pre-trained noise prediction model can perform cross-attention operations on the weighted image embedding vector and the text embedding vector, so that the generated target image can not only transfer the color, texture, composition, artistic style, etc. in the style reference image, but also well portray the content of the descriptive text.

[0270] The style reference image is processed by semantic erasing to obtain the semantically erased style reference image, and then the semantically erased style reference image and the description text are vectorized to obtain the image embedding vector and the text embedding vector, and the image embedding vector is assigned corresponding weights. The weighted image embedding vector and the text embedding vector are input into the pre-trained noise prediction model to obtain the target image. In this way, the semantic erasing of the style reference image ensures that the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and affecting the generation of the target image.

[0271] In addition, the image embedding vector is given a corresponding weight to further weaken the semantics of the style reference image. The pre-trained noise prediction model can perform cross-attention operations on the weighted image embedding vector and the text embedding vector, so that the generated target image can not only transfer the color, texture, composition, artistic style, etc. in the style reference image, but also well portray the content of the descriptive text.

[0272] For the pre-trained noise prediction model, it needs to be trained in advance. Based on this, Fig.10 As shown, it is a schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0273] S1001, obtaining a sample style reference image and a sample description text, and performing semantic erasure processing on the sample style reference image to obtain a semantically erased sample style reference image.

[0274] In the embodiment of the present application, a sample style reference image and a sample description text are obtained, and a semantic erasure process is performed on the sample style reference image to obtain a semantically erased sample style reference image, wherein the semantic erasure process specifically refers to cropping and inversion.

[0275] Based on this, the sample style reference image is cropped to obtain a sample image block in the sample style reference image; the sample image block is inverted to obtain an inverted sample image block; and the inverted sample image block is determined as the sample style reference image after semantic erasure.

[0276] It should be noted that, for the sample style reference image, a random cropping process can be performed, so that a sample image block in the sample style reference image can be randomly obtained. In addition, for the sample image block, a mirror inversion process in the horizontal direction or a mirror inversion process in the vertical direction can be performed, and of course, other inversion processes can also be performed, which is not limited in the embodiments of the present application.

[0277] In addition, the semantic erasure process can also be referred to as shuffle. Based on this, the sample style reference image is shuffled to obtain a sample image with shuffled content; the sample image with shuffled content is determined as the sample style reference image after semantic erasure. The size of the sample image with shuffled content is the same as the size of the sample style reference image.

[0278] The shuffling process of the sample style reference image can specifically be to obtain a first sample noise matrix and a second sample noise matrix, wherein the sizes of the first sample noise matrix and the second sample noise matrix are the same as those of the sample style reference image; a sample relative coordinate matrix is ​​formed according to the first sample noise matrix and the second sample noise matrix; for any sample relative coordinate in the sample relative coordinate matrix, the sample pixel corresponding to the sample relative coordinate is searched from the sample style reference image; the sample pixel is placed at the corresponding sample position of the sample relative coordinate in the sample relative coordinate matrix, and a sample image with scrambled content is obtained.

[0279] The first sample noise matrix and the second sample noise matrix can be randomly generated, and the value range of the first sample noise matrix and the second sample noise matrix is ​​limited to, for example, between 0 and 1, and the size is the same as the size of the sample style reference image. Based on this, the randomly generated first sample noise matrix and the second sample noise matrix are obtained; the first sample element in the first sample noise matrix and the second sample element in the second sample noise matrix are both in the preset value range.

[0280] The first sample noise matrix and the second sample noise matrix are used to form a sample relative coordinate matrix. Specifically, for any first sample element in the first sample noise matrix, the first sample position of the first sample element in the first sample noise matrix is ​​determined; according to the first sample position, the second sample element corresponding to the first sample element is searched from the second sample noise matrix; the first sample element and the second sample element are combined to obtain the first sample relative coordinate; the first sample relative coordinate is placed on the first sample position to obtain the sample relative coordinate matrix;

[0281] Alternatively, for any second sample element in the second sample noise matrix, determine the second sample position of the second sample element in the second sample noise matrix; according to the second sample position, find the first sample element corresponding to the second sample element from the first sample noise matrix; combine the first sample element with the second sample element to obtain the second sample relative coordinates; put the second sample relative coordinates on the second sample position to obtain the sample relative coordinate matrix.

[0282] In addition, since the sample relative coordinates in the sample relative coordinate matrix are sample relative coordinates, they need to be converted into absolute coordinates before the sample pixels corresponding to the sample relative coordinates can be found from the sample style reference image. Based on this, the sample relative coordinates are converted according to the size of the sample style reference image to obtain the sample image coordinates; the sample pixels corresponding to the sample image coordinates are found from the sample style reference image; and the sample pixels corresponding to the sample image coordinates are determined as the sample pixels corresponding to the sample relative coordinates.

[0283] S1002, performing vectorization processing on the semantically erased sample style reference image to obtain a sample image embedding vector.

[0284] In an embodiment of the present application, for the semantically erased sample style reference image, vectorization processing can be performed on it to obtain a sample image embedding vector, i.e., image embedding. Among them, an image encoder can be used to encode the semantically erased sample style reference image to obtain a sample image embedding vector, thereby completing the vectorization processing of the semantically erased sample style reference image.

[0285] It should be noted that the image encoder can be any common image encoder, including but not limited to CLIP (pre-trained model based on contrastive text-image pairs), Dino (unlabeled knowledge distillation), etc., and the embodiments of the present application are not limited to this.

[0286] S1003, vectorizing the sample description text to obtain a sample text embedding vector.

[0287] In an embodiment of the present application, for the sample description text, the sample description text can be vectorized to obtain a sample text embedding vector, i.e., text embedding. Among them, a text encoder can be used to encode the sample description text to obtain a sample text embedding vector, thereby completing the vectorization of the sample description text.

[0288] It should be noted that the text encoder, i.e., text encoder, can be any common text encoder, including but not limited to CLIP (a pre-trained model based on contrasting text-image pairs), and the embodiments of the present application are not limited to this.

[0289] S1004, input the sample image embedding vector and the preset learnable embedding vector into the attention model to obtain the sample style embedding vector.

[0290] In an embodiment of the present application, a learnable embedding vector, namely, learning embedding, is pre-set, and the dimension of the preset learnable embedding vector is the same as the dimension of the sample image embedding vector, but the length of the preset learnable embedding vector may be different from the length of the sample image embedding vector.

[0291] For example, a 4xn learning embedding is pre-set, 4 is the length of the learning embedding, which can be set to 4, but is not limited to 4, and n is the dimension of the learning embedding, which remains the same as the dimension of the image embedding.

[0292] The sample image embedding vector and the preset learnable embedding vector can be input into the attention model to obtain the sample style embedding vector. The attention model performs a cross-attention operation on the sample image embedding vector and the preset learnable embedding vector to generate a sample style embedding vector, namely, style embedding.

[0293] S1005, inputting the sample style embedding vector and the sample text embedding vector into the noise prediction model to obtain a predicted image.

[0294] In an embodiment of the present application, the sample style embedding vector and the sample text embedding vector are input into a noise prediction model to obtain a predicted image, wherein the noise prediction model can perform a cross-attention operation on the sample style embedding vector and the sample text embedding vector to obtain a predicted image.

[0295] S1006, determining a loss value between the predicted image and the style image corresponding to the sample style reference image.

[0296] S1007, training the attention model and the noise prediction model according to the loss value.

[0297] S1008, when the loss value converges, stop training and obtain the pre-trained attention model and the pre-trained noise prediction model.

[0298] In an embodiment of the present application, for a sample style reference image and a sample description text, there is a corresponding style image. At this time, the loss value between the predicted avatar and the style image can be determined, and the attention model and the noise prediction model are trained according to the loss value. When the loss value converges, the training is stopped to obtain a pre-trained attention model and a pre-trained noise prediction model.

[0299] After the above model training, a pre-trained attention model and a pre-trained noise prediction model can be obtained. Subsequently, the style reference image is processed with semantic erasure to obtain a semantically erased style reference image. Then, the semantically erased style reference image and the description text are vectorized to obtain an image embedding vector and a text embedding vector. The preset learnable embedding vector and the image embedding vector are input into the pre-trained attention model to generate a style embedding vector. The style embedding vector and the text embedding vector are input into the pre-trained noise prediction model to generate a target image. In this way, the semantic erasure of the style reference image ensures that the semantics of the style reference image will not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and affecting the generation of the target image.

[0300] In addition, a cross-attention operation is performed on the preset learnable embedding vector and the image embedding vector to learn the style features of the style reference image, thereby generating a style embedding vector. This can secondarily erase the semantics of the style reference image carried in the image embedding vector, so that the semantics of the style reference image does not cover the semantics of the description text, thereby avoiding the problem of semantic leakage and affecting the generation of the target image. The cross-attention operation is performed on the style embedding vector and the text embedding vector, so that the generated target image can not only transfer the color, texture, composition, artistic style, etc. in the style reference image, but also well portray the content of the description text.

[0301] Corresponding to the above method embodiment, the present application embodiment also provides an image generating device, such as Fig.11 As shown, the device may include: an image and text acquisition module 1110 , an image processing module 1120 , and an image generation module 1130 .

[0302] An image and text acquisition module 1110 is used to acquire a style reference image and description text;

[0303] An image processing module 1120 is used to perform semantic erasure processing on the style reference image to obtain the style reference image after semantic erasure;

[0304] The image generation module 1130 is used to generate a target image based on the style reference image after semantic erasure and the description text.

[0305] In an optional implementation, the image processing module is specifically used to:

[0306] Performing cropping processing on the style reference image to obtain an image block in the style reference image;

[0307] Performing an inversion process on the image block to obtain the inverted image block;

[0308] The inverted image block is determined as the style reference image after semantic erasure.

[0309] In an optional implementation, the image processing module specifically includes:

[0310] An image shuffling submodule, used for shuffling the style reference image to obtain an image with shuffled content;

[0311] The image determination submodule is used to determine the image with disrupted content as the style reference image after semantic erasure.

[0312] In an optional implementation, the image processing submodule specifically includes:

[0313] A matrix acquisition unit, used to acquire a first noise matrix and a second noise matrix, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image;

[0314] A matrix forming unit, used for forming a relative coordinate matrix according to the first noise matrix and the second noise matrix;

[0315] A pixel search unit, configured to search, for any relative coordinate in the relative coordinate matrix, a pixel corresponding to the relative coordinate from the style reference image;

[0316] The pixel processing unit is used to place the pixel at a corresponding position of the relative coordinate in the relative coordinate matrix to obtain an image with scrambled content.

[0317] In an optional implementation, the matrix acquisition unit is used to:

[0318] Obtaining a randomly generated first noise matrix and a second noise matrix;

[0319] The first element in the first noise matrix and the second element in the second noise matrix are both within a preset value range.

[0320] In an optional implementation, the matrix composition unit is specifically used for:

[0321] For any first element in the first noise matrix, determining a first position of the first element in the first noise matrix;

[0322] According to the first position, searching the second noise matrix for a second element corresponding to the first element;

[0323] Combine the first element and the second element to obtain a first relative coordinate, and place the first relative coordinate at the first position to obtain a relative coordinate matrix;

[0324] or,

[0325] For any second element in the second noise matrix, determining a second position of the second element in the second noise matrix;

[0326] According to the second position, searching the first noise matrix for a first element corresponding to the second element;

[0327] The first element and the second element are combined to obtain a second relative coordinate, and the second relative coordinate is placed at the second position to obtain a relative coordinate matrix.

[0328] In an optional implementation, the pixel search unit is specifically used to:

[0329] According to the size of the style reference image, the relative coordinates are converted to obtain image coordinates;

[0330] The pixel corresponding to the image coordinate is searched in the style reference image, and the pixel corresponding to the image coordinate is determined as the pixel corresponding to the relative coordinate.

[0331] In an optional implementation, the image generation module specifically includes:

[0332] An image vectorization processing submodule is used to perform vectorization processing on the style reference image after semantic erasure to obtain an image embedding vector;

[0333] A text vectorization processing submodule, used to perform vectorization processing on the description text to obtain a text embedding vector;

[0334] The image generation submodule is used to generate a target image according to the image embedding vector and the text embedding vector.

[0335] In an optional implementation, the image generation submodule is specifically used for:

[0336] Assigning corresponding weights to the image embedding vectors;

[0337] The weighted image embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0338] In an optional implementation, the image generation submodule specifically includes:

[0339] A vector acquisition unit, configured to acquire a preset learnable embedding vector, wherein the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector;

[0340] A vector generating unit, configured to generate a style embedding vector according to the preset learnable embedding vector and the image embedding vector;

[0341] An image generating unit is used to generate a target image according to the style embedding vector and the text embedding vector.

[0342] In an optional implementation, the vector generation unit is specifically used to:

[0343] Inputting the preset learnable embedding vector and the image embedding vector into a pre-trained attention model to obtain a style embedding vector;

[0344] The pre-trained attention model performs a cross-attention operation on the preset learnable embedding vector and the image embedding vector to generate a style embedding vector;

[0345] The image generation unit is specifically used for:

[0346] The style embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

[0347] In an optional embodiment, the device further comprises:

[0348] A sample style reference image and sample description text acquisition module is used to acquire a sample style reference image and a sample description text;

[0349] A sample style reference image processing module, used for performing semantic erasure processing on the sample style reference image to obtain the sample style reference image after semantic erasure;

[0350] An image vectorization processing module, used for performing vectorization processing on the sample style reference image after semantic erasure to obtain a sample image embedding vector;

[0351] A text vectorization processing module, used to perform vectorization processing on the sample description text to obtain a sample text embedding vector;

[0352] A sample style embedding vector generation module, used for inputting the sample image embedding vector and a preset learnable embedding vector into an attention model to obtain a sample style embedding vector;

[0353] An image prediction module, used for inputting the sample style embedding vector and the sample text embedding vector into a noise prediction model to obtain a predicted image;

[0354] A loss value determination module, used to determine a loss value between the predicted image and a style image corresponding to the sample style reference image;

[0355] A model training module is used to train the attention model and the noise prediction model according to the loss value; when the loss value converges, the training is stopped to obtain a pre-trained attention model and a pre-trained noise prediction model.

[0356] In an optional implementation, the sample style reference image processing module is specifically used to:

[0357] Performing cropping processing on the sample style reference image to obtain a sample image block in the sample style reference image;

[0358] Performing an inversion process on the sample image block to obtain the inverted sample image block;

[0359] The inverted sample image block is determined as the sample style reference image after semantic erasure.

[0360] In an optional implementation, the sample style reference image processing module specifically includes:

[0361] A sample style reference image shuffling submodule, used for shuffling the sample style reference image to obtain a sample image with shuffled content;

[0362] The sample style reference image determination submodule is used to determine the sample image with disrupted content as the sample style reference image after semantic erasure.

[0363] In an optional implementation, the sample style reference image shuffling submodule specifically includes:

[0364] A noise matrix acquisition unit, used to acquire a first sample noise matrix and a second sample noise matrix, wherein the size of the first sample noise matrix and the size of the second sample noise matrix are both the same as the size of the sample style reference image;

[0365] A coordinate matrix forming unit, used for forming a sample relative coordinate matrix according to the first sample noise matrix and the second sample noise matrix;

[0366] A sample pixel search unit, used for searching, for any sample relative coordinate in the sample relative coordinate matrix, a sample pixel corresponding to the sample relative coordinate from the sample style reference image;

[0367] The sample pixel processing unit is used to place the sample pixel at the corresponding sample position of the sample relative coordinate in the sample relative coordinate matrix to obtain a sample image with disrupted content.

[0368] In an optional implementation manner, the noise matrix acquisition unit is specifically used to:

[0369] Obtaining a randomly generated first sample noise matrix and a second sample noise matrix;

[0370] The first sample elements in the first sample noise matrix and the second sample elements in the second sample noise matrix are both within a preset value range.

[0371] In an optional implementation, the coordinate matrix composition unit is specifically used for:

[0372] For any first sample element in the first sample noise matrix, determining a first sample position of the first sample element in the first sample noise matrix;

[0373] According to the first sample position, searching the second sample noise matrix for a second sample element corresponding to the first sample element;

[0374] Combine the first sample element and the second sample element to obtain a first sample relative coordinate;

[0375] Put the first sample relative coordinates at the first sample position to obtain a sample relative coordinate matrix;

[0376] or,

[0377] For any second sample element in the second sample noise matrix, determining a second sample position of the second sample element in the second sample noise matrix;

[0378] According to the second sample position, searching the first sample element corresponding to the second sample element from the first sample noise matrix;

[0379] Combine the first sample element and the second sample element to obtain a second sample relative coordinate;

[0380] The second sample relative coordinates are placed on the second sample position to obtain a sample relative coordinate matrix.

[0381] In an optional implementation, the sample pixel search unit is specifically configured to:

[0382] According to the size of the sample style reference image, the sample relative coordinates are converted to obtain sample image coordinates;

[0383] Searching for sample pixels corresponding to the sample image coordinates from the sample style reference image;

[0384] The sample pixel corresponding to the sample image coordinate is determined as the sample pixel corresponding to the sample relative coordinate.

[0385] The present application also provides an electronic device, such as Fig.12 As shown, it includes a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124.

[0386] Memory 123, used for storing computer programs;

[0387] The processor 121 is used to execute the program stored in the memory 123 to implement the following steps:

[0388] A style reference image and a description text are obtained, and a semantic erasure process is performed on the style reference image to obtain the semantically erased style reference image; and a target image is generated based on the semantically erased style reference image and the description text.

[0389] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0390] The communication interface is used for communication between the above electronic device and other devices.

[0391] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0392] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0393] In another embodiment provided in the present application, a storage medium is also provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes any one of the image generation methods in the above embodiments.

[0394] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the image generation methods in the above embodiments.

[0395] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0396] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0397] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0398] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a style reference image and a description text, and perform semantic erasure processing on the style reference image to obtain the style reference image after semantic erasure; A target image is generated based on the semantically erased style reference image and the description text.

2. The method according to claim 1, characterized in that The performing semantic erasure processing on the style reference image to obtain the semantically erased style reference image includes: Performing cropping processing on the style reference image to obtain an image block in the style reference image; Performing an inversion process on the image block to obtain the inverted image block; The inverted image block is determined as the style reference image after semantic erasure.

3. The method according to claim 1, characterized in that The performing semantic erasure processing on the style reference image to obtain the semantically erased style reference image includes: Shuffling the style reference image to obtain an image with shuffled content; The image with disrupted contents is determined as the style reference image after semantic erasure.

4. The method according to claim 3, characterized in that: The step of shuffling the style reference image to obtain an image with shuffled content includes: Acquire a first noise matrix and a second noise matrix, wherein the size of the first noise matrix and the size of the second noise matrix are both the same as the size of the style reference image; According to the first noise matrix and the second noise matrix, a relative coordinate matrix is ​​formed; For any relative coordinate in the relative coordinate matrix, searching the style reference image for a pixel corresponding to the relative coordinate; The pixels are placed at corresponding positions of the relative coordinates in the relative coordinate matrix to obtain an image with scrambled contents.

5. The method according to claim 4, characterized in that The obtaining of the first noise matrix and the second noise matrix comprises: Obtaining a randomly generated first noise matrix and a second noise matrix; The first element in the first noise matrix and the second element in the second noise matrix are both within a preset value range.

6. The method according to claim 4, characterized in that The forming of a relative coordinate matrix according to the first noise matrix and the second noise matrix comprises: For any first element in the first noise matrix, determining a first position of the first element in the first noise matrix; According to the first position, searching the second noise matrix for a second element corresponding to the first element; Combine the first element and the second element to obtain a first relative coordinate, and place the first relative coordinate at the first position to obtain a relative coordinate matrix; or, For any second element in the second noise matrix, determining a second position of the second element in the second noise matrix; According to the second position, searching the first noise matrix for a first element corresponding to the second element; The first element and the second element are combined to obtain a second relative coordinate, and the second relative coordinate is placed at the second position to obtain a relative coordinate matrix.

7. The method according to claim 4, characterized in that The step of searching the style reference image for pixels corresponding to the relative coordinates includes: According to the size of the style reference image, the relative coordinates are converted to obtain image coordinates; The pixel corresponding to the image coordinate is searched in the style reference image, and the pixel corresponding to the image coordinate is determined as the pixel corresponding to the relative coordinate.

8. The method according to claim 1, characterized in that The generating a target image based on the semantically erased style reference image and the description text includes: Vectorizing the style reference image after semantic erasure to obtain an image embedding vector; The description text is vectorized to obtain a text embedding vector, and a target image is generated according to the image embedding vector and the text embedding vector.

9. The method according to claim 8, characterized in that The step of generating a target image according to the image embedding vector and the text embedding vector comprises: Assigning corresponding weights to the image embedding vector; The weighted image embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

10. The method according to claim 8, characterized in that The step of generating a target image according to the image embedding vector and the text embedding vector comprises: Obtaining a preset learnable embedding vector, wherein the dimension of the preset learnable embedding vector is the same as the dimension of the image embedding vector; Generate a style embedding vector according to the preset learnable embedding vector and the image embedding vector; A target image is generated according to the style embedding vector and the text embedding vector.

11. The method according to claim 10, characterized in that The generating a style embedding vector according to the preset learnable embedding vector and the image embedding vector includes: Inputting the preset learnable embedding vector and the image embedding vector into a pre-trained attention model to obtain a style embedding vector; The pre-trained attention model performs a cross-attention operation on the preset learnable embedding vector and the image embedding vector to generate a style embedding vector; The step of generating a target image according to the style embedding vector and the text embedding vector comprises: The style embedding vector and the text embedding vector are input into a pre-trained noise prediction model to obtain a target image.

12. The method according to claim 11, characterized in that Before executing the step of acquiring the style reference image and the description text, the method further includes: Acquire a sample style reference image and a sample description text, and perform semantic erasure processing on the sample style reference image to obtain the sample style reference image after semantic erasure; Vectorizing the sample style reference image after semantic erasure to obtain a sample image embedding vector; Vectorizing the sample description text to obtain a sample text embedding vector; Inputting the sample image embedding vector and the preset learnable embedding vector into the attention model to obtain a sample style embedding vector; Inputting the sample style embedding vector and the sample text embedding vector into a noise prediction model to obtain a predicted image; Determining a loss value between the predicted image and a style image corresponding to the sample style reference image; Training the attention model and the noise prediction model according to the loss value; When the loss value converges, the training is stopped to obtain the pre-trained attention model and the pre-trained noise prediction model.

13. An image generating device, characterized in that: The device comprises: Image and text acquisition module, used to obtain style reference images and description text; An image processing module, used for performing semantic erasure processing on the style reference image to obtain the style reference image after semantic erasure; The image generation module is used to generate a target image based on the style reference image after semantic erasure and the description text.

14. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images