Image generation method, apparatus, device, and storage medium

Abstract

The application discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of image generation. The method comprises the following steps: determining intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data comprises reference vector data and target vector data; generating a first target image based on the reference vector data; and generating a second target image by fusing the reference vector data and the target vector data, wherein the first target image and the second target image constitute a target image set corresponding to a style prompt word in the text prompt word set. In the application, the data of the first target image used as a reference in the generation process is shared to the generation process of the second target image, thereby realizing the generation of an image set that is consistent in style, diversified and high in quality, effectively meeting the image generation requirement of consistent style, and improving the image generation effect and efficiency of the diffusion model.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to an image generation method, apparatus, device and storage medium. Background Art

[0002] In recent years, diffusion models have revolutionized fields such as art, design, and game development due to their ability to generate high-quality and diverse images. Early diffusion models such as SD1.5 and SDXL used a U-Net architecture to generate images based on a prompt word. More recently, the more advanced diffusion model, Flux.1, replaces the self-attention and cross-attention used in previous models with a multimodal attention mechanism, achieving higher generated image quality and better alignment of text prompts.

[0003] A key task of diffusion models is to generate images with consistent style. Specifically, this task aims to generate images with diverse content but consistent style through unified style cues. However, style cues make it difficult to ensure style consistency between generated images. For example, an attention-based method, StyleAligned, proposes to share the key and value of the self-attention mechanism in the diffusion model to enhance style consistency between generated images. However, the design of StyleAligned is based on the self-attention mechanism used in earlier diffusion models such as SDXL. Directly applying StyleAligned to the diffusion model Flux.1 will lead to severe content leakage between the generated image sets, and all images have extremely similar content. While some methods used for other tasks can be used for the task of generating consistent images, these methods cannot achieve satisfactory results. For example, style image generation methods need to first provide a style reference image, and then generate a style-consistent image based on the reference image. Among them, style image generation methods can be divided into two categories: adapter-based methods and fine-tuning-based methods. Both methods require training and have inherent limitations: for adapter-based methods, due to the undefinability of "style", it is difficult to create high-quality training datasets for training, which leads to poor performance of adapter-based methods. For fine-tuning-based methods, collecting new data for each style and retraining significantly increases the computational and time costs; in addition, some image editing methods have also been explored to solve this task. However, since they are mainly designed for general image editing tasks, it is difficult to complete large-scale editing of image content while keeping the style unchanged. Summary of the Invention

[0004] The main purpose of this application is to provide an image generation method, apparatus, device and storage medium, aiming to solve the technical problem of how to enhance the style consistency and efficiency of images generated by diffusion models.

[0005] To achieve the above objectives, the present application proposes an image generation method, which includes:

[0006] Determining intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data;

[0007] generating a first target image based on the reference vector data;

[0008] The reference vector data and the target vector data are fused to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt words in the text prompt word set.

[0009] In one embodiment, the step of fusing the reference vector data and the target vector data to generate a second target image includes:

[0010] cropping the reference vector data to obtain a reference image feature vector in the reference vector data;

[0011] Cutting the target vector data to obtain a target image feature vector and a target text feature vector;

[0012] performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector;

[0013] splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a spliced ​​feature vector;

[0014] A second target image is generated based on the spliced ​​feature vectors.

[0015] In one embodiment, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector;

[0016] The step of performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector comprises:

[0017] calculating an adjusted target image query vector based on the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector;

[0018] An adjusted target image key vector is calculated according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

[0019] In one embodiment, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector; the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; the target image feature vector includes a target image key vector and a target image value vector; the reference image feature vector includes a reference image key vector and a reference image value vector; and the spliced ​​feature vector includes a spliced ​​query vector, a spliced ​​key vector, and a spliced ​​value vector;

[0020] The step of splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a spliced ​​feature vector includes:

[0021] splicing the target text query vector and the adjusted target image query vector to obtain a spliced ​​query vector;

[0022] splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector;

[0023] The target text value vector, the target image value vector and the reference image value vector are spliced ​​to obtain a spliced ​​value vector.

[0024] In one embodiment, the step of splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector includes:

[0025] Adjusting the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector;

[0026] The target text key vector, the adjusted target image key vector and the adjusted reference image key vector are spliced ​​to obtain a spliced ​​key vector.

[0027] In one embodiment, before the step of splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector, the step further includes:

[0028] The position index of any basic unit in the reference image key vector is offset to obtain a shifted reference image key vector, and the step of splicing the target text key vector, the adjusted target image key vector and the reference image key vector is performed based on the shifted reference image key vector, wherein the position index of the shifted reference image key vector does not overlap with the position index of the adjusted target image query vector.

[0029] In one embodiment, the image generation method is applied to a preset diffusion model, wherein the preset diffusion model includes a dual-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is performed in the single-stream module.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes an image generating device, which includes:

[0031] a vector determination module, configured to determine intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data;

[0032] A first generating module, configured to generate a first target image based on the reference vector data;

[0033] The second generating module is configured to fuse the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt word in the text prompt word set.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image generation method described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the image generation method described above are implemented.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] The present application first determines intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set. The intermediate vector data includes reference vector data and target vector data, thereby providing basic feature data for the subsequent generation of images with a specific style. This allows the generated images to be constructed based on the style-related information contained in the text prompt words, laying the foundation for the subsequent generation of a set of images with a consistent style. A first target image is then generated based on the reference vector data, providing a stylistic reference for generating the target image. The reference vector data and the target vector data are then fused to generate a second target image. The first and second target images form a target image set corresponding to the style prompt words in the text prompt word set. This achieves the goal of generating a second target image that matches the first target image used as a reference based on the features and information contained in the target vector data, while ensuring style consistency between the image and the reference image, while also reflecting the content characteristics contained in the target vector data. As a result, during the image set generation process, each image exhibits diverse content while maintaining a unified style, meeting user requirements for generating a set of images with a specific style. This effectively addresses the issue of style consistency, ensuring that the generated image set is stylistically consistent and consistent.

[0038] In summary, the present application first determines the self-attention vector data containing reference vector data and target vector data based on the text prompt word set and the noise image set, and then generates the first target image based on the reference vector data respectively, and simultaneously fuses the reference and target vector data to generate the second target image, that is, shares the data of the first target image in the generation process to the generation process of the second target image, and finally combines them into a target style generated image set corresponding to the style prompt word, thereby avoiding the content leakage and style inconsistency problems caused by directly applying the early methods to the diffusion model, as well as the technical problems of difficulty in creating training data, high computational time cost, and difficulty in significantly editing content while keeping the style unchanged in the style image generation method based on adapters or fine-tuning, thereby achieving the generation of a consistent, diverse, and high-quality image set, effectively meeting the needs of style-consistent image generation, and improving the image generation effect and efficiency of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0041] Figure 1 A schematic diagram of the process flow provided for the first embodiment of the image generation method of the present application;

[0042] Figure 2 A schematic diagram of a cropping scene of the image generation method provided in Example 1 of the present application;

[0043] Figure 3 A schematic diagram of a location index scenario of the image generation method provided in Example 2 of the present application;

[0044] Figure 4 A schematic diagram of another location index scenario of the image generation method provided in Example 2 of the present application;

[0045] Figure 5 A schematic diagram of a brief flow chart of the image generation method provided in Example 2 of the present application;

[0046] Figure 6 A schematic diagram of the model architecture of the image generation method provided in Example 3 of the present application;

[0047] Figure 7 A schematic diagram of the image generation effect of the image generation method provided in Example 3 of the present application;

[0048] Figure 8 This is a schematic diagram of the module structure of the image generating device according to an embodiment of the present application;

[0049] Figure 9 Schematic diagram of the device structure of the hardware operating environment involved in the image generation method in the embodiment of the present application. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: determining intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; generating a first target image based on the reference vector data; and fusing the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt words in the text prompt word set.

[0053] However, style cues make it difficult to ensure style consistency across generated images. For example, sharing the key and value in the self-attention mechanism in the diffusion model can enhance style consistency across generated images. However, this design is based on the self-attention mechanism used in earlier diffusion models such as SDXL. Directly applying StyleAligned to the diffusion model Flux.1 results in severe content leakage across the generated image set, with all images having extremely similar content. While some methods developed for other tasks can be adapted for style-consistent image generation, these methods do not achieve satisfactory results. For example, style image generation methods require providing a style reference image, from which they generate a style-consistent image. These methods can be categorized into two types: adapter-based methods and fine-tuning-based methods. Both require training and have inherent limitations, including the difficulty of creating high-quality training datasets or excessive computational and time costs. Furthermore, some image editing methods have been explored for this task. However, since they are primarily designed for general-purpose image editing, they struggle to significantly edit image content while preserving the style.

[0054] The present application provides a solution, which first determines self-attention vector data containing reference vector data and target vector data based on a text prompt word set and a noise image set, then generates a first target image based on the reference vector data respectively, and simultaneously fuses the reference and target vector data to generate a second target image, that is, shares the data of the first target image in the generation process to the generation process of the second target image, and finally combines them into a target style generated image set corresponding to the style prompt word. It avoids the content leakage and style inconsistency problems caused by directly applying early methods to the diffusion model, as well as technical problems such as difficulty in creating training data, high computational time cost, and difficulty in significantly editing content while keeping the style unchanged in adapter-based or fine-tuning style image generation methods. It achieves the generation of a consistent, diverse, and high-quality image set, effectively meets the needs of style-consistent image generation, and improves the image generation effect and efficiency of the diffusion model.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of performing the above functions. The following uses electronic devices as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides an image generation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the image generation method of the present application.

[0057] In this embodiment, the image generation method comprises steps S10-S30:

[0058] In step S10, intermediate vector data for generating a target image set is determined according to a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data comprises reference vector data and target vector data.

[0059] It should be noted that the text prompt word set refers to a set of text information containing image content description and style requirements, etc., which is used to guide the direction and style of image generation; the noise image set refers to a series of image data sets with random noise characteristics, which are used as initial input of the generation model and gradually guide the generation of images meeting the requirements in the diffusion model; the intermediate vector data refers to the intermediate data of the model generating images, that is, the vector data containing query vector, key vector, value vector, etc. related information obtained by processing the input text prompt word and noise image in the self-attention mechanism of the model, which is used to capture the association and importance between text and image features, etc. The reference vector data refers to the vector data extracted and determined from the text prompt word set and the noise image set, which is used to generate images with reference style, containing reference style related feature information; the target vector data refers to the vector data extracted and determined from the text prompt word set and the noise image set, which is used to generate images with target content and target style, containing target image content and style related feature information.

[0060] In addition, it should be noted that the image generation method proposed in this application can be applied to the Flux.1 diffusion model, aiming to encourage the style of the second target image generated based on the target vector data to align with the style of the first target image generated based on the reference vector data.

[0061] It can be understood that, since the existing image generation scheme cannot directly utilize the text prompt word and noise image to effectively generate feature vector data that can be used for subsequent style consistent image generation, step S10 is performed to avoid the problem of inconsistent image style or poor quality caused by unreasonable feature vector data construction, thereby providing accurate and effective initial feature vector data for subsequent style consistent image generation based on text prompt words.

[0062] For example, a set of text prompt words containing different contents is first formed around a style prompt word. Each prompt word in the set is in the form of: <content prompt word> in <style prompt word>, such as "dog, 3D real style", "table, 3D real style", "earth, 3D real style", etc. Then, each text prompt word in the text prompt word set is encoded into a text embedding vector using a text encoder. At the same time, each noise image in the noise image set is encoded using an image encoder to obtain an image embedding vector. Then, the text embedding vector and the image embedding vector are sent as input to a pre-trained self-attention network to calculate the intermediate vector data used in the self-attention mechanism. In this process, the intermediate vector data is divided into reference vector data and target vector data through the pre-set network structure and parameter settings. The reference vector data can be determined based on a text prompt word and a corresponding noise image, and the target vector data can be determined based on the remaining text prompt words and the corresponding noise images.

[0063] Step S20, generating a first target image based on the reference vector data;

[0064] It should be noted that the first target image refers to an example image with a specific target style generated based on the reference vector data, which is used to provide a style reference and benchmark for subsequently generated target images to ensure the style consistency of the entire image set.

[0065] For example, a specialized image generation network, such as a generator based on a generative adversarial network (GAN) or a decoder based on a variational autoencoder (VAE), can be used to take the reference vector data as input. Through multi-layer neural network processing, the reference vector data is gradually mapped into the image space. During this process, operations such as upsampling and convolution can be used to increase the spatial resolution and detail of the image, ultimately generating a reference image that reflects the target style, i.e., the first target image.

[0066] Step S30, fusing the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt word in the text prompt word set;

[0067] It should be noted that the second target image refers to an image with target style and target content generated based on the joint action of reference vector data and target vector data, and is the main component of the final target style generated image set; the target image set refers to an image set formed by combining reference images and target images, where all images have a target style corresponding to the style prompt words in the text prompt word set, meeting the user's demand for generating an image set with consistent style.

[0068] It is understandable that since it is often difficult to generate target images that meet the style requirements and have diverse content in existing image generation schemes, performing step S30 can avoid problems such as the style of the generated target image being inconsistent with the reference image, the content being single or not meeting expectations, and thus generate a series of target images that are consistent with the reference image in style and rich in content.

[0069] Exemplarily, the reference vector data and the target vector data are subjected to feature splicing or feature fusion operations to form a comprehensive feature vector, which is then input into another image generation network. Similar to the generation process of the reference image, the fused feature vector is mapped to the image space through the processing of a multi-layer neural network to generate a target image, i.e., a second target image, that conforms to the target style and contains the target content.

[0070] In a feasible implementation, step S30 may include steps S31 to S35:

[0071] Step S31, cropping the reference vector data to obtain a reference image feature vector in the reference vector data;

[0072] It should be noted that the reference image feature vector refers to the relevant vector information that is cut from the reference vector data and can reflect the style characteristics of the reference image, including the reference image query vector obtained by image feature mapping. , reference image key vector and the reference image value vector These vectors together carry the style features of the reference image and are used to guide the style generation of the target image.

[0073] In addition, it should be noted that the reference vector data also includes a reference text feature vector, which includes a reference text query vector obtained by text feature mapping. , reference text key vector and a vector of reference text values .

[0074] It can be understood that since the reference vector data contains a variety of information, and the part related to the image features is the key content for providing a style reference when generating the target image, step S31 is performed. Through the cropping operation, irrelevant or redundant information can be avoided from being brought into the subsequent image generation process, thereby achieving accurate extraction of the feature vector of the reference image and providing an accurate style reference basis for the subsequent target image generation.

[0075] For example, please refer to Figure 2 , the reference vector data can be queried key Sum (shown in the figure as 、 、 ) is considered as a high-dimensional vector or tensor, which contains various feature information related to the image. By performing a cropping operation on this vector, such as splitting it according to a predefined index range or feature dimension, the part specifically used to describe the image features is extracted, and the reference image feature vector is obtained. 、 and (shown in the figure as 、 and ), thereby accurately locating and obtaining feature vectors closely related to image style and other features from the reference vector data containing rich information, providing key reference information for the subsequent generation of target images with consistent style.

[0076] Step S32, cutting the target vector data to obtain a target image feature vector and a target text feature vector;

[0077] It should be noted that the target image feature vector refers to the feature vector part related to the target image content cut out from the target vector data, including the target image query vector, target image key vector and target image value vector, etc., which is used to reflect the content characteristics that the target image should have; the target text feature vector refers to the feature vector part related to the text prompt cut out from the target vector data, including the target text query vector, target text key vector and target text value vector, etc., which is used to reflect the content characteristics required by the text prompt.

[0078] It can be understood that since the target vector data contains both the content feature information of the target image and the feature information related to the text prompt, they need to be distinguished for use in subsequent processing. Therefore, step S32 is performed. The cropping operation can avoid confusing the image content features and text features when generating the target image, and the target image feature vector is used to reflect the image content, and the target text feature vector is used to reflect the content required by the text prompt, thereby ensuring that the image content and text requirements can be accurately integrated when generating the target image.

[0079] For example, please refer to Figure 2 , target vector data 、 and (shown in the figure as 、 、 ) is also a comprehensive high-dimensional vector, which integrates the content features of the target image and the relevant features of the text prompt. By designing a cutting strategy, such as segmentation according to different fields or feature segments in the data structure, the target vector data can be split into two parts: one part is a feature vector that focuses on describing the content of the target image, that is, the target image feature vector 、 、 (illustrated as 、 and in the figure); the other part is a feature vector corresponding to the content of the text prompt, that is, the target text feature vector and and (illustrated as 、 and in the figure). Thus, the image content features and text features are processed separately, so that the content of the target image and the text-related characteristics can be accurately controlled in the subsequent generation process.

[0080] Step S33, adaptively normalizing the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector;

[0081] It can be understood that, since the target image feature vector and the reference image feature vector may have differences in distribution, direct use may result in the target image being unable to accurately align with the reference image style in style, so step S33 is performed to avoid this inconsistency in style by adaptive normalization processing, so that the distribution of the target image feature vector is adjusted to be adapted to the distribution of the reference image feature vector, so that the generated target image can better maintain consistency with the reference image in style.

[0082] Exemplarily, the mean and standard deviation of the target image feature vector are calculated, and the mean and standard deviation of the reference image feature vector are combined to adjust the target image feature vector. Specifically, a method similar to instance normalization can be used to first normalize the target image feature vector to eliminate its own statistical characteristics, and then scale it to the same mean and standard deviation range as the reference image feature vector. After such processing, the adjusted target image feature vector obtained can match the reference image feature vector in distribution, so that when the target image is generated, the target image can maintain consistency with the reference image in style while retaining the content features of the target image itself.

[0083] Step S34, splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector and the reference image feature vector to obtain a spliced feature vector;

[0084] It can be understood that since it is necessary to comprehensively consider information from multiple aspects such as the text prompt content, the target image content features, and the reference image style features in order to generate a target image that meets the text requirements and has the target content and reference style, step S34 is performed. Through the splicing operation, the omission or isolated processing of information can be avoided, and multiple feature vectors can be integrated into a complete feature representation, so that when generating the target image, the text prompt content, the content features of the target image itself, and the style features of the reference image can be referenced at the same time, providing comprehensive and rich information input for the subsequent target image generation, thereby generating a target image that meets the requirements.

[0085] For example, the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector are concatenated in a certain order to form a higher-dimensional comprehensive feature vector. For example, concatenation can be performed along the channel dimension or along the depth direction of the feature map. This concatenated feature vector integrates multiple aspects of information, including text prompt information, target image content features, and reference image style features, providing a comprehensive and rich feature foundation for subsequent target image generation.

[0086] Step S35 : generating a second target image based on the spliced ​​feature vectors.

[0087] It can be understood that since the spliced ​​feature vector has integrated key information such as text prompts, target image content and reference image style, it has sufficient conditions to generate a target image that meets the requirements. Therefore, step S35 is performed. By generating an image based on the spliced ​​feature vector, the problem of the generated image not meeting the requirements in terms of content, style, etc. can be avoided, and a target image that meets the text prompt content, has the target image content characteristics and is consistent with the reference image style can be generated.

[0088] For example, the concatenated feature vectors are subjected to a dot product self-attention calculation, and the result is used as input to a pre-trained generative network, such as a generator based on a convolutional neural network. The generative network gradually converts the high-dimensional feature vectors into image data with spatial structure through a series of operations such as upsampling, convolution, and activation functions. In this process, the network will generate a target image that meets the requirements of the text prompt content, has the content characteristics of the target image, and is consistent with the style of the reference image based on the information such as the text prompt, target content, and reference style contained in the concatenated feature vectors.

[0089] This implementation, through the sequential steps of feature vector cropping and separation, adaptive normalization, and feature splicing, avoids issues such as inconsistent generated image style and conflicts between content and style, caused by mismatches between target image features and reference style features, or poor integration of text prompts and image content features. This results in high-quality images that meet both content and style requirements. Furthermore, the cropping operation accurately extracts the required feature information, improving processing efficiency and targeting. Adaptive normalization ensures that the target image features are distributed consistently with the reference style features, guaranteeing accurate style transfer and unification. Feature splicing integrates multi-source feature information, providing comprehensive guidance for image generation and ensuring that the generated image not only conforms to the text prompt content but also possesses both the target image content and the reference image style characteristics.

[0090] This embodiment provides an image generation method, which first determines self-attention vector data including reference vector data and target vector data based on a text prompt word set and a noise image set, then generates a first target image based on the reference vector data respectively, and simultaneously fuses the reference and target vector data to generate a second target image, that is, shares the data of the first target image during the generation process to the second target image during the generation process, and finally combines them into a target style generated image set corresponding to the style prompt word. This avoids the content leakage and style inconsistency problems caused by directly applying early methods to the diffusion model, as well as technical problems such as difficulty in creating training data, high computational time cost, and difficulty in significantly editing content while keeping the style unchanged in adapter-based or fine-tuning style image generation methods. It achieves the generation of a consistent, diverse, and high-quality image set with a consistent style, effectively meets the requirements for generating images with a consistent style, and improves the image generation effect and efficiency of the diffusion model.

[0091] In a feasible implementation, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector;

[0092] Step S33 may include steps S331-S332:

[0093] Step S331, calculating an adjusted target image query vector according to the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector;

[0094] It should be noted that the target image query vector is the query vector representation of the target image in the self-attention mechanism. It represents the desired feature direction for retrieval and matching of the target image and can be used to guide feature selection during the generation process, ensuring that the generated image contains the required content features and, after subsequent adjustments, can also adapt to the style characteristics of the reference image. The reference image query vector is the query vector representation of the reference image in the self-attention mechanism and is used to interact with other vectors (such as key vectors) to determine the degree of correlation between other vectors and itself. For the reference image, the query vector contains key information used to retrieve and match features of the reference image during the generation process, reflecting relevant information such as the reference image's style characteristics.

[0095] It is understandable that since the distribution of the target image query vector is often inconsistent with the distribution of the reference image query vector, step S331 is performed. By adjusting based on the mean and standard deviation, the style mismatch problem caused by the inconsistent distribution of the target image query vector and the reference image query vector can be avoided, and the distribution of the target image query vector is adjusted to be close to the reference image query vector. Therefore, when generating the target image, the style features of the reference image can be better shared, so that the target image is consistent with the reference image in style.

[0096] For example, the target image query vector is transformed into Query the reference image for the vector Align and obtain the adjusted target image query vector :

[0097]

[0098] Among them, the adaptive normalization operation is defined as:

[0099]

[0100] in, represents the mean of the variable, Represents the standard deviation of the variable.

[0101] Step S332: Calculate the adjusted target image key vector based on the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

[0102] It should be noted that the target image key vector is the key vector representation of the target image in the self-attention mechanism, which reflects a dimension of the target image's own characteristics. It interacts with the query vector in the self-attention mechanism, provides a basis for determining the correlation between different parts of the target image and the correlation with the reference image features, and is one of the key factors for achieving target image style adjustment and content generation; the reference image key vector is the key vector representation of the reference image in the self-attention mechanism, which is used to perform calculations with the query vector to obtain the correlation weight between the two. When interacting with other vectors (such as the query vector), it can reflect the characteristics of the reference image in a specific feature dimension.

[0103] It can be understood that since the distribution of the target image key vector is often different from the distribution of the reference image key vector, step S332 is performed to calculate the adjusted target image key vector so that the target image key vector matches the reference image key vector in distribution. This can avoid the style difference caused by the different distribution of the target image key vector and the reference image key vector, thereby ensuring that the generated target image can accurately reflect the style of the reference image and enhancing the consistency of the style of the image generated by the model.

[0104] For example, the target image key vector is transformed into Key vector to the reference image Align and get the adjusted target image key vector :

[0105]

[0106] In this embodiment, a statistically-based computational adjustment is performed to avoid the style mismatch problem caused by the inconsistency between the distribution of the target image feature vector and the reference image, and to adjust the distribution of the query and key vectors of the target image to match the reference image, so that the style features of the reference image can be better shared when generating the target image, ensuring the style consistency of the image generation.

[0107] In a feasible implementation, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector; the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; the target image feature vector includes a target image key vector and a target image value vector; the reference image feature vector includes a reference image key vector and a reference image value vector; and the spliced ​​feature vector includes a spliced ​​query vector, a spliced ​​key vector, and a spliced ​​value vector;

[0108] Step S34 may include steps S341 to S343:

[0109] Step S341, concatenating the target text query vector and the adjusted target image query vector to obtain a concatenated query vector;

[0110] It is understandable that since the text prompts and image content need to be fused at the query vector level to ensure that the generated image meets the text requirements and reflects the image features, step S341 is performed. The splicing operation can avoid the problem of inconsistent features or insufficient fusion caused by the separation of text and image query vectors, and integrate the query vectors of text and image into a unified query vector, providing a more comprehensive and accurate query basis for subsequent feature matching and attention calculation, so that the generated image can better take into account the text prompts and image content features.

[0111] For example, the target text query vector and the adjusted target image query vector Perform splicing to obtain the spliced ​​query vector :

[0112]

[0113] Step S342, splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector;

[0114] It is understandable that since it is necessary to comprehensively consider the information of text prompts, target image content and reference image style at the key vector level in order to more comprehensively calculate the correlation between features, step S342 is performed to avoid the problem of feature correlation calculation deviation or incompleteness caused by relying solely on a single key vector. The integration of multi-source key vector information is achieved through splicing, which enhances the comprehensiveness and accuracy of feature matching and makes the generated image more in line with the requirements in terms of content, style, etc.

[0115] For example, the target text key vector , the adjusted target image key vector and the reference image key vector Perform splicing to obtain the spliced ​​key vector :

[0116]

[0117] Step S343 : splicing the target text value vector, the target image value vector, and the reference image value vector to obtain a spliced ​​value vector.

[0118] It should be noted that the target text value vector is the representation of the text prompt word after encoding in the value vector part, which contains detailed information of the text features, and is used to embody the specific content and details described by the text in the image generation; the reference image value vector is the feature representation of the reference image at the value vector level, which contains detailed feature information of the reference image, and is used to provide style reference information in the image generation process to ensure that the generated image has consistent style features with the reference image.

[0119] It can be understood that, since the detailed information of the text prompt content, the target image features and the reference image style needs to be integrated to provide rich information support for the final image generation, step S343 is performed to avoid problems such as insufficient details, inaccurate style or content of the generated image caused by missing information or single value vector, and to realize the fusion of multi-aspect information through splicing to ensure that the generated image has rich details, accurate style and content meeting the requirements.

[0120] Exemplarily, the target text value vector , the target image value vector and the reference image value vector are spliced to obtain the spliced value vector .

[0121]

[0122] In the embodiment, by splicing the feature vectors, the problems such as incomplete content and inconsistent style of the generated image caused by the inability of a single feature vector to fuse multi-source information are avoided, and the technical effects of effectively integrating the text prompt, the target image content features and the reference image style features in the query, key and value dimensions are realized, so as to ensure that the generated target image can accurately embody the content of the text prompt, be consistent with the style of the reference image, and contain rich details.

[0123] In a feasible embodiment, step S342 can include steps S301-S302:

[0124] Step S301, adjusting the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector;

[0125] It should be noted that the preset intensity control parameter refers to a parameter used to adjust the influence degree of the reference image key vector on the spliced key vector in the image generation process. By setting this parameter, the attention weight between the spliced query vector and the reference image key vector can be adjusted, so as to realize fine adjustment of the style intensity of the generated image. For example, when it is desired to make the style of the generated image closer to the reference image, the value of the preset intensity control parameter can be increased; on the contrary, if the style features are weak, the value of the parameter can be reduced.

[0126] It is understandable that, since it is often necessary to control the degree of influence of the reference image style features on the target image generation when fusing the reference image key vector with other key vectors, performing step S301 can avoid the reference image style being too strong or insufficient, resulting in the generated image style not meeting expectations. By introducing a preset intensity control parameter to adjust the reference image key vector, it is possible to avoid the problem of the reference image key vector having an unreasonable proportion in the splicing process, causing the style to be too dominant or not prominent enough. Accurate control of the degree of influence of the reference image style features is achieved, so that the style of the generated image can be reflected according to the expected intensity, enhancing the flexibility and accuracy of style control.

[0127] Step S302 : splicing the target text key vector, the adjusted target image key vector, and the adjusted reference image key vector to obtain a spliced ​​key vector.

[0128] It is understandable that, because the textual hint, the target image content, and the reference image style must be comprehensively considered at the key vector level to guide the generation of the target image, step S302 can avoid incomplete or insufficiently integrated information regarding the content and style of the generated image. By combining these three key vectors, effective integration of multi-source information is achieved, providing a richer and more comprehensive feature representation for the subsequent generated image, ensuring that the generated image better meets requirements in terms of content and style.

[0129] For example, the target text key vector , the adjusted target image key vector and the adjusted reference image key vector Perform splicing to obtain the spliced ​​key vector :

[0130]

[0131] in, It is the preset intensity control parameter.

[0132] In this embodiment, by intensity-controlling and adjusting the reference image key vector and splicing it with the target text key vector and the adjusted target image key vector, the problem of unnatural style fusion and incoordination between content and style caused by the reference image key vector having too strong or too weak an influence on the target image generation during splicing is avoided. This achieves precise control of the degree of influence of the reference image style features and effective integration of multi-source features, thereby ensuring that the generated target image is consistent with the reference image in style and that the content meets the text requirements.

[0133] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can refer to the above introduction, and the subsequent will not be described. On this basis, before step S34, the image generation method further comprises step S01:

[0134] Step S01, offset the position index of any basic unit in the reference image key vector, obtain the offset reference image key vector, and perform the step of splicing the target text key vector, the adjusted target image key vector and the reference image key vector based on the offset reference image key vector, wherein the position index of the offset reference image key vector and the position index of the adjusted target image query vector do not overlap.

[0135] It should be noted that in the vector data structure, each basic unit refers to an independent element or dimension in the vector, which represents a specific aspect or attribute of image or text features. In the context of key vector, each basic unit may correspond to a specific dimension of image or text features. The position index is used to identify the position of each basic unit in the vector. In the process of vector splicing or processing, the position index helps the model to identify and distinguish the source and purpose of different feature vectors.

[0136] It can be understood that in the image generation process, when splicing the target text key vector, the adjusted target image key vector and the reference image key vector, it is necessary to ensure that the position indexes of these vectors do not overlap, so as to avoid causing the attention weight between the reference image key vector and the adjusted target image query vector to be too high, and thus causing the generated target image and the reference image to be too similar. Therefore, step S01 is performed to offset the position index of each basic unit in the reference image key vector, so as to ensure that the position index of the offset reference image key vector and the adjusted target image query vector do not overlap. This offset operation can effectively control the attention weight of the reference image key vector and the adjusted target image query vector within a reasonable range, effectively enhancing the alignment degree of the generated image and the text prompt word.

[0137] Exemplarily, the diffusion model Flux.1 often uses rotational position encoding to introduce position information for attention calculation. Specifically, for the image feature mapping obtained by and , a position index is set for each basic unit token in it, where , and , respectively represent the height and width of the latent variable. For the reference image key vector , the default position index is also the design of . For reference, please refer to Figure 3, but such a design will result in the adjusted target image query vector and The attention weight between them is too high, which leads to the extreme similarity between the generated image and the reference image. Therefore, the offset position index strategy is proposed. For details, please refer to Figure 4 ,Will The default position index is not set to , and set to , This design allows and The position indexes no longer overlap.

[0138] In this embodiment, by proposing an offset position index strategy, the problem of the generated target image being too similar to the reference image due to the overlap of position indexes between vectors during splicing is avoided, and each feature vector is ensured to have a unique position identifier in the overall vector after splicing, so that feature information from different sources can be correctly distinguished and utilized for image generation, and the attention weights of the reference image key vector and the adjusted target image query vector can be effectively controlled within a reasonable range, thereby effectively enhancing the alignment of the generated image and the text prompt word.

[0139] For example, to help understand the implementation process of the image generation method obtained by combining this embodiment with the above embodiment 1, please refer to Figure 5 , Figure 5 A brief flowchart of an image generation method is provided, specifically:

[0140] Share the key and value of the reference image during the generation process to the target image generation process. Specifically, for the reference image Query in Multimodal Attention key Sum Perform cropping, cropping is obtained by image feature mapping 、 and , and obtained from text feature mapping 、 and . Similarly, the target image will also be 、 and Crop to 、 、 and 、 、 .

[0141] Then the reference image 、 and Through a series of operations to share to the target image generation process. Specifically, first through the adaptive normalization operation to 、 、 Towards 、 、 To align:

[0142]

[0143]

[0144] After that, concatenate the length sequences to get the final query, key, and value:

[0145]

[0146]

[0147]

[0148] Afterwards, 、 、 The final result can be obtained by performing the self-attention calculation of the dot product, and the target image can be generated based on the result. At the same time, the offset position index strategy needs to be introduced in this process. The default position index is not set to , and set to , which can allow and The position indexes no longer overlap.

[0149] Based on the first and / or second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can be referred to the above introduction and will not be repeated later.

[0150] On this basis, please refer to Figure 6 The image generation method is applied to a preset diffusion model, wherein the preset diffusion model includes a dual-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is performed in the single-stream module. It can be understood that Figure 6 The style consistent attention unit in the embodiment executes the fusion process of the reference vector data and the target vector data corresponding to step S30 of this application.

[0151] It should be noted that the offset of the position index of any basic unit in the reference image key vector can also be performed in the single-stream module.

[0152] It is understandable that the single-stream module integrates the reference vector data and the target vector data into one processing flow, which can promote deep feature fusion and interaction, accurately capture the style cue word information in the text cue word set, and permeate it into all aspects of the second target image generation, thereby enhancing style consistency. At the same time, the single-stream module maintains feature consistency and coherence during the feature extraction, fusion and image generation process, reduces the differences and inconsistencies caused by feature separation and independent processing, and ensures that the style of the target image is highly consistent with the reference image. In contrast, the dual-stream module processes the reference and target vector data separately and then fuses them when generating the image, which can easily lead to style inconsistencies. In addition, the single-stream module can also simplify the model architecture, improve computing and storage efficiency, reduce resource consumption and time delays caused by data transmission and independent calculations between modules, improve the speed and efficiency of image generation, and is more suitable for image generation tasks with real-time or high-performance requirements.

[0153] For example, in order to further enhance the quality and style consistency of the images generated by the image generation method, an analysis is conducted on which modules in the diffusion model play a greater role in style control during generation. Specifically, there are 57 modules in the diffusion model Flux.1, of which the first 19 modules are dual-stream modules and the last 38 modules are single-stream modules. The 57 modules can be divided into 3 parts for analysis, where the first part is 19 dual-stream modules, the second part is the first 19 of the 38 single-stream modules, and the third part is the last 19 of the 38 single-stream modules. The analysis method is: using style-consistent attention units and offset position indexing strategies in modules of different parts, and analyzing which part of the target image generated after using the style-consistent attention units and offset position indexing strategies is most similar in style to the reference image. Please refer to Figure 7 , Figure 7 The results of experiments on different modules are presented. It is easy to observe that using style-consistent attention in the first part does not effectively enhance the style consistency between generated images. In contrast, applying style-consistent attention in the second and third parts significantly improves style consistency. Therefore, we specifically use the style-consistent attention mechanism in the second and third parts (i.e., the single-stream module) to enhance the style consistency between generated images.

[0154] This application also provides an image generation device, please refer to Figure 8 , the image generating device includes:

[0155] A vector determination module 10 determines intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data;

[0156] The first generation module 20 is configured to generate a first target image based on the reference vector data.

[0157] The second generation module 30 is configured to generate a second target image by fusing the reference vector data and the target vector data, wherein the first target image and the second target image constitute a target image set corresponding to a style prompt word in the set of text prompt words.

[0158] Optionally, the second generation module 30 is further configured to:

[0159] cutting the reference vector data to obtain a reference image feature vector in the reference vector data;

[0160] cutting the target vector data to obtain a target image feature vector and a target text feature vector;

[0161] performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector;

[0162] splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector and the reference image feature vector to obtain a spliced feature vector;

[0163] generating a second target image based on the spliced feature vector.

[0164] Optionally, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector.

[0165] The second generation module 30 is further configured to:

[0166] calculate the adjusted target image query vector according to the target image query vector, a mean value of the target image query vector, a standard deviation of the target image query vector, a mean value of the reference image query vector and a standard deviation of the reference image query vector;

[0167] calculate the adjusted target image key vector according to the target image key vector, a mean value of the target image key vector, a standard deviation of the target image key vector, a mean value of the reference image key vector and a standard deviation of the reference image key vector.

[0168] Optionally, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector; the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; the target image feature vector includes a target image key vector and a target image value vector; the reference image feature vector includes a reference image key vector and a reference image value vector; and the spliced ​​feature vector includes a spliced ​​query vector, a spliced ​​key vector, and a spliced ​​value vector;

[0169] The second generating module 30 is further configured to:

[0170] splicing the target text query vector and the adjusted target image query vector to obtain a spliced ​​query vector;

[0171] splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector;

[0172] The target text value vector, the target image value vector and the reference image value vector are spliced ​​to obtain a spliced ​​value vector.

[0173] Optionally, the second generating module 30 is further configured to:

[0174] Adjusting the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector;

[0175] The target text key vector, the adjusted target image key vector and the adjusted reference image key vector are spliced ​​to obtain a spliced ​​key vector.

[0176] Optionally, the second generating module 30 is further configured to:

[0177] The position index of any basic unit in the reference image key vector is offset to obtain a shifted reference image key vector, and the step of splicing the target text key vector, the adjusted target image key vector and the reference image key vector is performed based on the shifted reference image key vector, wherein the position index of the shifted reference image key vector does not overlap with the position index of the adjusted target image query vector.

[0178] Optionally, the image generating device is applied to a preset diffusion model, wherein the preset diffusion model includes a dual-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is performed in the single-stream module.

[0179] The image generation device provided in this application, utilizing the image generation method described in the aforementioned embodiments, can address the technical problem of enhancing the stylistic consistency and efficiency of images generated using a diffusion model. Compared to the prior art, the image generation device provided in this application achieves the same beneficial effects as the image generation method described in the aforementioned embodiments. Other technical features of the image generation device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0180] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image generation method in the above-mentioned embodiment one.

[0181] Reference below Figure 9 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0182] like Figure 9As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; a storage device 1003 including, for example, a magnetic tape or hard disk; and a communication device 1009. The communication device 1009 may allow the electronic device to communicate with other devices wirelessly or wired to exchange data.

[0183] In particular, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in this application are performed.

[0184] The electronic device provided in this application, using the image generation method in the above-mentioned embodiment, can solve the technical problem of how to enhance the style consistency and efficiency of images generated by the Flux.1 diffusion model. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the image generation method provided in the above-mentioned embodiment, and the other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and are not further described here.

[0185] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the image generation method in the above-mentioned embodiment.

[0186] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0187] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: determines intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; generates a first target image based on the reference vector data; and generates a second target image by fusing the reference vector data and the target vector data, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt words in the text prompt word set.

[0188] Computer program code for carrying out the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0189] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned image generation method. This computer-readable storage medium addresses the technical problem of enhancing the stylistic consistency and efficiency of image generation using a diffusion model. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the image generation method provided in the aforementioned embodiments and are not further elaborated here.

Claims

1. An image generation method, characterized in that: The image generation method comprises: Determining intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; generating a first target image based on the reference vector data; cropping the reference vector data to obtain a reference image feature vector in the reference vector data; Cutting the target vector data to obtain a target image feature vector and a target text feature vector; performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector; splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a spliced ​​feature vector; A second target image is generated based on the spliced ​​feature vector, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt words in the text prompt word set.

2. The image generation method according to claim 1, wherein: The reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; The step of performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector comprises: calculating an adjusted target image query vector based on the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector; An adjusted target image key vector is calculated according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

3. The image generation method according to claim 1, wherein: The target text feature vector includes a target text query vector, a target text key vector, and a target text value vector; the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; the target image feature vector includes a target image key vector and a target image value vector; the reference image feature vector includes a reference image key vector and a reference image value vector; and the spliced ​​feature vector includes a spliced ​​query vector, a spliced ​​key vector, and a spliced ​​value vector; The step of splicing the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a spliced ​​feature vector includes: splicing the target text query vector and the adjusted target image query vector to obtain a spliced ​​query vector; splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector; The target text value vector, the target image value vector and the reference image value vector are spliced ​​to obtain a spliced ​​value vector.

4. The image generation method according to claim 3, wherein: The step of splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector includes: Adjusting the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector; The target text key vector, the adjusted target image key vector and the adjusted reference image key vector are spliced ​​to obtain a spliced ​​key vector.

5. The image generation method according to claim 3, wherein: Before the step of splicing the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a spliced ​​key vector, the following step further comprises: The position index of any basic unit in the reference image key vector is offset to obtain a shifted reference image key vector, and the step of splicing the target text key vector, the adjusted target image key vector and the reference image key vector is performed based on the shifted reference image key vector, wherein the position index of the shifted reference image key vector does not overlap with the position index of the adjusted target image query vector.

6. The image generation method according to claim 1, wherein: The image generation method is applied to a preset diffusion model, wherein the preset diffusion model includes a dual-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is performed in the single-stream module.

7. An image generating device, characterized in that: The image generating device comprises: a vector determination module, configured to determine intermediate vector data for generating a target image set based on a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; A first generating module, configured to generate a first target image based on the reference vector data; The second generation module is configured to crop the reference vector data to obtain a reference image feature vector in the reference vector data; crop the target vector data to obtain a target image feature vector and a target text feature vector; perform adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector; splice the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a spliced ​​feature vector; and generate a second target image based on the spliced ​​feature vector, wherein the first target image and the second target image constitute a target image set corresponding to a style prompt word in the text prompt word set.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image generation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image generation method according to any one of claims 1 to 6 are implemented.

Images