Image generation method and device, equipment and storage medium

Abstract

The invention discloses an image generation method and device, equipment and a storage medium, and relates to the technical field of image generation, and the method comprises the steps: determining intermediate vector data used for generating a target image set according to a pre-constructed text prompt word set and a noise image set, and the intermediate vector data comprises reference vector data and target vector data; generating a first target image based on the reference vector data; the reference vector data and the target vector data are fused to generate a second target image, and the first target image and the second target image form a target image set corresponding to the style cue words in the text cue word set. According to the image generation method and device, the data of the first target image used as the reference in the generation process is shared to the generation process of the second target image, so that the image set which is consistent in style, diversified and high in quality is generated, the requirement for generating the images consistent in style is effectively met, and the image generation effect and efficiency of the diffusion model are improved.

Description

Technical Field

[0001] This application relates to the field of image generation technology, and in particular, to an image generation method, device, equipment and storage medium. Background Art

[0002] In recent years, diffusion models have revolutionized multiple fields such as art creation, design, and game development due to their ability in high-quality and diverse image generation. Early diffusion models such as SD1.5 and SDXL use the U-Net architecture to generate images according to prompts. Recently, the more advanced diffusion model Flux.1 has achieved higher generated image quality and better text prompt alignment by replacing the self-attention and cross-attention used in previous models with a multi-modal attention mechanism.

[0003] A key task of diffusion models is to generate images with consistent styles. Specifically, this task aims to generate images with diverse contents but consistent styles through unified style prompts. However, it is difficult for style prompts to ensure the style consistency between generated images. For example, an attention-based method StyleAligned proposes to share the Key and Value in the self-attention mechanism of the diffusion model to enhance the style consistency between generated images. However, the design of StyleAligned is based on the self-attention mechanism used in early diffusion models such as SDXL. Directly applying StyleAligned to the diffusion model Flux.1 will result in serious content leakage among the generated image sets, and all images have extremely similar contents. And some methods for other tasks can be used in the task of generating images with consistent styles, but these methods cannot achieve satisfactory results. For example, the style image generation method needs to first provide a style reference image and then generate images with consistent styles according to the reference image. Among them, the style image generation method can be divided into two categories: adapter-based methods and fine-tuning-based methods. Both of these two methods require training and have inherent limitations: for adapter-based methods, due to the indefinability of "style", it is difficult to create a high-quality training data set for training, which leads to poor performance of adapter-based methods. For fine-tuning-based methods, collecting new data and retraining for each style significantly increases the computational and time costs. In addition, some image editing methods have also been explored to solve this task. However, since they are mainly designed for general image editing tasks, it is difficult to complete a large-scale editing of image content while keeping the style unchanged. Summary of the Invention

[0004] The main purpose of this application is to provide an image generation method, device, equipment and storage medium, aiming to solve the technical problem of how to enhance the style consistency and efficiency of images generated by diffusion models.

[0005] To achieve the above object, the present application proposes an image generation method, which includes: Determine intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; Generate a first target image based on the reference vector data; Fuse the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image form a target image set corresponding to the style prompt words in the text prompt word set.

[0006] In one embodiment, the step of fusing the reference vector data and the target vector data to generate a second target image includes: Crop the reference vector data to obtain a reference image feature vector in the reference vector data; Crop the target vector data to obtain a target image feature vector and a target text feature vector; Perform adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector; Concatenate the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a concatenated feature vector; Generate a second target image based on the concatenated feature vector.

[0007] In one embodiment, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; The step of performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector includes: Calculate an adjusted target image query vector according to the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector; Calculate an adjusted target image key vector according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

[0008] In one embodiment, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector. The adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector. The target image feature vector includes a target image key vector and a target image value vector. The reference image feature vector includes a reference image key vector and a reference image value vector. The concatenated feature vector includes a concatenated query vector, a concatenated key vector, and a concatenated value vector; The step of concatenating the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a concatenated feature vector includes: Concatenate the target text query vector and the adjusted target image query vector to obtain a concatenated query vector; Concatenate the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector; Concatenate the target text value vector, the target image value vector, and the reference image value vector to obtain a concatenated value vector.

[0009] In one embodiment, the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector includes: Adjust the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector; Concatenate the target text key vector, the adjusted target image key vector, and the adjusted reference image key vector to obtain a concatenated key vector.

[0010] In one embodiment, before the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector, further includes: Offset the position index of any basic unit in the reference image key vector to obtain an offset reference image key vector, and perform the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector based on the offset reference image key vector, wherein there is no overlap between the position index of the offset reference image key vector and the position index of the adjusted target image query vector.

[0011] In one embodiment, the image generation method is applied to a preset diffusion model, wherein the preset diffusion model includes a dual-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is performed in the single-stream module.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes an image generating device, the image generating device comprising: A vector determination module, used to determine intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; A first generating module, used for generating a first target image based on the reference vector data; The second generating module is used to fuse the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt word in the text prompt word set.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image generation method described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image generation method described above are implemented.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application first determines intermediate vector data for generating a target image set based on a pre-constructed set of text prompts and a set of noise images. Among them, the intermediate vector data includes reference vector data and target vector data, thus providing basic feature data for subsequent generation of images with a specific style, enabling the generated images to be constructed based on the style-related information included in the text prompts, and laying a foundation for subsequent generation of a consistent-style image set. Then, a first target image is generated based on the reference vector data, providing a style reference benchmark for generating the target image. Then, the reference vector data and the target vector data are fused to generate a second target image. Among them, the first target image and the second target image constitute a target image set corresponding to the style prompts in the text prompt set. Thus, on the basis of referring to the style of the first target image, a second target image matching the first target image used as a reference is generated according to the features and information contained in the target vector data, ensuring both the style consistency between the image and the reference image and the content characteristics contained in the target vector data. Therefore, in the process of generating the image set, each image shows diverse content under the premise of unified style, meeting the user's need for generating a specific-style image set, effectively solving the problem of style consistency, and making the generated image set conform to the expectation and remain consistent in style.

[0016] In summary, this application first determines self-attention vector data including reference vector data and target vector data according to the text prompt set and the noise image set, then generates a first target image based on the reference vector data respectively, and at the same time fuses the reference and target vector data to generate a second target image, that is, sharing the data in the generation process of the first target image to the generation process of the second target image. Finally, a target style generation image set corresponding to the style prompts is combined, avoiding the problems of content leakage and style inconsistency caused by directly applying early methods to the diffusion model, as well as the technical problems existing in the style image generation method based on adapters or fine-tuning, such as difficult creation of training data, high computational time cost, and difficulty in substantially editing content while maintaining the style unchanged. It realizes the generation of a consistent-style, diverse, and high-quality image set, effectively meets the need for generating consistent-style images, and improves the image generation effect and efficiency of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art.

[0019] Figure 1 Schematic flow chart provided for the first embodiment of the image generation method of the present application; Figure 2 Schematic diagram of the cropping scenario of the image generation method provided for the first embodiment of the present application; Figure 3 Schematic diagram of the position indexing scenario of the image generation method provided for the second embodiment of the present application; Figure 4 Another schematic diagram of the position indexing scenario of the image generation method provided for the second embodiment of the present application; Figure 5 Schematic flow chart provided for the second embodiment of the image generation method of the present application; Figure 6 Schematic diagram of the model architecture of the image generation method provided for the third embodiment of the present application; Figure 7 Schematic diagram of the image generation effect of the image generation method provided for the third embodiment of the present application; Figure 8 Schematic diagram of the module structure of the image generation device according to the embodiment of the present application; Figure 9 Schematic diagram of the device structure of the hardware operating environment involved in the image generation method according to the embodiment of the present application. Detailed implementation manners

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0021] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the accompanying drawings of the specification and specific implementation manners.

[0022] The main solution of the embodiment of the present application is: determining intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; generating a first target image based on the reference vector data; fusing the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image form a target image set corresponding to the style prompt words in the text prompt word set.

[0023] Since it is difficult to ensure the style consistency among the generated images with style prompts, for example, sharing the Key and Value in the self-attention mechanism of the diffusion model to enhance the style consistency among the generated images. However, this design is based on the self-attention mechanism used in early diffusion models such as SDXL. Directly applying StyleAligned to the diffusion model Flux.1 will lead to serious content leakage among the generated image sets, and all images have extremely similar content. And some methods for other tasks can be used in the task of generating images with consistent style, but these methods cannot achieve satisfactory results. For example, the style image generation method needs to first provide a style reference image and then generate images with consistent style according to the reference image. Among them, the style image generation method can be divided into two categories: adapter-based methods and fine-tuning-based methods. Both of these two methods require training and have inherent limitations, including difficulty in creating a high-quality training data set for training, or excessive computational and time costs. In addition, some image editing methods have also been explored to solve this task. However, since they are mainly designed for general image editing tasks, it is difficult to greatly edit the image content while keeping the style unchanged.

[0024] This application provides a solution. First, determine the self-attention vector data including reference vector data and target vector data according to the text prompt set and the noise image set. Then, generate the first target image based on the reference vector data respectively, and at the same time generate the second target image by fusing the reference and target vector data, that is, share the data in the generation process of the first target image to the generation process of the second target image. Finally, combine them into a target style generated image set corresponding to the style prompt, avoiding the content leakage and style inconsistency problems caused by directly applying the early methods to the diffusion model, as well as the technical problems such as difficulty in creating training data, high computational time cost, and difficulty in greatly editing the content while keeping the style unchanged in the adapter-based or fine-tuning-based style image generation methods. It realizes the generation of a set of images with consistent, diverse and high-quality styles, effectively meets the demand for generating images with consistent style, and improves the image generation effect and efficiency of the diffusion model.

[0025] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions. Hereinafter, an electronic device will be taken as an example to illustrate this embodiment and the following embodiments.

[0026] Based on this, the embodiment of this application provides an image generation method, referring to Figure 1 , Figure 1 which is the flowchart of the first embodiment of the image generation method of this application.

[0027] In this embodiment, the image generation method includes steps S10 to S30: Step S10, determining intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, where the intermediate vector data includes reference vector data and target vector data; It should be noted that the text prompt word set refers to a set of text information including image content descriptions, style requirements, etc., which is used to guide the direction and style of image generation; the noise image set refers to a set of image data with random noise characteristics, which is used as the initial input of the generation model and is used to gradually guide the generation of images that meet the requirements in the diffusion model; the intermediate vector data refers to the intermediate data of the model when generating images, that is, in the self-attention mechanism of the model, the vector data containing relevant information such as query vectors, key vectors, and value vectors obtained by processing the input text prompt words and noise images, which is used to capture the association and importance between text and image features; the reference vector data refers to the vector data extracted and determined from the text prompt word set and the noise image set, which is used to generate images with a reference style and contains feature information related to the reference style; the target vector data refers to the vector data extracted and determined from the text prompt word set and the noise image set, which is used to generate images with target content and target style and contains feature information related to the target image content and style.

[0028] In addition, it should be noted that the image generation method proposed in this application can be applied to the Flux.1 diffusion model, aiming to encourage the style of the second target image generated based on the target vector data to align with the style of the first target image generated based on the reference vector data.

[0029] It can be understood that since existing image generation schemes have difficulty directly using text prompt words and noise images to effectively generate feature vector data that can be used for subsequent generation of images with consistent styles, performing step S10 can avoid problems such as inconsistent image styles or poor quality caused by unreasonable construction of feature vector data, thereby providing accurate and effective initial feature vector data for subsequent generation of images with consistent styles based on text prompt words.

[0030] Exemplarily, firstly, a set of text prompt words containing different contents needs to be formed around a style prompt word, and each prompt word in the set is in the form of: <content prompt word> in <style prompt word>, such as "dog, 3D real style", "table, 3D real style", "earth, 3D real style", etc., and then each text prompt word in the text prompt word set is encoded into a text embedding vector by a text encoder, and each noise image in the noise image set is encoded by an image encoder to obtain an image embedding vector. Then, the text embedding vector and the image embedding vector are sent as input to the pre-trained self-attention network to calculate the intermediate vector data used in the self-attention mechanism. In this process, the intermediate vector data is divided into reference vector data and target vector data through a pre-set network structure and parameter setting. The reference vector data can be determined based on a text prompt word and a corresponding noise image, and the target vector data can be determined based on the remaining text prompt words and the corresponding noise images.

[0031] Step S20, generating a first target image based on the reference vector data; It should be noted that the first target image refers to an example image with a specific target style generated based on the reference vector data, which is used to provide a style reference and benchmark for the subsequently generated target images to ensure the style consistency of the entire image set.

[0032] For example, a dedicated image generation network, such as a generator based on a generative adversarial network (GAN) or a decoder based on a variational autoencoder (VAE), can be used to take the reference vector data as input, and after being processed by a multi-layer neural network, the reference vector data is gradually mapped to the image space. In this process, the spatial resolution and detail information of the image can be increased through operations such as upsampling and convolution, and finally a reference image that can reflect the target style, i.e., the first target image, is generated.

[0033] Step S30, fusing the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image constitute a target image set corresponding to the style prompt word in the text prompt word set; It should be noted that the second target image refers to an image with target style and target content generated based on the joint action of reference vector data and target vector data, and is the main component of the final target style generated image set; the target image set refers to an image set formed by combining reference images and target images, where all images have a target style corresponding to the style prompt words in the text prompt word set, satisfying the user's demand for generating an image set with a consistent style.

[0034] It can be understood that since it is often difficult to generate a target image that meets both style requirements and has diverse content in existing image generation schemes, performing step S30 can avoid problems such as the style of the generated target image being inconsistent with the reference image, the content being single, or not meeting expectations, thereby generating a series of target images that are consistent with the reference image in style and rich in content.

[0035] Exemplarily, the reference vector data and the target vector data are subjected to feature splicing or feature fusion operations to form a comprehensive feature vector, and then this comprehensive feature vector is input into another image generation network. Similar to the generation process of the reference image, through the processing of a multi-layer neural network, the fused feature vector is mapped into the image space to generate a target image that meets both the target style and contains the target content, that is, the second target image.

[0036] In a feasible implementation manner, step S30 may include steps S31 to S35: Step S31, crop the reference vector data to obtain a reference image feature vector in the reference vector data; It should be noted that the reference image feature vector refers to the relevant vector information that can reflect the style characteristics of the reference image, which is cropped from the reference vector data, and includes the reference image query vector obtained by mapping the image features , the reference image key vector and the reference image value vector etc. These vectors jointly carry the style characteristics of the reference image and are used to guide the style generation of the target image.

[0037] In addition, it should be noted that the reference vector data also includes a reference text feature vector, which includes a reference text query vector obtained by mapping the text features , the reference text key vector and the reference text value vector .

[0038] It can be understood that since the reference vector data contains various information, and the part related to the image features is the key content for providing style reference when generating the target image, performing step S31 can avoid bringing irrelevant or redundant information into the subsequent image generation process through the cropping operation, thereby accurately extracting the feature vector of the reference image and providing an accurate style reference basis for the subsequent generation of the target image.

[0039] Exemplarily, please refer to Figure 2 , the reference vector data, that is, the query key and value (shown in the figure as 、 、 ) is regarded as a high-dimensional vector or tensor, which contains various feature information related to the image. By cutting this vector, for example, segmenting it according to a predefined index range or feature dimension, the part specifically used to describe the image features is extracted, and the reference image feature vector is obtained. , and (shown in the figure as , and ), so as to accurately locate and obtain feature vectors closely related to image style and other features from the reference vector data containing rich information, providing key reference information for the subsequent generation of target images with consistent style.

[0040] Step S32, cutting the target vector data to obtain a target image feature vector and a target text feature vector; It should be noted that the target image feature vector refers to the feature vector part related to the target image content cut from the target vector data, including the target image query vector, the target image key vector and the target image value vector, etc., which are used to reflect the content characteristics that the target image should have; the target text feature vector refers to the feature vector part related to the text prompt cut from the target vector data, including the target text query vector, the target text key vector and the target text value vector, etc., which are used to reflect the content characteristics required by the text prompt.

[0041] It can be understood that since the target vector data contains both the content feature information of the target image and the feature information related to the text prompt, they need to be distinguished for subsequent processing. Therefore, step S32 is performed. The cropping operation can avoid confusing the image content features and text features when generating the target image, and the target image feature vector is used to reflect the image content, and the target text feature vector is used to reflect the content required by the text prompt, thereby ensuring that the image content and text requirements can be accurately integrated when generating the target image.

[0042] For example, please refer to Figure 2 , target vector data , and (shown in the figure as 、 、 )It is also a comprehensive high-dimensional vector that integrates the content features of the target image and the relevant features of the text prompt. By designing a cropping strategy, such as splitting according to different fields or feature segments in the data structure, the target vector data can be split into two parts: one is the feature vector that focuses on describing the content of the target image itself, i.e., the target image feature vector 、 、 (shown in the figure as 、 and ); the other part is the feature vector corresponding to the text prompt content, i.e., the target text feature vector and and (shown in the figure as 、 and ). Thus, the image content features and text features are processed separately so that the content and text-related characteristics of the target image can be accurately controlled respectively during the subsequent generation process.

[0043] Step S33: Perform adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain the adjusted target image feature vector; It can be understood that since there may be differences in the distributions of the target image feature vector and the reference image feature vector, directly using them may cause the target image to not accurately align with the reference image style in terms of style. Therefore, step S33 is performed. Through adaptive normalization processing, this problem of style inconsistency can be avoided, and the distribution of the target image feature vector can be adjusted to be adapted to the distribution of the reference image feature vector, so that the subsequently generated target image can better match the reference image in terms of style.

[0044] Exemplarily, by calculating the mean and standard deviation of the target image feature vector and combining them with the mean and standard deviation of the reference image feature vector, the target image feature vector is adjusted. Specifically, a method similar to instance normalization can be used. First, the target image feature vector is normalized to eliminate its own statistical characteristics, and then it is scaled to the same mean and standard deviation range as the reference image feature vector. After such processing, the obtained adjusted target image feature vector can match the reference image feature vector in terms of distribution. Thus, when generating the target image, the target image can match the reference image in terms of style while retaining the content features of the target image itself.

[0045] Step S34: Concatenate the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain the concatenated feature vector; It can be understood that since it is necessary to comprehensively consider various aspects of information such as the text prompt content, the content features of the target image, and the style features of the reference image in order to generate a target image that not only meets the text requirements but also has the target content and reference style, step S34 is performed. Through the splicing operation, the omission or isolated processing of information can be avoided, and multiple feature vectors can be integrated into a complete feature representation, enabling the text prompt content, the content features of the target image itself, and the style features of the reference image to be referred to simultaneously when generating the target image, providing a comprehensive and rich information input for the subsequent generation of the target image, thereby generating a target image that meets the requirements.

[0046] Exemplarily, the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector are concatenated together in a certain order to form a higher-dimensional comprehensive feature vector. For example, the concatenation can be performed along the channel dimension or the depth direction of the feature map. The concatenated feature vector integrates various aspects of information such as the text prompt information, the content features of the target image, and the style features of the reference image, providing a comprehensive and rich feature basis for the subsequent generation of the target image.

[0047] Step S35, generating a second target image based on the concatenated feature vector.

[0048] It can be understood that since the concatenated feature vector has already integrated key information in various aspects such as the text prompt, the target image content, and the reference image style, and has the sufficient conditions for generating a target image that meets the requirements, step S35 is performed. By generating an image based on the concatenated feature vector, the problem that the generated image does not meet the requirements in terms of content, style, etc. can be avoided, and a target image that not only meets the text prompt content, but also has the content features of the target image and is consistent with the reference image style can be generated.

[0049] Exemplarily, self-attention calculation of the dot product is performed on the concatenated feature vector, and the calculation result is used as the input and input into a pre-trained generation network, such as a generator based on a convolutional neural network. The generation network gradually converts the high-dimensional feature vector into image data with a spatial structure through a series of operations such as upsampling, convolution, and activation functions. In this process, the network will generate a target image that not only meets the requirements of the text prompt content, but also has the content features of the target image and is consistent with the reference image style according to the information such as the text prompt, the target content, and the reference style contained in the concatenated feature vector.

[0050] In this embodiment, by sequentially performing feature vector cropping and separation, adaptive normalization processing, and feature stitching, problems such as inconsistent generated image styles and content-style conflicts caused by mismatches between target image features and reference style features, and poor fusion of text prompts and image content features are avoided, achieving a high-quality image effect that meets the requirements in both content and style. At the same time, the cropping operation accurately extracts the required feature information, improving the processing efficiency and pertinence; the adaptive normalization processing ensures that the target image features and the reference style features have the same distribution, guaranteeing the accurate transfer and unity of the style; the feature stitching integrates multi-source feature information, providing comprehensive guidance for image generation, ensuring that the generated image not only conforms to the text prompt content but also has the content features of the target image and the style features of the reference image.

[0051] This embodiment provides an image generation method. First, self-attention vector data including reference vector data and target vector data is determined according to a text prompt word set and a noise image set. Then, a first target image is generated based on the reference vector data, and at the same time, a second target image is generated by fusing the reference and target vector data, that is, sharing the data of the first target image in the generation process to the generation process of the second target image. Finally, a target style generated image set corresponding to the style prompt word is combined, avoiding the problems of content leakage and inconsistent style caused by directly applying early methods to diffusion models, as well as the technical problems of existing style image generation methods based on adapters or fine-tuning, such as difficulty in creating training data, high computational time cost, and difficulty in significantly editing content while maintaining the style unchanged. It realizes the generation of a consistent, diverse, and high-quality image set, effectively meeting the demand for generating consistent-style images, and improving the image generation effect and efficiency of the diffusion model.

[0052] In a feasible embodiment, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; Step S33 may include steps S331 to S332: Step S331, calculate the adjusted target image query vector according to the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector; It should be noted that the target image query vector is the query vector representation of the target image in the self-attention mechanism, representing the feature direction that the target image expects to retrieve and match, which can be used to guide feature selection in the generation process, so that the generated image can contain content features that meet the requirements, and after subsequent adjustment, it can also adapt to the style features of the reference image; the reference image query vector is the query vector representation of the reference image in the self-attention mechanism, which is used to interact with other vectors (such as key vectors) to determine the degree of correlation between other vectors and itself. For the reference image, the query vector contains the key information for retrieving and matching features in the generation process, reflecting relevant information such as the style features of the reference image.

[0053] It can be understood that since the distributions of the target image query vector and the reference image query vector often do not match, step S331 is performed. Through adjustment based on the mean and standard deviation, it is possible to avoid the problem of style mismatch caused by the inconsistent distributions of the target image query vector and the reference image query vector, and adjust the distribution of the target image query vector to be close to that of the reference image query vector. Thus, when generating the target image, it is possible to better share the style features of the reference image and keep the target image unified with the reference image in terms of style. Exemplarily, the target image query vector is aligned with the reference image query vector to obtain the adjusted target image query vector :

[0054] Among them, the operation of adaptive normalization is defined as:

[0055] Among them, represents the mean of the variable, represents the standard deviation of the variable.

[0056] Step S332, calculate the adjusted target image key vector according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

[0057] It should be noted that the target image key vector is the key vector representation of the target image in the self-attention mechanism, which reflects a dimension of the characteristics of the target image itself. It interacts with the query vector in the self-attention mechanism, provides a basis for determining the correlation between different parts of the target image and the correlation with the features of the reference image, and is one of the key factors for realizing the style adjustment and content generation of the target image; the reference image key vector is the key vector representation of the reference image in the self-attention mechanism, which is used to calculate with the query vector to obtain the correlation weight between the two. When interacting with other vectors (such as the query vector), it can reflect the characteristics of the reference image in a specific feature dimension.

[0058] It can be understood that since the distributions of the target image key vector and the reference image key vector are often different, step S332 is performed to calculate the adjusted target image key vector so that the distribution of the target image key vector matches the reference image key vector, which can avoid the style differences caused by the different distributions of the target image key vector and the reference image key vector, thereby ensuring that the generated target image can accurately reflect the style of the reference image and enhancing the consistency of the image style generated by the model.

[0059] Exemplarily, the target image key vector is aligned to the reference image key vector to obtain the adjusted target image key vector :

[0060] In this embodiment, a calculation adjustment based on statistics is performed, which avoids the style mismatch problem caused by the inconsistent distribution of the target image feature vector and the reference image, and realizes the adjustment of the distributions of the query and key vectors of the target image to match the reference image, so that the style features of the reference image can be better shared when generating the target image, ensuring the consistency of the image generation style.

[0061] In a feasible embodiment, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector, the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector, the target image feature vector includes a target image key vector and a target image value vector, the reference image feature vector includes a reference image key vector and a reference image value vector, and the spliced feature vector includes a spliced query vector, a spliced key vector, and a spliced value vector; Step S34 may include steps S341 to S343: Step S341: Concatenate the target text query vector and the adjusted target image query vector to obtain a concatenated query vector. It can be understood that since it is necessary to fuse the text prompt and the image content at the query vector level to ensure that the generated image meets both the text requirements and reflects the image features, step S341 is performed. Through the concatenation operation, problems such as inconsistent features or insufficient fusion caused by the separation of the text and image query vectors can be avoided, realizing the integration of the text and image query vectors into a unified query vector, providing a more comprehensive and accurate query basis for subsequent feature matching and attention calculation, and enabling the generated image to better balance the text prompt and the image content features.

[0062] Exemplarily, the target text query vector and the adjusted target image query vector are concatenated to obtain a concatenated query vector :

[0063] Step S342: Concatenate the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector. It can be understood that since it is necessary to comprehensively consider the information of the text prompt, the target image content, and the reference image style at the key vector level in order to more comprehensively calculate the correlation between features, step S342 is performed. This can avoid problems such as deviation or incompleteness in the calculation of feature correlation caused by relying only on a single key vector. By concatenation, the integration of multi-source key vector information is realized, enhancing the comprehensiveness and accuracy of feature matching, and making the generated image more in line with the requirements in terms of content, style, etc.

[0064] Exemplarily, the target text key vector , the adjusted target image key vector and the reference image key vector are concatenated to obtain a concatenated key vector :

[0065] Step S343: Concatenate the target text value vector, the target image value vector, and the reference image value vector to obtain a concatenated value vector.

[0066] It should be noted that the target text value vector is the representation of the text prompt after encoding in the value vector part, which contains detailed information about the text features and is used to reflect the specific content and details described in the text during image generation; the reference image value vector is the feature representation of the reference image at the value vector level, which contains detailed feature information of the reference image and is used to provide reference information such as style during the image generation process to ensure that the generated image has the same style features as the reference image.

[0067] It can be understood that since it is necessary to integrate the detailed information of the text prompt content, the target image features, and the reference image style to provide rich information support for the final image generation, step S343 is performed, which can avoid problems such as insufficient details in image generation, inaccurate style or content caused by information omission or a single value vector. By splicing, the integration of multi-faceted information is achieved, ensuring that the generated image has rich details, accurate style, and content that meets the requirements.

[0068] Exemplarily, the target text value vector , the target image value vector and the reference image value vector are spliced to obtain the spliced value vector :

[0069] In this embodiment, by splicing the feature vectors, problems such as incomplete content and inconsistent style of the generated image caused by the inability of a single feature vector to fuse multi-source information are avoided, and the effective integration of the text prompt, the content features of the target image, and the style features of the reference image in the query, key, and value dimensions is realized, so as to ensure that the generated target image can accurately reflect the text prompt content, be consistent with the reference image style, and have the technical effect of rich details.

[0070] In a feasible embodiment, step S342 may include steps S301 to S302: Step S301, adjust the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector; It should be noted that the preset intensity control parameter refers to a parameter used to adjust the influence degree of the reference image key vector on the spliced key vector during the image generation process. By setting this parameter, the attention weight between the spliced query vector and the reference image key vector can be regulated, so as to realize the fine regulation of the style intensity of the generated image. For example, when it is desired that the style of the generated image is closer to the reference image, the value of the preset intensity control parameter can be increased; conversely, if a weaker style feature is desired, the value of this parameter can be decreased.

[0071] It can be understood that when fusing the reference image key vector with other key vectors, it is often necessary to control the influence degree of the reference image style features on the generation of the target image. Therefore, performing step S301 can avoid the situation where the reference image style is too strong or insufficient, resulting in the generated image style not meeting the expectations. By introducing a preset intensity control parameter to adjust the reference image key vector, it is possible to avoid the problem that the proportion of the reference image key vector in the splicing process is unreasonable, leading to the style being overly dominant or not prominent enough. This realizes the precise control of the influence degree of the reference image style features, enabling the style of the generated image to be reflected according to the expected intensity, and enhancing the flexibility and accuracy of style control.

[0072] Step S302: Splice the target text key vector, the adjusted target image key vector, and the adjusted reference image key vector to obtain a spliced key vector.

[0073] It can be understood that since it is necessary to comprehensively consider the information of the text prompt, the target image content, and the reference image style at the key vector level to comprehensively guide the generation of the target image, performing step S302 can avoid the situation where the information of the generated image in terms of content, style, etc. is incomplete or the fusion is insufficient. By splicing these three key vectors together, the effective integration of multi-source information is achieved, providing a richer and more comprehensive feature representation for the subsequent generated image, enabling the generated image to better meet the requirements in terms of content, style, etc.

[0074] Exemplarily, the target text key vector 、the adjusted target image key vector and the adjusted reference image key vector are spliced to obtain the spliced key vector :

[0075] wherein, is a preset intensity control parameter.

[0076] In this embodiment, by performing intensity control adjustment on the reference image key vector and splicing it with the target text key vector and the adjusted target image key vector, the problem of unnatural style fusion and disharmony between content and style caused by the reference image key vector having too strong or too weak an impact on the generation of the target image during splicing is avoided. The precise control of the influence degree of the reference image style features and the effective integration of multi-source features are realized, thereby ensuring that the generated target image is consistent with the reference image in style and the content meets the text requirements.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, before step S34, the image generation method further includes step S01: Step S01, offset the position index of any basic unit in the reference image key vector to obtain an offset reference image key vector, so as to perform the step of splicing the target text key vector, the adjusted target image key vector and the reference image key vector based on the offset reference image key vector, wherein, there is no overlap between the position index of the offset reference image key vector and the position index of the adjusted target image query vector.

[0078] It should be noted that in the vector data structure, each basic unit refers to an independent element or dimension in the vector, representing a specific aspect or attribute of the image or text feature. In the context of the key vector, each basic unit may correspond to a specific dimension of the image or text feature; the position index is used to identify the position of each basic unit in the vector. During the vector splicing or processing, the position index helps the model to identify and distinguish the sources and uses of different feature vectors.

[0079] It can be understood that during the image generation process, when splicing the target text key vector, the adjusted target image key vector and the reference image key vector, it is necessary to ensure that the position indexes of these vectors do not overlap, so as to avoid too high attention weights between the reference image key vector and the adjusted target image query vector, which may cause the generated target image to be too similar to the reference image. Therefore, step S01 is performed. By offsetting the position index of each basic unit in the reference image key vector, it can be ensured that the position index of the offset reference image key vector does not overlap with the position index of the adjusted target image query vector. This offset operation can effectively control the attention weights of the reference image key vector and the adjusted target image query vector within a reasonable range, and effectively enhance the alignment degree between the generated image and the text prompt.

[0080] Exemplarily, the diffusion model Flux.1 often uses rotational position encoding to introduce position information for attention calculation. Specifically, for the and obtained by mapping the image features, a position index is set for each basic unit token therein, where , respectively represent the height and width of the latent variable. For the reference image key vector , the default position index is also The design of, can be referred to Figure 3, but such a design will result in an overly high attention weight between the adjusted target image query vector and , leading to an extremely high similarity between the generated image and the reference image. Therefore, an offset position indexing strategy is proposed. Specifically, please refer to Figure 4 , and the default position index of is not set to , but is set to , , such a design can prevent the position indexes of and from overlapping.

[0081] In this embodiment, by proposing an offset position indexing strategy, the problem that the generated target image may be too similar to the reference image due to overlapping position indexes between vectors during splicing is avoided, realizing that each feature vector has a unique position identifier in the overall vector after splicing, so as to correctly distinguish and utilize feature information from different sources for image generation, effectively controlling the attention weights of the reference image key vector and the adjusted target image query vector within a reasonable range, and effectively enhancing the alignment between the generated image and the text prompt.

[0082] Exemplarily, to help understand the implementation process of the image generation method obtained by combining this embodiment with the above-mentioned Embodiment 1, please refer to Figure 5 , Figure 5 , which provides a schematic diagram of the brief process of an image generation method. Specifically: Share the keys and values of the shared reference image during the generation process to the generation process of the target image. Specifically, for the reference image The query key and value in the multi-modal attention are cropped into , and obtained by image feature mapping, as well as , and obtained by text feature mapping. Similarly, the , and in the target image are also cropped into , , and , , .

[0083] Then, the , and Shared during the generation process of the target image through a series of operations. Specifically, first, through an adaptive normalization operation, , , to , , are aligned:

[0084]

[0085] After that, concatenation is performed on the length sequences to obtain the final queries, keys, and values:

[0086]

[0087]

[0088] After that, , , are subjected to dot-product self-attention calculation to obtain the final result, and thus the target image is generated based on this result. At the same time, during this process, an offset position index strategy needs to be introduced, and the default position index of is not set to , but set to , which can prevent the position indices of and from overlapping.

[0089] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, for the same or similar content as in the above-mentioned first and second embodiments, reference can be made to the above introduction and will not be elaborated hereinafter.

[0090] On this basis, please refer to Figure 6 , the image generation method is applied to a preset diffusion model, where the preset diffusion model includes a two-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is executed in the single-stream module. It can be understood that Figure 6 the style-consistent attention unit in

[0091] executes the fusion process of the reference vector data and the target vector data corresponding to step S30 of the present application.

[0092] It can be understood that the single - stream module integrates reference vector data and target vector data into one processing flow, which can promote in - depth feature fusion and interaction, accurately capture the style prompt word information in the text prompt word set, and penetrate it into all aspects of the generation of the second target image, thereby enhancing style consistency. At the same time, the single - stream module maintains feature consistency and coherence during feature extraction, fusion, and image generation, reducing the differences and incoordination caused by feature separation and independent processing, and ensuring that the style of the target image is highly consistent with that of the reference image. In contrast, the two - stream module processes reference and target vector data separately and then fuses them during image generation, which is prone to style inconsistency. In addition, the single - stream module can also simplify the model architecture, improve computational and storage efficiency, reduce resource consumption and time delay caused by data transmission and independent operations between modules, and improve the speed and efficiency of image generation, making it more suitable for image generation tasks with real - time or high - performance requirements.

[0093] Exemplarily, in order to further enhance the quality and style consistency of the images generated by the image generation method, an analysis was conducted on which modules in the diffusion model play a greater role in style control during generation. Specifically, there are 57 modules in the diffusion model Flux.1, among which the first 19 modules are two - stream modules and the last 38 modules are single - stream modules. The 57 modules can be evenly divided into 3 parts for analysis. The first part is 19 two - stream modules, the second part is the first 19 of the 38 single - stream modules, and the third part is the last 19 of the 38 single - stream modules. The analysis method is as follows: Use the style - consistent attention unit and offset position index strategy in the modules of different parts, and analyze which part generates the target image with the most similar style to the reference image after using the style - consistent attention unit and offset position index strategy. Please refer to Figure 7 , Figure 7 shows the results of experiments conducted on different modules. It is easy to observe that using style - consistent attention in the first part cannot effectively enhance the style consistency between the generated images. In contrast, applying style - consistent attention in the second part and the third part significantly improves the style consistency. Therefore, we specifically adopt the style - consistent attention mechanism in the second part and the third part (i.e., the single - stream module) to enhance the style consistency between the generated images.

[0094] This application also provides an image generation device. Please refer to Figure 8 , the image generation device includes: A vector determination module 10, which determines intermediate vector data for generating a target image set according to a pre - constructed text prompt word set and a noise image set, where the intermediate vector data includes reference vector data and target vector data; A first generation module 20, which is used to generate a first target image based on the reference vector data; The second generation module 30 is used to generate a second target image by fusing the reference vector data and the target vector data, where the first target image and the second target image form a target image set corresponding to the style prompt words in the text prompt word set.

[0095] Optionally, the second generation module 30 is further used for: Cropping the reference vector data to obtain a reference image feature vector in the reference vector data; Cropping the target vector data to obtain a target image feature vector and a target text feature vector; Performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector; Concatenating the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a concatenated feature vector; Generating a second target image based on the concatenated feature vector.

[0096] Optionally, the reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; The second generation module 30 is further used for: Calculating the adjusted target image query vector according to the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector; Calculating the adjusted target image key vector according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

[0097] Optionally, the target text feature vector includes a target text query vector, a target text key vector, and a target text value vector, the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector, the target image feature vector includes a target image key vector and a target image value vector, the reference image feature vector includes a reference image key vector and a reference image value vector, and the concatenated feature vector includes a concatenated query vector, a concatenated key vector, and a concatenated value vector; The second generation module 30 is further used for: Concatenate the target text query vector and the adjusted target image query vector to obtain a concatenated query vector; Concatenate the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector; Concatenate the target text value vector, the target image value vector, and the reference image value vector to obtain a concatenated value vector.

[0098] Optionally, the second generation module 30 is further configured to: Adjust the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector; Concatenate the target text key vector, the adjusted target image key vector, and the adjusted reference image key vector to obtain a concatenated key vector.

[0099] Optionally, the second generation module 30 is further configured to: Offset the position index of any basic unit in the reference image key vector to obtain an offset reference image key vector, and perform the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector based on the offset reference image key vector, where there is no overlap between the position index of the offset reference image key vector and the position index of the adjusted target image query vector.

[0100] Optionally, the image generation device is applied to a preset diffusion model, where the preset diffusion model includes a two-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is executed in the single-stream module.

[0101] The image generation device provided by the present application, adopting the image generation method in the above embodiment, can solve the technical problem of how to enhance the style consistency and efficiency of the images generated by the diffusion model. Compared with the prior art, the beneficial effects of the image generation device provided by the present application are the same as those of the image generation method provided by the above embodiment, and other technical features in the image generation device are the same as those disclosed in the method of the above embodiment, and will not be elaborated here.

[0102] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image generation method in the first embodiment above.

[0103] Next, refer to Figure 9, which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 9 The electronic device shown is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0104] As Figure 9 shown, the electronic device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1002 or the program loaded from the storage device 1003 into the random access memory 1004. In the random access memory 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other through a bus 1005. The input / output interface 1006 is also connected to the bus. Generally, the following systems may be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data.

[0105] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device, or installed from the storage device 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the disclosed embodiments of the present application are executed.

[0106] The electronic device provided by this application adopts the image generation method in the above-mentioned embodiment, and can solve the technical problem of how to enhance the style consistency and efficiency of the images generated by the Flux.1 diffusion model. Compared with the prior art, the beneficial effects of the electronic device provided by this application are the same as those of the image generation method provided by the above-mentioned embodiment, and other technical features in this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0107] This application provides a computer-readable storage medium, which has computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the image generation method in the above-mentioned embodiment.

[0108] The computer-readable storage medium provided by this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM: Random Access Memory), read-only memory (ROM: Read Only Memory), erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0109] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device is enabled to: determine intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, where the intermediate vector data includes reference vector data and target vector data; generate a first target image based on the reference vector data; fuse the reference vector data and the target vector data to generate a second target image, where the first target image and the second target image form a target image set corresponding to the style prompt words in the text prompt word set.

[0110] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages.

[0111] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above image generation method, which can solve the technical problem of how to enhance the style consistency and efficiency of the images generated by the diffusion model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the image generation method provided in the above embodiments, and will not be elaborated here.

Claims

1. An image generation method, characterized in that, The described image generation method includes: Determining intermediate vector data for generating a target image set according to a pre - constructed text prompt set and a noise image set, wherein the intermediate vector data includes reference vector data and target vector data; Generating a first target image based on the reference vector data; Fusing the reference vector data and the target vector data to generate a second target image, wherein the first target image and the second target image form a target image set corresponding to the style prompts in the text prompt set.

2. The image generation method according to claim 1, wherein The step of fusing the reference vector data and the target vector data to generate a second target image includes: Cropping the reference vector data to obtain a reference image feature vector in the reference vector data; Cropping the target vector data to obtain a target image feature vector and a target text feature vector; Performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector; Concatenating the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a concatenated feature vector; Generating a second target image based on the concatenated feature vector.

3. The image generation method according to claim 2, characterized in that, The reference image feature vector includes a reference image query vector and a reference image key vector, the target image feature vector includes a target image query vector and a target image key vector, and the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector; The step of performing adaptive normalization processing on the reference image feature vector and the target image feature vector to obtain an adjusted target image feature vector includes: Calculating an adjusted target image query vector according to the target image query vector, the mean of the target image query vector, the standard deviation of the target image query vector, the mean of the reference image query vector, and the standard deviation of the reference image query vector; Calculating an adjusted target image key vector according to the target image key vector, the mean of the target image key vector, the standard deviation of the target image key vector, the mean of the reference image key vector, and the standard deviation of the reference image key vector.

4. The image generation method according to claim 2, wherein The target text feature vector includes a target text query vector, a target text key vector, and a target text value vector, the adjusted target image feature vector includes an adjusted target image query vector and an adjusted target image key vector, the target image feature vector includes a target image key vector and a target image value vector, the reference image feature vector includes a reference image key vector and a reference image value vector, and the concatenated feature vector includes a concatenated query vector, a concatenated key vector, and a concatenated value vector; The step of concatenating the target text feature vector, the adjusted target image feature vector, the target image feature vector, and the reference image feature vector to obtain a concatenated feature vector includes: Concatenate the target text query vector and the adjusted target image query vector to obtain a concatenated query vector; Concatenate the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector; Concatenate the target text value vector, the target image value vector, and the reference image value vector to obtain a concatenated value vector.

5. The image generation method according to claim 4, wherein The step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector includes: Adjust the reference image key vector according to a preset intensity control parameter to obtain an adjusted reference image key vector; Concatenate the target text key vector, the adjusted target image key vector, and the adjusted reference image key vector to obtain a concatenated key vector.

6. The image generation method according to claim 4, wherein Before the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector to obtain a concatenated key vector, it further includes: Offset the position index of any basic unit in the reference image key vector to obtain an offset reference image key vector, and perform the step of concatenating the target text key vector, the adjusted target image key vector, and the reference image key vector based on the offset reference image key vector, where there is no overlap between the position index of the offset reference image key vector and the position index of the adjusted target image query vector.

7. The image generation method according to claim 1, wherein The image generation method is applied to a preset diffusion model, where the preset diffusion model includes a two-stream module and a single-stream module, and the fusion process of the reference vector data and the target vector data is executed in the single-stream module.

8. An image generation device, characterized in that, The image generation device includes: A vector determination module, configured to determine intermediate vector data for generating a target image set according to a pre-constructed text prompt word set and a noise image set, where the intermediate vector data includes reference vector data and target vector data; A first generation module, configured to generate a first target image based on the reference vector data; A second generation module, configured to fuse the reference vector data and the target vector data to generate a second target image, where the first target image and the second target image form a target image set corresponding to the style prompt words in the text prompt word set.

9. An electronic device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the image generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the image generation method according to any one of claims 1 to 7.

Images