Image generation method and device and electronic equipment

Abstract

The invention provides an image generation method and device and electronic equipment. The method comprises the following steps: acquiring a guide signal; wherein the guide signal comprises a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; extracting a first semantic feature of the text description signal, a second semantic feature of the reference image and an image visual feature of the reference image; performing feature fusion on the second semantic feature and the image visual feature to obtain a fusion feature corresponding to the reference image; extracting a relationship between the fusion feature and the first semantic feature to obtain a relationship feature; and based on the first semantic feature, the fusion feature and the relation feature, performing denoising processing on a preset noise image to obtain an output image. According to the mode, the content of the output image is more accurate and is highly matched with the intention of the user, so that the control flexibility and accuracy of image processing are improved, and the image processing quality is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image generation method, device and electronic equipment. BACKGROUND

[0002] In order to obtain an image meeting the expectation, it is usually necessary to process the image using multiple tools; for example, generating an initial image using an image generation tool, and then editing the initial image using at least one image editing tool. Due to poor compatibility between the multiple tools, problems such as image content loss or failure to edit according to the user's editing instructions are likely to occur, the control flexibility and accuracy of image processing are poor, and thus the image quality is low. Therefore, there is an urgent need in the field for a technical solution capable of overcoming the above-mentioned disadvantages of separate processing. SUMMARY

[0003] In view of the above, the purpose of the present application is to provide an image generation method, device and electronic equipment to improve the control flexibility and accuracy of image processing and improve the image processing quality.

[0004] In a first aspect, an embodiment of the present application provides an image generation method, which comprises: acquiring a guide signal; wherein the guide signal comprises a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; extracting a first semantic feature of the text description signal, a second semantic feature of the reference image and an image visual feature of the reference image; performing feature fusion on the second semantic feature and the image visual feature to obtain a fusion feature corresponding to the reference image; extracting a relationship between the fusion feature and the first semantic feature to obtain a relationship feature; and performing denoising processing on a preset noise image based on the first semantic feature, the fusion feature and the relationship feature to obtain an output image.

[0005] In a second aspect, an embodiment of the present application provides an image generation device, which comprises: a signal acquisition module configured to acquire a guide signal; wherein the guide signal comprises a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; a feature extraction module configured to extract a first semantic feature of the text description signal, a second semantic feature of the reference image and an image visual feature of the reference image; a feature fusion module configured to perform feature fusion on the second semantic feature and the image visual feature to obtain a fusion feature corresponding to the reference image; a relationship extraction module configured to extract a relationship between the fusion feature and the first semantic feature to obtain a relationship feature; and a denoising processing module configured to perform denoising processing on a preset noise image based on the first semantic feature, the fusion feature and the relationship feature to obtain an output image.

[0006] In a third aspect, an electronic device is provided, including a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the image processing method described above.

[0007] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing computer executable instructions, and when the computer executable instructions are invoked and executed by a processor, the computer executable instructions cause the processor to implement the video generation method described above.

[0008] The embodiments of the present application bring the following beneficial effects:

[0009] The image generation method, device and electronic device described above obtain a guidance signal, wherein the guidance signal includes a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; a first semantic feature of the text description signal, a second semantic feature of the reference image and an image visual feature of the reference image are extracted; the second semantic feature and the image visual feature are fused to obtain a fusion feature corresponding to the reference image; a relationship feature between the fusion feature and the first semantic feature is extracted; and a preset noise image is denoised based on the first semantic feature, the fusion feature and the relationship feature to obtain an output image.

[0010] In this way, the guidance signal obtained by combining the text description signal and the image indication signal is input into the model, the fusion feature is obtained by fusing the second semantic feature and the image visual feature extracted according to the image indication signal, the relationship feature is obtained by extracting the relationship between the first semantic feature extracted according to the text description signal and the fusion feature, and the denoising of the noise image is realized in the same model framework, so that the content of the output image is more accurate and has higher matching with the user's intention, thereby improving the control flexibility and accuracy of image processing and improving the image processing quality.

[0011] The present application is precisely aimed at providing such a unified technical framework, which realizes seamless and end-to-end processing of the two core tasks of instance-driven content generation and instruction-driven content modification in a single model architecture by introducing a novel composite guidance signal mechanism and a conditional content synthesis strategy. This not only simplifies the creation process and improves efficiency, but more importantly, it solves the core technical pain points of traditional methods in handling complex combination tasks (such as 'adding A with a specific appearance to scene C according to instruction B') in terms of balancing instance fidelity and instruction compliance through unified internal representation and fine control of multi-scale features.

[0012] Other features and advantages of the present application will be set forth in the descriptions that follow, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structures particularly pointed out in the description, claims and drawings.

[0013] In order to make the above objectives, features and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be referred to, as follows. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0015] Figure 1 A flow chart of an image generation method provided by an embodiment of the present application;

[0016] Figure 2 A flow chart of obtaining an output image according to an input condition provided by an embodiment of the present application;

[0017] Figure 3 A flow chart of a training method of a diffusion model provided by an embodiment of the present application;

[0018] Figure 4 A structural schematic diagram of an image generation device provided by an embodiment of the present application;

[0019] Figure 5 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0021] In order to obtain an image meeting the expectation, it is usually required to process the image by using multiple tools; for example, generating an initial image by using an image generation tool, and editing the initial image by using at least one image editing tool. Due to poor compatibility between the multiple tools, the edited image is prone to losing content or not being edited according to the editing instruction of the user, the control flexibility and accuracy of image processing are poor, and thus the image quality is low.

[0022] In the field of digital content creation, especially in the process of game development and video production, a user often needs to process an image by using a tool, for example, inputting one or more visual sample images (such as a game character, a prop or a scene style) into the tool to generate an initial image, or inputting an instruction (such as a textual description) into the tool to modify visual content in the image (such as changing the style of the image, adding or removing an object).

[0023] At present, in order to obtain an image meeting the expectation, it is usually required to process the image by using multiple tools; for example, generating an initial image by using an image generation tool, and editing the initial image by using at least one image editing tool. Due to poor compatibility between the multiple tools, the edited image is prone to losing content or not being edited according to the editing instruction of the user, the control flexibility and accuracy of image processing are poor, and thus the image quality is low.

[0024] Based on this, an image generation method, device and electronic equipment provided by an embodiment of the present application can be applied to the process of generating an image, for example, creating image content or editing original content of an image.

[0025] An image generation method disclosed by an embodiment of the present application comprises the following steps, as shown in the figure: Figure 1 The image generation method comprises the following steps:

[0026] In step S102, a guide signal is acquired; wherein the guide signal comprises a textual description signal and an image indication signal; the image indication signal corresponds to a preset reference image;

[0027] The guide signal can be mixed data provided by a user, and specifically can be in the form of a text string, such as Chinese, English, etc.; the guide signal is used to indicate image content of an output image to be generated. In the guide signal, the textual description signal and the image indication signal are two different signals associated with each other.

[0028] The text description signal can include a requirement for an output image to be generated, or a supplementary description of the image indication signal, etc.; for example, the text description signal is "add a role <role skin mark> in the scene <scene screenshot mark> to the central left position of the picture, and make it in a harp posture", wherein <scene screenshot mark> and <role skin mark> are both image indication signals.

[0029] The image indication signal corresponds to a preset reference image, for example, a user-provided role portrait image, prop image or scene screenshot. The number of reference images is usually consistent with the number of image indication signals, as in the foregoing example, <scene screenshot mark> can point to a preset high-resolution game screenshot of the Yuelun Palace, and <role skin mark> can point to a preset model rendering image of the "Moyun" role wearing the "Qingtujia Shi" skin, which includes complete clothing and role appearance.

[0030] The composite guidance signal adopted by the present application combines natural language text instructions with image indication signals pointing to specific visual data, giving users unprecedented flexibility and precision in control dimensions. Users are no longer limited to simple text descriptions or a single style reference, but can specify operation intent, creative theme and precise visual entities involved in one instruction. This rich input expression capability is the key basis for the present application to handle complex, fine-grained generation and editing tasks and ensure that the results are highly consistent with user expectations.

[0031] Step S104, extracting first semantic features of the text description signal, second semantic features of the reference image, and image visual features of the reference image;

[0032] The first semantic features of the text description signal can be extracted by a preset large language model, such as LLM (Large Language Model), or other natural language processing models; for example, input the text description signal of the foregoing example into the preset large language model for encoding to capture the semantics of the text description signal.

[0033] The reference image usually includes two types of features, of course, it can also be three types of features, four types of features, when the reference image includes two types of features, it is usually the second semantic feature and the image visual feature. At least two types of features can be extracted from each reference image corresponding to the image indication signal, in series or in parallel.

[0034] The second semantic feature of the reference image can be extracted by a semantic-level feature extractor. The second semantic feature mainly summarizes high-level and categorical semantic concepts of the reference image, such as “Gothic style” and “night atmosphere” for a building scene, or “male warrior” and “fantasy equipment” for a role. The semantic-level feature extractor can be a model based on a hierarchical visual transformer, such as ViT (Vision Transformer), or a query compression mechanism module, such as Q-Former (Querying Transformer).

[0035] The image visual feature of the reference image can be extracted by a detail-level feature extractor. The image visual feature mainly includes fine and low-to-medium-level visual appearance information of the reference image, such as accurate color distribution, surface texture details, local shape contour, and light and shadow effects. These features are crucial for accurately restoring the unique visual identity of the reference image. The detail-level feature extractor can be a pre-trained encoder VAE Encoder of a variational autoencoder (VAE). The encoder can better capture low-level and medium-level visual features of the reference image, such as texture, color distribution, and local shape.

[0036] In step S106, the second semantic feature and the image visual feature are fused to obtain a fusion feature corresponding to the reference image.

[0037] In this embodiment, the second semantic feature and the image visual feature extracted for each reference image are fused to obtain a multi-scale fusion feature.

[0038] The above feature fusion process can be implemented by an attention mechanism and a feedforward neural network. In the attention mechanism, the attention weight of the image visual feature can be calculated, which reflects the importance of the image visual feature in the context information of the current semantic. The feature output by the attention mechanism can be further processed and adjusted by a feedforward neural network (FFN), and then superimposed with the second semantic feature. In this way, the visual information that has been filtered and adjusted based on the second semantic feature is supplemented.

[0039] The above method can adaptively fuse the image visual feature according to the guidance of the second semantic feature, so that the generated fusion feature contains both macro semantic information and micro visual information related to the semantic information, avoiding the problem of losing visual details when only using semantic features.

[0040] Step S108, extracting the relationship between the fusion feature and the first semantic feature to obtain a relationship feature;

[0041] The relationship between the fusion feature and the first semantic feature can be extracted by a pre-set large language model or other natural language processing model. For example, after sorting and integrating the fusion feature and the first semantic feature, all or part of the large language model is input into the context-aware processing to obtain the relationship feature.

[0042] The above relationship feature is beneficial to the model to understand the mutual relationship between the text description signal and the image indication signal, for example, the position information in the reference image corresponding to the image indication signal. In one example, the pre-trained model can be a diffusion model.

[0043] Step S110, based on the first semantic feature, the fusion feature and the relationship feature, the pre-set noise image is denoised to obtain an output image.

[0044] The pre-trained model, such as the diffusion model, can be used to realize the denoising of the pre-set noise image. The denoising process usually includes multiple iterations of time steps, and the iteration denoising process starts from the initial time step t and iterates to t=0.

[0045] The above first semantic feature, fusion feature and relationship feature can obtain a latent space vector for representing latent features or attributes, wherein each vector represents a latent feature or attribute. The pre-set noise image can be determined according to the task type. For example, if the task type is a generation task, the pre-set noise image can be a tensor with the same dimension as the latent space vector and full of random noise. If the task type is an editing task, the pre-set noise image is obtained based on the noise state of the latent variable of the conditional image at an intermediate time step after being positively added.

[0046] In time step t, the noise image of the current time step, the embedding variable of the current time step t, the latent space vector and the latent variable are input into the pre-trained model, and the predicted noise to be removed is obtained through attention mechanism processing. This process combines the latent space representation corresponding to the guide signal and the task type corresponding to the input condition. Adding the initial noise to the latent variable can obtain the final predicted noise.

[0047] In the denoising process, a classifier-free guidance (CFG) can be used to guide the process. The predicted noise is calculated in both guided and unguided conditions, and then weighted to enhance the guidance effect.

[0048] The latent variable closer to the output image is gradually obtained from the noise image of the current time step using the calculation of the predicted noise and the diffusion scheduling coefficient, which can be understood as gradually "erasing" the noise and "drawing" the content according to the guide signal. The "erasing" process can be realized by the inverse sampling rule of the diffusion process. In an example, the calculation process including the predicted noise and the diffusion scheduling coefficient can be implemented in a denoising diffusion probabilistic model (DDPM).

[0049] The above denoising process is repeated in the next time step t-1 of the time step t until t=0, and finally the denoised latent variable is obtained. The denoised latent variable can be understood as a result representation synthesized or modified in the latent space by the model under all requirements.

[0050] The denoised latent variable can be input into a pre-trained decoder, which is usually a VAE Decoder matched with the aforementioned encoder VAE Encoder, to generate an output image in which the denoised latent variable is mapped in the pixel space. The output image can be a target image or a target video frame, which can be a new image or an edited image.

[0051] The output image obtained by the above method can not only meet the text instructions of the text description signal in the guide signal, but also maintain the features pointed to by the image indication signal in the reference image.

[0052] The above image generation method obtains a guide signal. The guide signal includes a text description signal and an image indication signal. The image indication signal corresponds to a preset reference image. A first semantic feature of the text description signal, a second semantic feature of the reference image, and an image visual feature of the reference image are extracted. The second semantic feature and the image visual feature are fused to obtain a fusion feature corresponding to the reference image. A relationship feature is obtained by extracting the relationship between the fusion feature and the first semantic feature. The first semantic feature, the fusion feature, and the relationship feature are used to denoise a preset noise image to obtain an output image.

[0053] In this way, the guide signal obtained by combining the text description signal and the image indication signal is input into the model. The fusion feature is obtained by fusing the second semantic feature and the image visual feature extracted from the image indication signal. The relationship feature is obtained by extracting the relationship between the first semantic feature extracted from the text description signal and the fusion feature. The denoising of the noise image is realized in the same model framework, so that the content of the output image is more accurate and has higher matching with the user's intention, thereby improving the control flexibility and accuracy of image processing and improving the image processing quality.

[0054] Specifically, the image indication signal comprises an image access interface of the reference image.

[0055] The image access interface can establish a link with the at least one reference image to access the at least one reference image in the process of image processing.

[0056] Each image indication signal can be associated with an externally provided reference image, such as a user-provided character portrait image, prop image, or scene screenshot, through the image access interface.

[0057] In an implementation manner, the second semantic feature and the image visual feature are processed by an attention mechanism to obtain an intermediate feature; and after the intermediate feature is processed by a preset feedforward neural network, the intermediate feature is superimposed with the second semantic feature to obtain a fusion feature corresponding to the reference image.

[0058] The attention mechanism processing of the second semantic feature and the image visual feature can be implemented by a scaled dot-product attention mechanism; the obtained intermediate feature can indicate important image visual features, and the intermediate feature can specifically include an attention weight of the image visual feature, which reflects the importance of the image visual feature in the context information of the current semantic to filter important visual information therefrom.

[0059] The intermediate feature is processed by a preset feedforward neural network, which can be a multi-layer perceptron (MLP), for further processing and adjusting the features output by the attention mechanism; and then the intermediate feature is superimposed with the second semantic feature to obtain a fusion feature corresponding to the reference image, and the superimposition process can be understood as a residual connection, so that the visual information that is filtered and adjusted by the semantic is supplemented on the basis of the second semantic feature.

[0060] The above manner can fuse the visual information indicated by the image visual feature into the basic semantic under the guidance of the second semantic feature.

[0061] In one manner, the second semantic feature is taken as query information of the attention mechanism, and the image visual feature is taken as key-value information of the attention mechanism, and the second semantic feature and the image visual feature are processed by the attention mechanism to obtain an intermediate feature.

[0062] For example, in the attention mechanism, the second semantic feature is taken as the query information Q, the image visual feature is taken as the key K information and the value V information, the dot product of the matrix Q (i.e., the second semantic feature) and the matrix K (i.e., the image visual feature) is first calculated to obtain a score matrix, the score matrix is scaled by dividing the score matrix by the square root of the dimension of the K (i.e., the image visual feature) vector, then the normalized exponential function softmax is used to calculate the scaled score matrix to obtain the weight of each matrix V (i.e., the image visual feature), and the weighted sum of the matrix V (i.e., the image visual feature) is obtained by using the weight to obtain the intermediate feature of the output.

[0063] The model in the related art mainly relies on macro semantic features extracted by CLIP (Contrastive Language-Image Pre-Training, a multi-modal pre-training neural network) and the like for guidance, but such methods often have difficulty in accurately preserving the appearance details of specific instances, and are prone to cause 'identity drift' or detail ambiguity in the generated or edited results.

[0064] Based on this, the multi-scale feature fusion mechanism proposed in the above embodiments of the application can greatly improve the detail fidelity and identity consistency of the reference visual instance while accurately following the text instructions by explicitly extracting and integrating the fine visual features (i.e., the image visual features) of the reference image and adaptively fusing them under the guidance of the macro semantic features (i.e., the second semantic features), thereby obtaining a visual content of higher quality and more in line with the user's fine expectations.

[0065] In one way, the first semantic feature and the fusion feature are sorted according to the signal ordering of the guidance signal, the text description signal and the image indication signal to obtain a sequence feature; the sequence feature is subjected to context-aware processing by a pre-set language model to output a final feature; wherein the final feature includes: the first semantic feature, the fusion feature and a relationship feature between the fusion feature and the first semantic feature; the final feature is used for denoising processing of a noisy image.

[0066] The first semantic feature extracted from the text description signal in the guidance signal and the fusion feature obtained by extracting and multi-scale fusion processing of the image indication signal can be sorted according to the signal ordering in the guidance signal to form a sequence, i.e., a sequence feature.

[0067] The aforementioned sequence features are input into a preset language model, such as a full or partial LLM large language model, for context-aware processing. This enables the model to understand the relationship between the text description signal and the reference image corresponding to the image indication signal. For example, the text description signal of the running action is for a specific character in the reference image, while the text description signal of style modification is for a specific scene in the reference image.

[0068] The pre-defined language model undergoes context-aware processing to output a final feature, which serves as a unified representation of the guiding latent space. This final feature not only includes individual textual semantic features (i.e., first semantic features) and visual fusion features, but more importantly, through the context-aware processing of the language model, it encodes the precise correspondence and interaction between the textual instruction (carried by the first semantic features) and a specific visual reference (carried by its fusion features). For example, if the textual instruction is 'make character <reference A> smile', the final feature can show that the action of 'smiling' is applied to the specific character instance specified by visual reference A. This final feature, which encodes complex contextual relationships, will be used to guide subsequent denoising processing of noisy images.

[0069] The aforementioned final features not only include the independent features of each signal, but more importantly, the final features contain the interrelationships between features to comprehensively guide the content of the generated image; for example, according to the aforementioned example, the final feature can be expressed as "generating a 'Moyun' character with precise appearance details of 'Qingzhu Yashi' in the center-left position of the Moon Pavilion scene (preserving its features), and making it present a zither-playing posture."

[0070] In one implementation, such as Figure 2 As shown, the process of denoising a preset noisy image to obtain an output image includes the following steps:

[0071] Step S202: Determine whether a conditional image has been obtained; this conditional image is usually provided by the user.

[0072] Step S204: If a conditional image is obtained, generate the first image features of the conditional image; generate prediction noise based on the first image features, the first semantic features, the fusion features, and the relational features;

[0073] The first image feature for generating the conditional image can be achieved through a pre-trained encoder, such as a VAE Encoder. If the conditional image is obtained, it can be input into the pre-trained encoder VAE Encoder to generate the latent space representation of the conditional image, i.e., the first image feature. This first image feature is a tensor in a preset format, which can contain the main structural and content information of the conditional image.

[0074] The process of generating the predicted noise can be implemented by a Transformer block inside the model using an attention mechanism. The attention mechanism can process the first image feature corresponding to the conditional image, and the first semantic feature, the fusion feature, and the relationship feature corresponding to the guide signal to generate the predicted noise.

[0075] In step S206, if the conditional image is not obtained, a second image feature corresponding to a preset base image is generated, and the predicted noise is generated based on the second image feature, the first semantic feature, the fusion feature, and the relationship feature. The preset base image is usually an image in a blank content state.

[0076] The second image feature of the preset base image can be generated by a pre-trained encoder, such as a VAE Encoder. If the conditional image is not obtained, the preset base image can be input into the pre-trained encoder VAE Encoder to generate a tensor with the same dimension as the latent space representation and all elements being zero, i.e., the second image feature, which is a zero-value tensor.

[0077] The process of generating the predicted noise can be implemented by a Transformer block inside the model using an attention mechanism. The attention mechanism can process the second image feature corresponding to the preset base image, and the first semantic feature, the fusion feature, and the relationship feature corresponding to the guide signal to generate the predicted noise.

[0078] In step S208, the predicted noise is removed from the preset noise image to obtain an output image.

[0079] After the iterative denoising process at multiple time steps, the predicted noise can be removed from the preset noise image to update the latent space representation of the noise image. Then, the updated latent space representation is input into the pre-trained decoder to obtain the output image.

[0080] For example, in the output image, the aforementioned Moon Wheel Palace scene can be completely preserved, the light is natural, and the "Ink Rhythm" character with the skin of "Green Bamboo Elegant" with highly restored clothing details is accurately added to the central left position of the picture and presents a natural piano posture, which is integrated with the environment.

[0081] The above method can perform the steps of iterative denoising, generating, or modifying image features under the constraints of the first semantic feature, the fusion feature, and the relationship feature, and the input condition, thereby improving the matching of the output image with the user's intention.

[0082] Specifically, the above step S204 includes: performing attention mechanism processing on the first image feature, the first semantic feature, the fusion feature, and the relationship feature to obtain the predicted noise.

[0083] The first image feature, the first semantic feature, the fusion feature, and the relationship feature can be processed through the cross-attention mechanism of the internal Transformer block of the model to obtain the predicted noise.

[0084] The cross-attention mechanism in the above manner can determine the weights of different features, reflect whether the features are important, and reflect the association between different modalities, thereby improving the accuracy and robustness of the model.

[0085] For example, the cross-attention mechanism guides the model to focus on the role information and posture instructions contained in the first semantic feature, the fusion feature, and the relationship feature, and the scene information in the first image feature, so that the model can evolve the latent variable under the guidance of the predicted noise to generate a role that contains the original scene information and meets the skin detail and posture requirements in the specified position.

[0086] In an implementation manner, the first semantic feature, the fusion feature, and the relationship feature are input into a pre-trained diffusion model to perform denoising processing on the preset noise image through the diffusion model to obtain an output image.

[0087] The first semantic feature, the fusion feature, and the relationship feature can be input into a pre-trained diffusion model; the pre-trained diffusion model can be a generator based on a diffusion model, for example, a diffusion model based on a Transformer module.

[0088] Through the above diffusion model, the preset noise image can be denoised through a plurality of time step iteration denoising processes to obtain an output image.

[0089] The above manner can integrate a plurality of independently completed image generation and image editing tasks into a pre-trained diffusion model, thereby simplifying the creation process and avoiding problems that occur when switching tools and converting data.

[0090] In an implementation manner, as shown in FIG. 2, the diffusion model is trained in the following manner: Figure 3

[0091] In step S302, sample data of the diffusion model is obtained; the sample data includes a sample guide signal, a sample condition image, and a sample target image.

[0092] In the training stage, the sample data of the diffusion model can be obtained according to data batches, and each batch (Batch) of sample data can include a sample guide signal, a sample condition image, and a sample target image.

[0093] ​The sample guiding signal can include a sample text description signal of training requirements and a sample image indication signal. The sample condition image can be distinguished according to the task type. If the task type is a generation task, the sample condition image is usually a zero-value tensor, which can represent a blank canvas. If the task type is an editing task, the sample condition image is usually a representation in the latent space after being encoded by the encoder VAE Encoder. The sample target image usually corresponds to an expected output image or a video frame.

[0094] In step S304, a noise latent variable is generated based on the sample target image and a preset actual noise.

[0095] The preset actual noise is usually a target learned and predicted by the model in the current iteration process. The noise latent variable can simulate the blurred state at the time step in the intermediate process in the iteration process.

[0096] First, the sample target image can be converted into a representation in the latent space by the encoder VAE Encoder to obtain a sample target feature. Then, the preset actual noise can be used to simulate the forward noise in the diffusion process. For example, a time step is randomly selected from the preset total time step T (such as T = 1000), and the preset actual noise is added to the sample target feature according to a noise scheduling rule to generate a noise latent variable.

[0097] It should be noted that for different time steps of the same iteration training or different iteration trainings, even if the sample target features are the same, the preset actual noises used are different. In the time steps after the current time step, the sample target feature will no longer be directly used as input.

[0098] In step S306, a prediction noise is generated based on the noise latent variable, the sample guiding signal, and the sample condition image.

[0099] In the training phase, for the sample guiding signal in the sample data, a sample first semantic feature of the sample text description signal and a sample second semantic feature and a sample image visual feature of the sample image indication signal are usually extracted. After multi-scale feature fusion, all feature information is integrated to obtain a sample final feature including the sample first semantic feature, the sample fusion feature, and the sample relationship feature.

[0100] The prediction noise can be generated by a specified module, such as a convolutional neural network U-Net or a Transformer based on deep learning.

[0101] The specified module generally receives the noise latent variable of the current time step, the embedding representation of the current time step t, the sample guide signal and the sample condition image, and obtains the noise added to the sample target feature to obtain the noise latent variable after a plurality of time steps, that is, the predicted noise.

[0102] In the time step t, the information of the noise latent variable of the current time step, the embedding representation of the current time step t, the sample guide signal and the sample condition image can be input into the diffusion model; wherein the embedding representation can indicate the specific stage of the model in the current diffusion process, the sample guide signal can indicate the current training target of the model, and the sample condition image can provide the starting position of the editing or the generated canvas state for the model.

[0103] In step S308, the noise loss between the predicted noise and the actual noise is calculated, and the model parameters of the diffusion model are adjusted through the noise loss.

[0104] The noise loss between the predicted noise and the actual noise can be calculated by a loss function, for example, by a mean square error (MSE) loss. The predicted noise and the actual noise can be input into the loss function to calculate the noise loss between the predicted noise and the actual noise, thereby quantifying the difference between the predicted noise and the actual noise. If the loss value is smaller, it means that the model prediction is more accurate.

[0105] The noise loss can be used to adjust the model parameters of the diffusion model, for example, the noise loss is backpropagated to calculate the gradient of the noise loss with respect to the model parameters of the diffusion model, which generally refers to the trainable part of the diffusion model, and the feature extractor is generally frozen. The model parameters of the diffusion model can be adjusted by an optimizer such as AdamW, so that the diffusion model can output predicted noise closer to the actual noise under such input conditions, thereby gradually reducing the loss.

[0106] The above process is a batch training step, that is, the process from input data to completion of weight update. This process will be repeated for a large number of batches in the training data set until the performance measured on the validation set reaches the expected level or no longer improves significantly. Through this cycle, the diffusion model can learn to accurately perform denoising operations under various guide signals and input conditions.

[0107] Specifically, the step S304 includes extracting sample image features of the sample target image, and sampling actual noise from a pre-set normal distribution noise and adding the actual noise to the sample image features to obtain the noise latent variable.

[0108] The sample image feature of the sample target image can be extracted by a pre-trained encoder, such as a VAE Encoder, and the sample image feature is a representation of the sample target image in the latent space.

[0109] In each iteration of the training, a time step is usually randomly selected, and actual noise is sampled from the preset normal distribution noise; the sampled actual noise is usually independent and randomly sampled in each time step of the iteration, that is, the actual noise obtained each time is usually different.

[0110] In one example, the above actual noise is added to the sample image feature to obtain a calculation formula of the noisy latent variable, which can also indicate the process of forward noise addition:

[0111]

[0112] wherein z_t represents the noisy latent variable, z_0 represents the sample image feature, denotes a diffusion scheduling coefficient, and ε denotes the actual noise sampled from the normal distribution noise N(0, I).

[0113] The above calculation can be understood that the time step does not change the sampling distribution of the actual noise, and mainly controls the degree of "amplification" or "mixing" of the actual noise into the sample image feature through the diffusion scheduling coefficient.

[0114] In one implementation, after the step S304, the method further includes: deleting, according to a preset probability, at least part of the sample guide signals and / or at least part of the sample condition images in the sample data.

[0115] That is, at least part of the sample guide signals in the sample data can be deleted, at least part of the sample condition images in the sample data can be deleted, or at least part of the sample guide signals and at least part of the sample condition images in the sample data can be deleted at the same time. The above preset probability can be the probability of pseudo-random number generation.

[0116] In one example, the diffusion model is trained in a non-classifier guided manner, so that the trained diffusion model learns both conditional generation and unconditional generation.

[0117] For example, during the training, all sample guide signals can be deleted; or sample guide signals of different types can be deleted according to a preset probability, so that the diffusion model learns to decouple different sample guide signals, thereby improving the robustness of the diffusion model.

[0118] When all sample condition images are deleted, a zero-value tensor, i.e., a blank canvas, can be obtained; when part of the sample condition images are deleted, the influence of the input condition on the diffusion model can be weakened, thereby guiding the model to accurately distinguish between the two modes of "active image condition" and "inactive image condition", and enhancing the adaptability and robustness of the diffusion model to different input conditions.

[0119] The above embodiments of the present application have the following advantages:

[0120] (1) Efficiency improvement and process simplification: integrating multiple independent tools to complete generation and editing tasks into one framework simplifies the content creation process, reduces the cost of tool switching and data conversion, and improves overall work efficiency.

[0121] (2) Consistency and fidelity enhancement: since different tasks are processed using a unified model, the visual style and feature consistency of reference objects (such as characters and props) can be better maintained; in particular, the multi-scale feature fusion mechanism ensures that the visual sample details and identity features are preserved to the maximum extent while following the instructions.

[0122] (3) Flexibility and control improvement: the composite guidance signal allows users to combine text instructions and visual references in a very flexible manner, enabling more precise and complex control; this enables the system to handle complex combined tasks, such as accurately adding or replacing specific visual instances in an image based on text instructions.

[0123] (4) Enable new creative possibilities: this embodiment is particularly suitable for game content creation scenarios that require rapid iteration and fine-tuning; for example, when making a game promo, a specific, player-defined character skin (visual reference) can be seamlessly integrated into a pre-set game scene screenshot, and its pose can be adjusted according to the text instructions (such as "make the character make a celebration gesture"), generating high-quality promo images or video clips while maintaining the high fidelity of the character skin details; this has wide application prospects in concept art design, marketing material production, and even procedural content generation.

[0124] A schematic diagram of an image processing device provided by an embodiment of the present application is shown in Figure 4 The device comprises:

[0125] The signal acquisition module 41 is configured to acquire a guidance signal; wherein the guidance signal comprises a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image;

[0126] The feature extraction module 42 is configured to extract a first semantic feature of the text description signal, a second semantic feature of the reference image, and an image visual feature of the reference image;

[0127] a feature fusion module 43 configured to perform feature fusion on the second semantic feature and the image visual feature to obtain a fusion feature corresponding to the reference image;

[0128] a relationship extraction module 44 configured to extract a relationship between the fusion feature and the first semantic feature to obtain a relationship feature;

[0129] a denoising processing module 45 configured to perform denoising processing on a preset noisy image based on the first semantic feature, the fusion feature and the relationship feature to obtain an output image.

[0130] The image generation apparatus described above acquires a guide signal; the guide signal includes a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; a first semantic feature of the text description signal, a second semantic feature of the reference image and an image visual feature of the reference image are extracted; the second semantic feature and the image visual feature are subjected to feature fusion to obtain a fusion feature corresponding to the reference image; a relationship between the fusion feature and the first semantic feature is extracted to obtain a relationship feature; and a preset noisy image is subjected to denoising processing based on the first semantic feature, the fusion feature and the relationship feature to obtain an output image.

[0131] In this mode, the guide signal obtained by combining the text description signal and the image indication signal is input into the model, the fusion feature is obtained by performing feature fusion on the second semantic feature and the image visual feature extracted according to the image indication signal, the relationship feature is obtained by performing relationship extraction on the first semantic feature extracted according to the text description signal and the fusion feature, and the denoising processing of the noisy image is realized in the same model framework, so that the content of the output image is more accurate and the matching with the user's intention is higher, thereby improving the control flexibility and accuracy of image processing and improving the image processing quality.

[0132] The image indication signal described above includes an image access interface of the reference image.

[0133] The feature fusion module described above is configured to perform attention mechanism processing on the second semantic feature and the image visual feature to obtain an intermediate feature; and perform feature superposition on the intermediate feature after being processed by a preset feedforward neural network and the second semantic feature to obtain the fusion feature corresponding to the reference image.

[0134] The feature fusion module described above is configured to take the second semantic feature as query information of the attention mechanism and take the image visual feature as key-value information of the attention mechanism, and perform attention mechanism processing on the second semantic feature and the image visual feature to obtain an intermediate feature.

[0135] The relationship extraction module is configured to perform feature sorting on the first semantic feature and the fusion feature according to signal ordering of the guidance signal, the text description signal and the image indication signal, to obtain sequence features; and perform context-aware processing on the sequence features through a preset language model, to output final features; wherein the final features include the first semantic feature, the fusion feature and a relationship feature between the fusion feature and the first semantic feature; and the final features are used for denoising processing of the noisy image.

[0136] The denoising processing module is configured to determine whether the conditional image is obtained; if the conditional image is obtained, generate a first image feature of the conditional image; generate a predicted noise based on the first image feature, the first semantic feature, the fusion feature and the relationship feature; if the conditional image is not obtained, generate a second image feature corresponding to a preset base image, and generate a predicted noise based on the second image feature, the first semantic feature, the fusion feature and the relationship feature; and remove the predicted noise from the preset noisy image to obtain an output image.

[0137] The denoising processing module is configured to perform attention mechanism processing on the first image feature, the first semantic feature, the fusion feature and the relationship feature to obtain the predicted noise.

[0138] The denoising processing module is configured to input the first semantic feature, the fusion feature and the relationship feature into a pre-trained diffusion model, to perform denoising processing on the preset noisy image through the diffusion model, and to obtain the output image.

[0139] The diffusion model is trained in the following manner: sample data of the diffusion model is obtained; wherein the sample data includes a sample guidance signal, a sample conditional image and a sample target image; a noise-ized latent variable is generated based on the sample target image and a preset actual noise; a predicted noise is generated based on the noise-ized latent variable, the sample guidance signal and the sample conditional image; a noise loss between the predicted noise and the actual noise is calculated, and a model parameter of the diffusion model is adjusted through the noise loss.

[0140] The device further includes an adding module configured to extract a sample image feature of the sample target image; sample the actual noise from a preset normal distribution noise, and add the actual noise to the sample image feature to obtain the noise-ized latent variable.

[0141] The device further includes a deleting module configured to delete at least part of the sample guidance signals and / or at least part of the sample conditional images in the sample data according to a preset probability.

[0142] The embodiment also provides an electronic device including a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the image generation method. The electronic device can be a server or a terminal device.

[0143] Referring to Figure 5 As shown in the figure, the electronic device includes a processor 100 and a memory 101 storing computer executable instructions executable by the processor 100, and the processor 100 executes the computer executable instructions to implement the image generation method described above.

[0144] Further, Figure 5 The electronic device shown also includes a bus 102 and a communication interface 103, and the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102.

[0145] Among them, the memory 101 can contain a high-speed random access memory (RAM, Random Access Memory), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 102 can be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 In the figure, only one bidirectional arrow is used to represent, but it does not mean that there is only one bus or one type of bus.

[0146] The processor 100 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 100 or the instruction in the form of software. The processor 100 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 101, and the processor 100 reads the information in the storage 101 and combines the hardware to complete the steps of the method of the above embodiment.

[0147] The processor in the above electronic device can realize the following operations in the above image generation method by executing computer executable instructions.

[0148] Obtain a guide signal; wherein the guide signal includes a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; extract the first semantic feature of the text description signal, the second semantic feature of the reference image and the image visual feature of the reference image; perform feature fusion on the second semantic feature and the image visual feature to obtain the fusion feature corresponding to the reference image; extract the relationship between the fusion feature and the first semantic feature to obtain the relationship feature; based on the first semantic feature, the fusion feature and the relationship feature, denoising processing is performed on the preset noise image to obtain an output image.

[0149] The image indication signal includes an image access interface of the reference image.

[0150] The second semantic feature and the image visual feature are processed by an attention mechanism to obtain an intermediate feature; and the intermediate feature is processed by a preset feedforward neural network and then superimposed with the second semantic feature to obtain a fusion feature corresponding to the reference image.

[0151] The second semantic feature is taken as query information of the attention mechanism, and the image visual feature is taken as key value information of the attention mechanism, and the second semantic feature and the image visual feature are processed by the attention mechanism to obtain an intermediate feature.

[0152] The first semantic feature and the fusion feature are sorted according to the signal order of the guidance signal text description signal and the image indication signal to obtain a sequence feature; and the sequence feature is processed by a preset language model for context perception to output a final feature; wherein the final feature includes the first semantic feature, the fusion feature and a relationship feature between the fusion feature and the first semantic feature; and the final feature is used for denoising processing of the noise image.

[0153] It is judged whether a condition image is obtained; if the condition image is obtained, a first image feature of the condition image is generated; a predicted noise is generated based on the first image feature, the first semantic feature, the fusion feature and the relationship feature; if the condition image is not obtained, a second image feature corresponding to a preset base image is generated, and a predicted noise is generated based on the second image feature, the first semantic feature, the fusion feature and the relationship feature; and the predicted noise is removed from a preset noise image to obtain an output image.

[0154] The first image feature, the first semantic feature, the fusion feature and the relationship feature are processed by an attention mechanism to obtain a predicted noise.

[0155] The first semantic feature, the fusion feature and the relationship feature are input into a diffusion model which is pre-trained to denoise a preset noise image by the diffusion model to obtain an output image.

[0156] The diffusion model is trained in the following manner: sample data of the diffusion model is obtained; wherein the sample data includes a sample guidance signal, a sample condition image and a sample target image; a noise latent variable is generated based on the sample target image and a preset actual noise; a predicted noise is generated based on the noise latent variable, the sample guidance signal and the sample condition image; a noise loss between the predicted noise and the actual noise is calculated, and the model parameters of the diffusion model are adjusted through the noise loss.

[0157] A sample image feature of the sample target image is extracted; the actual noise is sampled from a preset normal distribution noise and added to the sample image feature to obtain the noise latent variable.

[0158] According to a preset probability, at least part of the sample guide signals and / or at least part of the sample condition images in the sample data are deleted.

[0159] In the above manner, the guide signal obtained by combining the text description signal and the image indication signal is input into the model, the second semantic feature extracted according to the image indication signal and the image visual feature are subjected to feature fusion to obtain a fusion feature, the relationship feature is obtained by extracting the relationship between the first semantic feature extracted according to the text description signal and the fusion feature, and the denoising processing of the noise image is realized in the same model framework, so that the output image content is more accurate and has higher matching with the user's intention, thereby improving the control flexibility and accuracy of image processing and improving the image processing quality.

[0160] The embodiment also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are invoked and executed by a processor, the computer-executable instructions cause the processor to implement the above image generation method.

[0161] The computer-executable instructions stored in the above computer-readable storage medium can implement the following operations in the above image generation method by executing the computer-executable instructions:

[0162] obtaining a guide signal; wherein the guide signal comprises a text description signal and an image indication signal; the image indication signal corresponds to a preset reference image; extracting a first semantic feature of the text description signal, a second semantic feature of the reference image, and an image visual feature of the reference image; performing feature fusion on the second semantic feature and the image visual feature to obtain a fusion feature corresponding to the reference image; extracting the relationship between the fusion feature and the first semantic feature to obtain a relationship feature; and performing denoising processing on a preset noise image based on the first semantic feature, the fusion feature, and the relationship feature to obtain an output image.

[0163] The image indication signal comprises an image access interface of the reference image.

[0164] performing attention mechanism processing on the second semantic feature and the image visual feature to obtain an intermediate feature; and performing feature superposition on the intermediate feature after being processed by a preset feedforward neural network and the second semantic feature to obtain the fusion feature corresponding to the reference image.

[0165] performing attention mechanism processing on the second semantic feature and the image visual feature by taking the second semantic feature as query information of the attention mechanism and taking the image visual feature as key-value information of the attention mechanism to obtain an intermediate feature.

[0166] According to the signal order of the guidance signal text description signal and the image indication signal, the first semantic feature and the fusion feature are sorted to obtain a sequence feature; the sequence feature is subjected to context perception processing through a preset language model to output a final feature; wherein the final feature includes: the first semantic feature, the fusion feature and a relationship feature between the fusion feature and the first semantic feature; the final feature is used for denoising processing of the noise image.

[0167] It is judged whether the condition image is obtained; if the condition image is obtained, the first image feature of the condition image is generated; the prediction noise is generated based on the first image feature, the first semantic feature, the fusion feature and the relationship feature; if the condition image is not obtained, the second image feature corresponding to the preset basic image is generated, and the prediction noise is generated based on the second image feature, the first semantic feature, the fusion feature and the relationship feature; the prediction noise is removed from the preset noise image to obtain an output image.

[0168] The first image feature, the first semantic feature, the fusion feature and the relationship feature are subjected to attention mechanism processing to obtain the prediction noise.

[0169] The first semantic feature, the fusion feature and the relationship feature are input into a diffusion model trained in advance to denoise the preset noise image through the diffusion model to obtain an output image.

[0170] The diffusion model is trained in the following manner: sample data of the diffusion model is obtained; wherein the sample data includes: sample guidance signals, sample condition images and sample target images; a noise latent variable is generated based on the sample target images and a preset actual noise; a prediction noise is generated based on the noise latent variable, the sample guidance signals and the sample condition images; a noise loss between the prediction noise and the actual noise is calculated, and the model parameters of the diffusion model are adjusted through the noise loss.

[0171] Sample image features of the sample target images are extracted; the actual noise is sampled from a preset normal distribution noise and added to the sample image features to obtain the noise latent variable.

[0172] According to a preset probability, at least part of the sample guidance signals and / or at least part of the sample condition images in the sample data are deleted.

[0173] In the above manner, the guide signal obtained by combining the text description signal and the image indication signal is input into the model, the second semantic feature extracted according to the image indication signal and the image visual feature are fused to obtain a fusion feature, the relationship feature is extracted according to the first semantic feature extracted according to the text description signal and the fusion feature, the noise image is denoised in the same model framework, the output image content is more accurate and has higher matching with the user intention, and therefore the control flexibility and accuracy of image processing are improved, and the image processing quality is improved.

[0174] The image generation method and device and the computer program product of the electronic device provided in the embodiments of the present application include a computer readable storage medium storing program codes, the program codes include instructions for executing the method described in the foregoing method embodiments, and specific implementation can be referred to the method embodiments, which will not be described here.

[0175] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0176] In addition, in the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection" and "connection" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0177] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of software products, which are stored in a storage medium and include a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and various program code storage media.

[0178] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0179] Finally, it should be noted that the above embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, and are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent substitutions for some technical features; and these modifications, changes or substitutions do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image generation method, characterized in that, The method includes: Acquire guidance signals; wherein the guidance signals include: text description signals and image indication signals; the image indication signals correspond to a preset reference image; Extract the first semantic feature of the text description signal, the second semantic feature of the reference image, and the image visual feature of the reference image; The second semantic feature and the image visual feature are fused to obtain the fused feature corresponding to the reference image; Extract the relationship between the fused feature and the first semantic feature to obtain the relation feature; Based on the first semantic feature, the fusion feature, and the relational feature, the preset noisy image is denoised to obtain the output image.

2. The method according to claim 1, characterized in that, The image indication signal includes: the image access interface of the reference image.

3. The method according to claim 1, characterized in that, The step of fusing the second semantic feature and the image visual feature to obtain the fused feature corresponding to the reference image includes: The second semantic feature and the image visual feature are processed by an attention mechanism to obtain intermediate features; The intermediate features are processed by a preset feedforward neural network and then superimposed with the second semantic features to obtain the fused features corresponding to the reference image.

4. The method according to claim 3, characterized in that, The step of performing attention mechanism processing on the second semantic feature and the image visual feature to obtain intermediate features includes: The second semantic feature is used as the query information for the attention mechanism, and the image visual feature is used as the key value information for the attention mechanism. The attention mechanism is then applied to the second semantic feature and the image visual feature to obtain intermediate features.

5. The method according to claim 1, characterized in that, The step of extracting the relationship between the fused features and the first semantic features to obtain the relationship features includes: Based on the signal ordering of the text description signal and the image indication signal in the guidance signal, the first semantic feature and the fused feature are sorted to obtain the sequence feature; The sequence features are subjected to context-aware processing using a preset language model to output final features; wherein, the final features include: the first semantic feature, the fusion feature, and the relationship feature between the fusion feature and the first semantic feature; the final features are used to denoise the noisy image.

6. The method according to claim 1, characterized in that, The step of denoising a preset noisy image based on the first semantic feature, the fusion feature, and the relational feature to obtain an output image includes: Determine whether the conditional image has been obtained; If the conditional image is obtained, a first image feature of the conditional image is generated; prediction noise is generated based on the first image feature, the first semantic feature, the fusion feature, and the relational feature; If the conditional image is not obtained, a second image feature corresponding to the preset base image is generated, and prediction noise is generated based on the second image feature, the first semantic feature, the fusion feature, and the relational feature; The predicted noise is removed from the preset noisy image to obtain the output image.

7. The method according to claim 6, characterized in that, The step of generating prediction noise based on the first image features, the first semantic features, the fusion features, and the relation features includes: The first image feature, the first semantic feature, the fusion feature, and the relation feature are processed by an attention mechanism to obtain prediction noise.

8. The method according to claim 1, characterized in that, The step of denoising a preset noisy image based on the first semantic feature, the fusion feature, and the relational feature to obtain an output image includes: The first semantic feature, the fusion feature, and the relational feature are input into a pre-trained diffusion model to denoise a preset noisy image and obtain an output image.

9. The method according to claim 8, characterized in that, The diffusion model was trained in the following manner: Obtain sample data for the diffusion model; wherein the sample data includes: sample guidance signal, sample condition image, and sample target image; Based on the sample target image and the preset actual noise, generate noise latent variables; Based on the noisy latent variable, the sample guidance signal, and the sample conditional image, predictive noise is generated; Calculate the noise loss between the predicted noise and the actual noise, and adjust the model parameters of the diffusion model based on the noise loss.

10. The method according to claim 9, characterized in that, The step of generating noisy latent variables based on the sample target image and the preset actual noise includes: Extract the sample image features from the target image; Actual noise is sampled from a preset normal distribution noise, and the actual noise is added to the features of the sample image to obtain a noise-enhanced latent variable.

11. The method according to claim 9, characterized in that, After the step of obtaining sample data for the diffusion model, the method further includes: According to a preset probability, at least a portion of the sample guidance signals and / or at least a portion of the sample condition images in the sample data are deleted.

12. An image generation apparatus, characterized in that, The device includes: A signal acquisition module is used to acquire guidance signals; wherein, the guidance signals include: text description signals and image indication signals; the image indication signals correspond to a preset reference image; The feature extraction module is used to extract the first semantic features of the text description signal, the second semantic features of the reference image, and the image visual features of the reference image; The feature fusion module is used to fuse the second semantic feature and the image visual feature to obtain the fused feature corresponding to the reference image; The relationship extraction module is used to extract the relationship between the fused feature and the first semantic feature to obtain the relationship feature; The denoising module is used to denoise a preset noisy image based on the first semantic feature, the fusion feature, and the relational feature to obtain an output image.

13. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the image generation method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the image generation method according to any one of claims 1-11.

Images