Image generation method and device, equipment, storage medium and product

Abstract

The invention relates to an image generation method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: performing content extraction on an input reference image through a content extraction module to obtain an image feature vector and a text feature vector; mapping the image feature vector and the text feature vector into a reference image feature through a feature adaptation module; injecting the reference image features into an image generation model; and based on the reference image features, performing image generation processing on the image generation instruction through an image generation model to obtain a target generated image. According to the method, the target image is generated based on the content features of the reference image, so that the style of the generated image is adjusted, and individual requirements of different application scenes on image generation are met. And the image feature vector and the text feature vector are converted into reference image features adaptive to the corresponding image generation model through a feature adaptation module, so that the adaptability with different types of image generation models is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to an image generation method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the development of Artificial General Intelligence (AGI) technology, AI products and functions based on text-to-image generation are becoming increasingly common. These models typically utilize pre-trained language decoders to convert textual prompts into latent representations to guide the diffusion process of generating or editing images. Most image generation models leverage text-guided diffusion architectures and CLIP (Contrastive Language-Image Pre-training) guidance to enhance the fidelity and relevance of generated images. However, existing image generation models still struggle to meet users' needs for personalized images. When generating images with different styles, fine-tuning is often required for each new reference image. Furthermore, different image generation models require independently training corresponding reference image feature extraction modules, a process that consumes significant computational resources. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image generation method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to a first aspect of the present disclosure, an image generation method is provided, the method comprising: extracting content from an input reference image using a content extraction module to obtain an image feature vector and a text feature vector; mapping the image feature vector and the text feature vector to reference image features adapted to an image generation model using a feature adaptation module; injecting the reference image features into the image generation model; inputting an image generation instruction into the image generation model; and performing image generation processing on the image generation instruction using the image generation model based on the reference image features to obtain a target generated image.

[0005] In some exemplary embodiments of this disclosure, the step of extracting content from the reference image using a content extraction module to obtain image feature vectors and text feature vectors includes: performing image analysis on the reference image using a segmentation model to obtain local image labeling information; the local image labeling information is used to label at least one local region in the reference image; extracting text features from the reference image using a large language model to obtain a global text description; extracting text features from the reference image based on the local image labeling information using the large language model to obtain at least one local text description; performing text encoding processing on the global text description and at least one local text description using a text encoder to obtain the text feature vector; determining at least one local image based on the local image labeling information and the reference image; and performing image encoding processing on the reference image and at least one local image using an image encoder to obtain the image feature vector.

[0006] In some exemplary embodiments of this disclosure, the large language model extracts text features from the reference image based on at least one pre-set set of target cues.

[0007] In some exemplary embodiments of this disclosure, the step of extracting text features from the reference image based on the image local labeling information and using the large language model to obtain at least one local text description further includes: sorting the at least one local text description according to the area of ​​the local region corresponding to the local text description.

[0008] In some exemplary embodiments of this disclosure, the step of performing text encoding processing on the global text description and at least one local text description using a text encoder to obtain the text feature vector includes: performing text encoding processing on the global text description using the text encoder to obtain a global text feature vector; performing text encoding processing on the at least one local text description using the text encoder to obtain at least one local text feature vector; the local text feature vectors correspond to the local text descriptions respectively; and concatenating the global text feature vector with the at least one local text feature vector to obtain the text feature vector.

[0009] In some exemplary embodiments of this disclosure, concatenating the global text feature vector with at least one local text feature vector to obtain the text feature vector includes: concatenating at least two of the global text feature vectors with the at least one local text feature vector to obtain the text feature vector.

[0010] In some exemplary embodiments of this disclosure, the step of performing image encoding processing on the reference image and at least one local image through an image encoder to obtain the image feature vector includes: performing image encoding processing on the reference image and at least one local image through the image encoder to obtain a first image feature vector; injecting the text feature vector into a feature fusion model; and performing feature fusion on the first image feature vector and the text feature vector through the feature fusion model to obtain the image feature vector.

[0011] In some exemplary embodiments of this disclosure, injecting the text feature vector into the feature fusion model includes: fusing the text feature vector through at least two attention layers to obtain a first text feature vector; and injecting the first text feature vector into the feature fusion model after resizing it through at least one fully connected layer.

[0012] In some exemplary embodiments of this disclosure, the step of fusing the first image feature vector and the text feature vector through the feature fusion model to obtain the image feature vector includes: fusing the first image feature vector and the text feature vector through the feature fusion model to obtain a second image feature vector; downsampling the text feature vector to obtain a second text feature vector; and fusing the second image feature vector and the second text feature vector through at least two cross-attention layers to obtain the image feature vector.

[0013] In some exemplary embodiments of this disclosure, the step of mapping the image feature vector and text feature vector to the reference image feature adapted to the image generation model through the feature adaptation module includes: adjusting the scale of the image feature vector through the first feature mapping module according to the feature scale of the image generation model to obtain the first image feature; and injecting the first image feature into the image generation model.

[0014] In some exemplary embodiments of this disclosure, the step of mapping the image feature vector and text feature vector to the reference image features adapted to the image generation model through the feature adaptation module includes: adjusting the scale of the image feature vector according to the feature scale of the image generation model through the second feature mapping module to obtain a second image feature; adjusting the scale of the second image feature according to the feature scale of each intermediate layer in the image generation model through the third feature mapping module to obtain at least one third image feature; the third image feature corresponds to the intermediate layer respectively; and injecting the third image feature into the corresponding intermediate layer in the image generation model respectively.

[0015] In some exemplary embodiments of this disclosure, the step of mapping the image feature vector and text feature vector to the reference image feature adapted to the image generation model through the feature adaptation module further includes: concatenating the at least one third image feature with the text feature vector to obtain text features; and injecting the text features into the image generation model.

[0016] According to a second aspect of the present disclosure, an image generation apparatus is provided, comprising: a content extraction unit, configured to extract content from an input reference image using a content extraction module to obtain an image feature vector and a text feature vector; a feature adaptation unit, configured to map the image feature vector and the text feature vector to reference image features adapted to an image generation model using the feature adaptation module; a reference image feature injection unit, configured to inject the reference image features into the image generation model; an image generation instruction input unit, configured to input an image generation instruction into the image generation model; and an image generation unit, configured to perform image generation processing on the image generation instruction based on the reference image features using the image generation model to obtain a target generated image.

[0017] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement any of the image generation methods described herein.

[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform any of the image generation methods described herein.

[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the image generation methods described above.

[0020] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0021] The image generation method disclosed herein extracts content from an input reference image using a content extraction module to obtain image feature vectors and text feature vectors. A feature adaptation module maps these image feature vectors and text feature vectors to reference image features adapted to an image generation model. Based on these reference image features, the image generation model processes the image generation instructions to obtain the target generated image. This embodiment of the disclosure generates the target image based on the content features of the reference image, thereby adjusting the style of the generated image to meet the personalized image generation needs of different application scenarios and expanding the applicability of the image generation model. Simultaneously, the feature adaptation module transforms the image feature vectors and text feature vectors into reference image features adapted to the corresponding image generation model, thereby improving compatibility with different types of image generation models.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0024] Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment of the present disclosure.

[0025] Figure 2 This is a schematic diagram illustrating the application process of an image generation method according to an exemplary embodiment of the present disclosure.

[0026] Figure 3 This is a flowchart illustrating a reference image feature vector extraction method according to an exemplary embodiment of the present disclosure.

[0027] Figure 4 This is a flowchart illustrating a text feature vector extraction method according to an exemplary embodiment of the present disclosure.

[0028] Figure 5 This is a flowchart illustrating an image feature vector extraction method according to an exemplary embodiment of the present disclosure.

[0029] Figure 6 This is a schematic diagram of the content extraction module framework according to an exemplary embodiment of the present disclosure.

[0030] Figure 7 This is a schematic diagram of a feature adaptation module framework according to an exemplary embodiment of the present disclosure.

[0031] Figure 8 This is a block diagram illustrating an image generation apparatus according to an exemplary embodiment of the present disclosure.

[0032] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0033] Exemplary embodiments of this disclosure will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0034] The embodiments described below, which are examples of some of the embodiments of this disclosure, do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] In related technologies, most image generation models utilize text-guided diffusion architectures and CLIP guidance to enhance the fidelity and relevance of generated images, and introduce UNet models and text encoders to improve the influence of text on images. However, existing image generation models often require fine-tuning for each new reference image, or even retraining the model, when generating images with different styles. Furthermore, different reference image feature extraction modules need to be trained independently for different image generation models, a process that consumes significant computational resources. Additionally, transferred image features are almost entirely incompatible with target cues. This makes it difficult to strike a balance between cue fidelity and image fidelity.

[0036] The steps of the method in the exemplary embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings and examples.

[0037] Figure 1 This is a flowchart illustrating an image generation method according to an exemplary embodiment of the present disclosure. Figure 2 This is a schematic diagram illustrating the application process of an image generation method according to an exemplary embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices, including smart speakers, smartphones, and smart tablet terminal devices, and may also include server-side components such as local servers and cloud servers, which can be deployed in a computer or a computer cluster consisting of multiple computers.

[0038] like Figure 1 , 2 As shown, in some embodiments, the image generation method of this disclosure includes:

[0039] In step S110, the content extraction module extracts the content of the input reference image to obtain image feature vector and text feature vector.

[0040] In this embodiment, a reference image is first obtained. This reference image serves as a template for image generation, allowing the image generation model to refer to it when generating images, aiming to produce images that conform to the desired image style. In this way, different types of images can be generated by inputting different reference images without changing the core model of the image generation model. By inputting different reference images into the image generation model, the model can meet the needs of different application scenarios and generate the images required for those scenarios.

[0041] In this embodiment of the disclosure, such as Figure 2 As shown, this content extraction module is used to extract content from the reference image, obtaining its feature vector. To effectively extract image content, this module employs a multimodal approach to model the image content from different angles and stages, constructing image understanding from various perspectives through multimodal methods. Specifically, this multimodality can include image and text modalities, resulting in image embeddings and text embeddings for the reference image. The text features are features extracted from the reference image and represented in text form. These text features can better guide the image generation model to complete the image generation task and effectively enable interaction with task objective prompts. Furthermore, this content extraction module can also extract content from the reference image at different granularities. For example, it can extract content from the global image of the reference image to represent its global features; or it can extract content from a local image of the reference image to represent its local features.

[0042] In step S120, the image feature vector and text feature vector are mapped to the reference image features that are adapted to the image generation model by the feature adaptation module.

[0043] In this embodiment, since most mainstream image generation models, such as the SD model and Flux model, are based on the Transformer architecture, the main difference between different types of image generation models lies in their feature scales. Therefore, to ensure that the reference image features output by the feature extraction module are adaptable to different types of image generation models, a feature adaptation module is used to map and adjust the image feature vector and text feature vector so that the adjusted image feature vector and text feature vector can be adapted to the feature scale requirements of the image generation model. By setting this feature adaptation module, the extracted image feature vector and text feature vector can be transformed into a feature scale adapted to the corresponding image generation model, thereby improving the adaptability of the feature extraction module and achieving a plug-and-play effect.

[0044] The image generation method disclosed herein extracts image feature vectors and text feature vectors from a reference image through a content extraction module, and adapts the extracted features to the image generation model through a feature adaptation module. This method extracts content from the reference image from both image and text modalities in a multimodal manner, enriching the expression of reference image features. Simultaneously, the feature adaptation module transforms the image feature vectors and text feature vectors into feature scales adapted to the corresponding image generation model, thereby improving the adaptability of the feature extraction module and achieving a plug-and-play effect.

[0045] In an exemplary embodiment, such as Figure 2 As shown, the content extraction module and the feature adaptation module are both located within the feature extraction module. This feature extraction module can be obtained through pre-training. The pre-trained feature extraction module extracts features from the input reference image to obtain reference image features. These reference image features can be used to characterize the texture, style, content, structural, and / or detail features of the reference image. By extracting relevant reference image features from the reference image, the image generation model can generate images based on these relevant reference image features.

[0046] In step S130, the reference image features are injected into the image generation model.

[0047] In this embodiment of the disclosure, the aforementioned reference image features are injected into a pre-trained image generation model. By injecting the reference image features, the matrix parameters in the image generation model are adjusted, so that the general image generation model can adjust the style of image generation according to the reference image features to adapt to the needs of different application scenarios.

[0048] In exemplary embodiments, the image generation model can be modeled based on various large-scale image or video generation frameworks. Examples include the SD series, SDXL series, Flux series, and DiT series. The feature extraction module designed in this disclosure can be adapted to different types of image generation model frameworks, therefore, the specific modeling framework of the image generation model is not limited.

[0049] In step S140, the image generation instruction is input into the image generation model.

[0050] In this embodiment of the disclosure, an image generation instruction input by a user is received. This image generation instruction is used to instruct an image generation model to generate a corresponding image. The image generation instruction may include a textual description of the image to be generated, or it may include the original image of the image to be generated. The image generation instruction is input into the image generation model so that the image generation model generates the corresponding target image based on the image generation instruction.

[0051] In step S150, based on the features of the reference image, the image generation instruction is processed by the image generation model to obtain the target generated image.

[0052] In this embodiment of the disclosure, the image generation model executes the image generation task corresponding to the image generation instruction based on the reference image features, to obtain a target generated image based on the reference image features and the image generation instruction. The target generated image is generated according to the image generation instruction. Simultaneously, the image style of the target generated image references the reference image features. The image generation model can adjust the image generation style according to the reference image features, thereby generating a target generated image that conforms to the style of the reference image.

[0053] In an exemplary embodiment, this image generation method can be applied to photo editing scenarios. By adding a reference image to a photo, users can quickly complete photo editing and optimization to make the artistic style of the photo closely resemble the reference image, thereby improving the quality of the photo.

[0054] In an exemplary embodiment, this image generation method can be applied to video editing scenarios. By adding reference images to the video, users can be helped to transform the video into an animation or art style video with a specific aesthetic, thereby improving the video's quality.

[0055] In an exemplary embodiment, this image generation method can be applied to e-commerce scenarios. By adding reference images, e-commerce platforms can generate high-quality product images and advertising posters related to the reference images, thereby improving product sales and promotional effectiveness.

[0056] In an exemplary embodiment, this image generation method can be applied to advertising scenarios. By adding reference images, product images or advertising videos with a consistent style can be quickly generated, making the relevant product images or advertising videos more attractive and creative, thereby improving the effectiveness of the advertisement and the user experience.

[0057] The image generation method disclosed herein extracts content from an input reference image using a content extraction module to obtain image feature vectors and text feature vectors. A feature adaptation module maps these image feature vectors and text feature vectors to reference image features adapted to an image generation model. An image generation instruction is input into the image generation model. Based on the reference image features, the image generation model processes the image generation instruction to obtain the target generated image. This embodiment of the disclosure generates the target image based on the content features of a reference image, thereby adjusting the style of the generated image to meet the personalized image generation needs of different application scenarios and expanding the applicability of the image generation model. Simultaneously, the feature adaptation module transforms the image feature vectors and text feature vectors into reference image features adapted to the corresponding image generation model, thereby improving compatibility with different types of image generation models.

[0058] Figure 3 This is a flowchart illustrating a reference image feature vector extraction method according to an exemplary embodiment of the present disclosure. Figure 6 This is a schematic diagram of the content extraction module framework according to an exemplary embodiment of the present disclosure.

[0059] like Figure 3 , 6 As shown, in Figure 1 Based on the image generation method shown, step S110 may include the following steps.

[0060] In step S310, the reference image is analyzed by a segmentation model to obtain local image labeling information; the local image labeling information is used to label at least one local region in the reference image.

[0061] In this embodiment of the disclosure, a segmentation model is used to analyze the reference image, thereby labeling different parts of the image and obtaining local image labeling information. The segmentation model is pre-trained to analyze and determine the region category to which each pixel in the reference image belongs, and labels each pixel with a label value corresponding to that region category. This label value corresponds to the region category and is used to identify the local region range of that category. For example... Figure 7As shown, based on local image labeling information, the subject and background in the reference image can be labeled. Furthermore, different parts of the subject can be labeled differently. This local image labeling information allows for subsequent differential feature extraction of different local regions of the reference image, thereby capturing more detailed features of local regions while avoiding the influence of non-focused local regions. For example, the image style may focus only on the content of the head region, without needing the content of the hair region.

[0062] In step S320, text features are extracted from the reference image using a large language model to obtain a global text description.

[0063] In this embodiment, a Large Language Model (LLM) is used to extract text features from the reference image to obtain a global text description, denoted as Text (Full pic). The LLM model is a deep learning-based natural language processing model with a large number of parameters and a complex computational structure, capable of learning the syntax, semantics, and contextual relationships of natural language. This LLM model can be a multimodal LLM model, capable of processing and understanding different modalities such as images, videos, and audio content in addition to text. Through this multimodal LLM model, features that can be understood as text can be extracted from the reference image, thereby generating textual description information based on the global content of the reference image.

[0064] In an exemplary embodiment, the LLM model has at least one set of pre-set target cues. The LLM model can extract text features from a reference image based on these multiple sets of target cues. The target cues are pre-set prompts indicating the content extraction required by the LLM model. For example, the target cues could be for identifying the global structure, content, scene, elements, etc., of the image. The target cues could also be for describing the main content and details of the image, and how to reconstruct the image through textual description. By setting different target cues, the direction of content extraction by the LLM model can be pre-intervened.

[0065] In an exemplary embodiment, the LLM model can employ coarse-grained content feature extraction. For example, the granularity can be set to Int4. Coarse-grained content feature extraction captures the main features of the image, discarding irrelevant details and redundant model representations. Simultaneously, it reduces the computational load of image processing and improves content extraction efficiency. Furthermore, the granularity used by the aforementioned segmentation model in image analysis can be consistent with the coarse-grained granularity used by the LLM model to avoid wasting unnecessary computation.

[0066] In an exemplary embodiment, the existing LLM model framework can be simplified by removing the BN (Batch Normalization Layer), FN (Feedforward Layer), and some activation function layers. Simultaneously, the dropout rate can be increased by 30%, and the entire model size is further compressed using Int4 quantization. These simplifications significantly improve the overall computational efficiency of the LLM model.

[0067] In step S330, based on the local image labeling information, text features are extracted from the reference image using the large language model to obtain at least one local text description.

[0068] In this embodiment of the disclosure, based on the image local labeling information obtained by the aforementioned segmentation model, local text features can also be extracted from local regions in the reference image using the LLM model to obtain at least one local text description, denoted as Seg Text (Seg pic). Each local text description corresponds to a different local region labeled by the image local labeling information. The related extraction process is similar to the aforementioned step S320 and will not be repeated here.

[0069] It should be noted that, based on the different content focuses of global and local features, different target cues can be pre-set for the LLM model. When the LLM model extracts text features, it can perform feature extraction based on these different target cues to satisfy the differences between global and local feature extraction.

[0070] In an exemplary embodiment, before the LLM performs local text feature extraction on a local region of the reference image, the local region can be filtered according to the style settings of the reference image. Local regions that are not of interest to the style settings can be filtered to avoid feature extraction on related uninterested local regions. Relevant local text feature extraction is performed only on local regions that are of interest to the style settings. For example, the background region in the reference image can be filtered out, and feature extraction can be performed only on the main body of the image.

[0071] In an exemplary embodiment, this disclosure may also sort the multiple local text descriptions according to the area of ​​the local region corresponding to the local text description, thereby increasing the feature priority of the local text description corresponding to the local region with a larger area.

[0072] In step S340, the global text description and at least one local text description are processed by a text encoder to obtain the text feature vector.

[0073] In this embodiment of the disclosure, the global text description and at least one local text description obtained in the aforementioned steps S320 and S330 are input into a text encoder. The text encoder performs text encoding processing on the global text description and at least one local text description to obtain a text feature vector (text embedding).

[0074] In an exemplary embodiment, the text encoder may employ a CLIP-based text encoder. This CLIP text encoder performs Int8 quantization and removes the last layer from the existing model framework, thus ensuring generalization while appropriately reducing computational load and optimizing the encoding efficiency of the text encoder.

[0075] In step S350, at least one local image is determined based on the image local marker information and the reference image.

[0076] In this embodiment of the disclosure, the reference image is segmented into multiple local images based on the local image labeling information obtained from the aforementioned segmentation model. Specifically, a mask image corresponding to different local regions can be generated based on the local image labeling information, and each local image corresponding to the local region labeled by the local image labeling information can be obtained through the mask image.

[0077] In step S360, the reference image and at least one local image are image encoded by an image encoder to obtain the image feature vector.

[0078] In this embodiment of the disclosure, the reference image and at least one local image obtained in step S350 are input into an image encoder. The image encoder performs image encoding processing on the reference image and the at least one local image to obtain an image feature vector (Image Embedding).

[0079] In an exemplary embodiment, the image encoder may employ a CLIP-based image encoder. For a reference image, the CLIP image encoder is used to encode the global image features of the reference image. Simultaneously, to ensure the modeling of local region image features, the CLIP image encoder is also used to encode each local image, obtaining local image features corresponding to the local region. These global image features and at least one local image feature together constitute the image feature vector.

[0080] In an exemplary embodiment, the CILP image encoder is subjected to Int8 quantization, and the last two layers of parameters in the existing model framework are removed, thereby optimizing the encoding efficiency of the image encoder.

[0081] In this embodiment, a text feature vector is generated by a text encoder and an image feature vector is generated by an image encoder. These image and text feature vectors extract the content of the reference image based on two different modalities: image features and text features. These are then processed by subsequent modules to obtain reference image features for injection into the image generation model.

[0082] Figure 4 This is a flowchart illustrating a text feature vector extraction method according to an exemplary embodiment of the present disclosure. Figure 6 This is a schematic diagram of the content extraction module framework according to an exemplary embodiment of the present disclosure.

[0083] like Figure 4 , 6 As shown, in Figure 3 Based on the image generation method shown, step S340 may include the following steps.

[0084] In step S410, the global text description is processed by the text encoder to obtain a global text feature vector.

[0085] In this embodiment of the disclosure, for the global content of the reference image, a text encoder is used to perform text encoding processing on the global text description to obtain the global text feature vector of the reference image.

[0086] In step S420, the text encoder performs text encoding processing on the at least one local text description to obtain at least one local text feature vector; the local text feature vector corresponds to the local text description respectively.

[0087] In this embodiment of the disclosure, in order to ensure the modeling of local region text features, for the local region content of the reference image, a text encoder is used to perform text encoding processing on at least one local text description to obtain at least one local text feature vector. Each local text feature vector corresponds to a local text description, and thus corresponds to the local region marked by the segmentation model.

[0088] In step S430, the global text feature vector is concatenated with at least one local text feature vector to obtain the text feature vector.

[0089] In this embodiment of the disclosure, the aforementioned global text feature vector and at least one local text feature vector together constitute the text feature vector of the reference image. For example... Figure 6 As shown, this disclosure concatenates the global text feature vector with at least one local text feature vector to form the text feature vector for further processing by subsequent modules.

[0090] In an exemplary embodiment, to avoid the model becoming biased towards a particular local feature due to an excessive number of local text feature vectors in subsequent modeling, this disclosure adds weights to the global text feature vector during the concatenation process to enhance the effectiveness of global information. Specifically, at least two of the global text feature vectors are concatenated with at least one local text feature vector to obtain the text feature vector. For example... Figure 6 As shown, during the splicing process, the dimensions of the data channel are expanded. The global text feature vector is placed at both ends of the text feature vector, and the multiple local text feature vectors are placed in the middle according to the aforementioned arrangement. The global text feature vector and the multiple local text feature vectors are spliced ​​together in the above form to form the text feature vector.

[0091] In an exemplary embodiment, such as Figure 6 As shown, the text feature vector can be further downsampled (DS) to adjust the data sampling rate and reduce the feature dimension, thereby improving data processing efficiency.

[0092] Figure 5 This is a flowchart illustrating an image feature vector extraction method according to an exemplary embodiment of the present disclosure. Figure 6 This is a schematic diagram of the content extraction module framework according to an exemplary embodiment of the present disclosure.

[0093] like Figure 5 , 6 As shown, in Figure 3 Based on the image generation method shown, step S360 may include the following steps.

[0094] In step S510, the reference image and at least one local image are image encoded by the image encoder to obtain a first image feature vector.

[0095] In this embodiment of the disclosure, an image encoder performs image encoding processing on a reference image and at least one local image to obtain a first image feature vector. This encoding process is similar to the aforementioned step S360 and will not be described again here.

[0096] In step S520, the text feature vector is injected into the feature fusion model.

[0097] In this embodiment, a text encoder and an image encoder extract features from the content of a reference image based on two different modalities: text feature vectors and image feature vectors, respectively. To achieve effective fusion of the text and image feature vectors, a feature fusion model is also designed to integrate the features from the two modalities.

[0098] In this embodiment of the disclosure, the text feature vector output by the text encoder and the first image feature vector output by the image encoder are respectively input into the feature fusion model.

[0099] In an exemplary embodiment, in order to achieve effective fusion of text features from different perspectives, remove redundancy, and ensure the accuracy of text information, this disclosure preprocesses the text feature vector before injecting it into the feature fusion model.

[0100] like Figure 6 As shown, firstly, the text feature vector output by the text encoder is fused through at least two attention layers to obtain the first text feature vector. This attention layer, when processing sequential or multimodal data, enables the model to focus on key parts of the input data, thereby improving the model's sensitivity to relevant information and reducing interference from irrelevant information. By passing the text feature vector through multiple attention layers, more valuable features can be filtered out layer by layer, and redundant information can be removed, thus achieving effective fusion of multiple text feature vectors.

[0101] In an exemplary embodiment, the text feature vector fused through multiple attention layers can also be processed by downsampling (DS) to adjust the data sampling rate and reduce the feature dimensionality, thereby improving data processing efficiency. Furthermore, this attention layer can be modeled using Fp16 precision.

[0102] Secondly, the first text feature vector is resized by passing it through at least one fully connected layer (FC layer) before being injected into the feature fusion model. By using FC layers of different scales, the first text feature vector is mapped to feature vectors that meet the different scale requirements of the intermediate layers of the feature fusion model, and then injected into different intermediate layers of the feature fusion model to achieve more thorough fusion with the image feature vector.

[0103] In step S530, the first image feature vector and the text feature vector are fused using the feature fusion model to obtain the image feature vector.

[0104] In this embodiment, the first image feature vector and the text feature vector are fused using the feature fusion model to obtain an image feature vector. During this feature fusion process, the first image feature vector is fused using the text feature vector as a reference. The fused image feature vector incorporates relevant feature information from the text feature vector, thereby providing a more accurate and comprehensive image feature vector for processing by subsequent modules in multimodal tasks.

[0105] In an exemplary embodiment, the feature fusion model can be modeled based on the UNet architecture. UNet is a Convolutional Neural Network (CNN) architecture that includes an encoder and a decoder, tightly linked together by skip connections. Figure 6 As shown, this feature fusion model based on the UNet structure extracts image features through multiple self-attention layers and introduces text features into the image features through multiple cross-attention layers. After training this feature fusion model, text features can be fused into image features, thereby achieving feature fusion of image feature vectors and text feature vectors to generate an image feature vector with fused text features.

[0106] In an exemplary embodiment, the feature fusion model is trained using Fp16 precision during the training phase and the quantization granularity can be adjusted to Int 8 during the inference phase to further reduce the feature size and alleviate the computational burden in subsequent processing.

[0107] In an exemplary embodiment, in order to further improve the detail accuracy, in addition to using the feature fusion model to perform first-level feature fusion of text feature vectors and image feature vectors, a second-level feature fusion can also be performed on the image feature vectors output by the feature fusion model based on the text feature vectors.

[0108] like Figure 6 As shown, on the one hand, the feature fusion model fuses the first image feature vector and the text feature vector to obtain the second image feature vector. On the other hand, the text feature vector output by the text encoder is downsampled to obtain the second text feature vector. The second image feature vector and the second text feature vector are then passed through at least two cross-attention layers, and the second image feature vector is fused with the second text feature vector again to obtain the image feature vector. This second feature fusion enhances the effectiveness of the text features, as text features themselves contain relatively important conceptual details and are easier to focus on than image feature modeling methods. Furthermore, it prevents the text features injected into the feature fusion model from being distorted during processing, leading to misunderstandings of the text features.

[0109] The image generation method disclosed herein uses a segmentation model to label local regions of a reference image, enabling the extracted reference image features to encompass both global and local features, thus enriching the content of the extracted features. By using a large language model to express the extracted reference image features as text features, the characteristic information of the reference image can be represented more accurately. By using an image encoder to pre-define the reference image features and fully integrating text and image features, the detail accuracy of feature representation is improved, and the effectiveness of text features is enhanced. By selectively simplifying the structure of related modules, the computational load of related processing can be reduced, improving the efficiency of related data processing.

[0110] Figure 7 This is a schematic diagram of a feature adaptation module framework according to an exemplary embodiment of the present disclosure. As previously described, the present disclosure uses a feature adaptation module to map the image feature vector and text feature vector into reference image features adapted to the image generation model. Depending on the model framework of the image generation model and the needs of different application scenarios, the feature adaptation module can perform different degrees of mapping adjustments on the image feature vector and text feature vector. Figure 7 The schematic diagram of the feature adaptation module framework shown illustrates several methods for feature adaptation and adjustment. In practical applications, one or more of these adaptation methods can be selected to map and adjust image feature vectors and text feature vectors as needed.

[0111] In some embodiments, such as Figure 7 As shown, in Figure 1 Based on the image generation method shown, step S120 may include the following steps.

[0112] Based on the feature scale of the image generation model, the image feature vector is scaled by the first feature mapping module to obtain a first image feature. This first image feature is then injected into the image generation model.

[0113] In this embodiment, since most mainstream image generation models, such as the SD model and the Flux model, are based on the Transformer architecture, the main difference between different types of image generation models lies in their feature scale. The first feature mapping module adjusts the feature scale of the image feature vector according to the feature scale requirements of the image generation model to obtain a first image feature adapted to the image generation model.

[0114] like Figure 7As shown, the first feature mapping module consists of multiple convolutional layers and linear layers. The image feature vector output by the content extraction module is scaled through multiple convolutional and linear layers, and finally transformed into a first image feature adapted to the input feature scale of the image generation model through channel dimension mapping. This first image feature is then injected into the image generation model as the initial feature.

[0115] In some embodiments, such as Figure 7 As shown, in Figure 1 Based on the image generation method shown, step S120 may include the following steps.

[0116] Based on the feature scale of the image generation model, the image feature vector is scaled by the second feature mapping module to obtain the second image feature.

[0117] Based on the feature scale of each intermediate layer in the image generation model, the scale of the second image feature is adjusted by the third feature mapping module to obtain at least one third image feature; the third image feature corresponds to each of the intermediate layers.

[0118] The third image features are injected into the corresponding intermediate layers of the image generation model.

[0119] In this embodiment, the aforementioned first image feature already satisfies the scale requirements of the image generation model's backbone for input features. However, as the image generation model network deepens, the required feature scale changes at different stages of the intermediate layers. To fully inject the feature information of the reference image into the intermediate layers of the image generation model, this disclosure further generates image features adapted to the intermediate layers.

[0120] like Figure 7 As shown, firstly, based on the feature scale of the image generation model, the image feature vector is scaled by the second feature mapping module to obtain the second image feature. This process is similar to the aforementioned first image feature generation process. The second feature mapping module can also adopt a similar module structure to the aforementioned first feature mapping module, which will not be elaborated here. The second image feature is an image feature adapted to the input feature scale of the image generation model.

[0121] Secondly, a third feature mapping module is provided following the second feature mapping module. This third feature mapping module readjusts the feature scale of the second image feature according to the feature scale requirements of each intermediate layer in the image generation model to generate at least one third image feature. Each of these third image features corresponds to a different intermediate layer of the image generation model. This third feature mapping module can be implemented using multiple convolutional layers and channel mapping.

[0122] Finally, the generated third image features corresponding to each intermediate layer of the image generation model are injected into the corresponding intermediate layer of the image generation model, so that the intermediate layer of the image generation model can better refer to the relevant features for processing.

[0123] In some embodiments, such as Figure 7 As shown, in Figure 1 Based on the image generation method shown, step S120 may include the following steps.

[0124] The at least one third image feature is concatenated with the text feature vector to obtain text features. The text features are then injected into the image generation model.

[0125] In this embodiment of the disclosure, based on the aforementioned at least one third image feature, the at least one third image feature is concatenated with the aforementioned text feature vector to obtain text features. The concatenated text features are then injected into the text feature injection layer (generally a Transformer layer or a Cross-Attention layer) of the image generation model to achieve text feature injection.

[0126] The image generation method disclosed herein transforms image feature vectors and text feature vectors into feature scales adapted to the corresponding image generation model through a feature adaptation module, thereby improving the adaptability of the feature extraction module. It provides multiple feature adaptation adjustment methods, enabling adjustment of the feature scale from input features to intermediate layer features and then to text feature injection for the image generation model, meeting the different feature scale adjustment needs of the image generation model according to the application scenario.

[0127] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0128] Figure 8 This is a block diagram illustrating an image generation apparatus according to an exemplary embodiment of the present disclosure. The apparatus of this embodiment can be applied to electronic devices, including smart speakers, smartphones, and smart tablet terminal devices, and may also include server-side components such as local servers and cloud servers, which can be deployed in a computer or a computer cluster consisting of multiple computers.

[0129] like Figure 8 As shown, the image generation apparatus 800 may include: a content extraction unit 810, a feature adaptation unit 820, a reference image feature injection unit 830, an image generation instruction input unit 840, and an image generation unit 850.

[0130] The content extraction unit 810 is used to extract content from the input reference image through the content extraction module to obtain image feature vectors and text feature vectors.

[0131] The feature adaptation unit 820 is used to map the image feature vector and text feature vector to reference image features that are adapted to the image generation model through the feature adaptation module.

[0132] The reference image feature injection unit 830 is used to inject the reference image features into the image generation model.

[0133] The image generation instruction input unit 840 is used to input image generation instructions into the image generation model.

[0134] The image generation unit 850 is used to perform image generation processing on the image generation instruction based on the features of the reference image and through the image generation model to obtain the target generated image.

[0135] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to perform image analysis on the reference image using a segmentation model to obtain local image labeling information; the local image labeling information is used to label at least one local region in the reference image; extract text features from the reference image using a large language model to obtain a global text description; extract text features from the reference image using the large language model based on the local image labeling information to obtain at least one local text description; perform text encoding processing on the global text description and at least one local text description using a text encoder to obtain the text feature vector; determine at least one local image based on the reference image using the local image labeling information; and perform image encoding processing on the reference image and at least one local image using an image encoder to obtain the image feature vector.

[0136] In some exemplary embodiments of this disclosure, the large language model extracts text features from the reference image based on at least one pre-set set of target cues.

[0137] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to sort the at least one local text description according to the area of ​​the local region corresponding to the local text description.

[0138] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to perform text encoding processing on the global text description by the text encoder to obtain a global text feature vector; perform text encoding processing on the at least one local text description by the text encoder to obtain at least one local text feature vector; the local text feature vectors correspond to the local text descriptions respectively; and concatenate the global text feature vector with the at least one local text feature vector to obtain the text feature vector.

[0139] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to concatenate at least two of the global text feature vectors with the at least one local text feature vector to obtain the text feature vector.

[0140] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to perform image encoding processing on the reference image and at least one local image through the image encoder to obtain a first image feature vector; inject the text feature vector into a feature fusion model; and perform feature fusion on the first image feature vector and the text feature vector through the feature fusion model to obtain the image feature vector.

[0141] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to perform text feature fusion on the text feature vector through at least two attention layers to obtain a first text feature vector; and to inject the first text feature vector into the feature fusion model after adjusting its size through at least one fully connected layer.

[0142] In some exemplary embodiments of this disclosure, the content extraction unit 810 is further configured to perform feature fusion on the first image feature vector and the text feature vector through the feature fusion model to obtain a second image feature vector; perform downsampling processing on the text feature vector to obtain a second text feature vector; and perform feature fusion on the second image feature vector and the second text feature vector through at least two cross-attention layers to obtain the image feature vector.

[0143] In some exemplary embodiments of this disclosure, the feature adaptation unit 820 is further configured to adjust the scale of the image feature vector through the first feature mapping module according to the feature scale of the image generation model to obtain a first image feature; and inject the first image feature into the image generation model.

[0144] In some exemplary embodiments of this disclosure, the feature adaptation unit 820 is further configured to: adjust the scale of the image feature vector through a second feature mapping module according to the feature scale of the image generation model to obtain a second image feature; adjust the scale of the second image feature through a third feature mapping module according to the feature scale of each intermediate layer in the image generation model to obtain at least one third image feature; the third image feature corresponds to each intermediate layer; and the third image feature is injected into the corresponding intermediate layer in the image generation model.

[0145] In some exemplary embodiments of this disclosure, the feature adaptation unit 820 is further configured to concatenate the at least one third image feature with the text feature vector to obtain text features; and inject the text features into the image generation model.

[0146] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0147] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, device 900 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0148] Reference Figure 9 The device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0149] Processing component 902 typically controls the overall operation of device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0150] Memory 904 is configured to store various types of data to support the operation of device 900. Examples of this data include instructions for any application or method operating on device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0151] Power supply component 906 provides power to various components of device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 900.

[0152] Multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0153] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0154] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0155] Sensor assembly 914 includes one or more sensors for providing status assessments of various aspects of device 900. For example, sensor assembly 914 may detect the on / off state of device 900, the relative positioning of components such as the display and keypad of device 900, changes in position of device 900 or a component of device 900, the presence or absence of user contact with device 900, orientation or acceleration / deceleration of device 900, and temperature changes of device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0156] Communication component 916 is configured to facilitate wired or wireless communication between device 900 and other devices. Device 900 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 916 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0157] In some embodiments of this disclosure, the apparatus 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0158] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions that can be executed by a processor 920 of device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0159] In some embodiments of this disclosure, a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform an image generation method, the method comprising:

[0160] The content extraction module extracts content from the input reference image to obtain image feature vectors and text feature vectors.

[0161] The feature adaptation module maps the image feature vector and text feature vector to reference image features that are adapted to the image generation model.

[0162] The reference image features are injected into the image generation model;

[0163] Input the image generation instructions into the image generation model;

[0164] Based on the features of the reference image, the image generation model is used to process the image generation instruction to obtain the target generated image.

[0165] In some embodiments of this disclosure, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement an image generation method, the method comprising:

[0166] The content extraction module extracts content from the input reference image to obtain image feature vectors and text feature vectors.

[0167] The feature adaptation module maps the image feature vector and text feature vector to reference image features that are adapted to the image generation model.

[0168] The reference image features are injected into the image generation model;

[0169] Input the image generation instructions into the image generation model;

[0170] Based on the features of the reference image, the image generation model is used to process the image generation instruction to obtain the target generated image.

[0171] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0172] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that, The method includes: The content extraction module extracts content from the input reference image to obtain image feature vectors and text feature vectors. The feature adaptation module maps the image feature vector and text feature vector to reference image features that are adapted to the image generation model. The reference image features are injected into the image generation model; Input the image generation instructions into the image generation model; Based on the features of the reference image, the image generation model is used to process the image generation instruction to obtain the target generated image.

2. The method according to claim 1, characterized in that, The step of extracting content from the input reference image using the content extraction module to obtain image feature vectors and text feature vectors includes: Image analysis is performed on the reference image using a segmentation model to obtain local image labeling information; the local image labeling information is used to label at least one local region in the reference image; The reference image is used to extract text features using a large language model to obtain a global text description; Based on the local image labeling information, text features are extracted from the reference image using the large language model to obtain at least one local text description. The text feature vector is obtained by performing text encoding processing on the global text description and at least one local text description through a text encoder. Based on the image local labeling information, at least one local image is determined according to the reference image; The reference image and at least one local image are image encoded using an image encoder to obtain the image feature vector.

3. The method according to claim 2, characterized in that, The large language model extracts text features from the reference image based on at least one pre-set set of target cues.

4. The method according to claim 2, characterized in that, The step of extracting text features from the reference image based on the local image labeling information and using the large language model to obtain at least one local text description further includes: The at least one local text description is sorted according to the area of ​​the local region corresponding to the local text description.

5. The method according to claim 2, characterized in that, The step of encoding the global text description and at least one local text description using a text encoder to obtain the text feature vector includes: The global text description is encoded using the text encoder to obtain a global text feature vector. The text encoder performs text encoding processing on the at least one local text description to obtain at least one local text feature vector; the local text feature vectors correspond to the local text descriptions respectively. The global text feature vector is concatenated with at least one local text feature vector to obtain the text feature vector.

6. The method according to claim 5, characterized in that, The step of concatenating the global text feature vector with at least one local text feature vector to obtain the text feature vector includes: The text feature vector is obtained by concatenating at least two global text feature vectors with at least one local text feature vector.

7. The method according to claim 2, characterized in that, The step of performing image encoding processing on the reference image and at least one local image using an image encoder to obtain the image feature vector includes: The image encoder performs image encoding processing on the reference image and at least one local image to obtain a first image feature vector. Inject the text feature vectors into the feature fusion model; The first image feature vector and the text feature vector are fused using the feature fusion model to obtain the image feature vector.

8. The method according to claim 7, characterized in that, The step of injecting the text feature vector into the feature fusion model includes: The text feature vector is fused through at least two attention layers to obtain the first text feature vector. The first text feature vector is resized by passing it through at least one fully connected layer and then injected into the feature fusion model.

9. The method according to claim 7, characterized in that, The step of fusing the first image feature vector and the text feature vector using the feature fusion model to obtain the image feature vector includes: The first image feature vector and the text feature vector are fused using the feature fusion model to obtain the second image feature vector; The text feature vector is downsampled to obtain a second text feature vector; The second image feature vector and the second text feature vector are fused through at least two cross-attention layers to obtain the image feature vector.

10. The method according to claim 1, characterized in that, The step of mapping the image feature vector and text feature vector to reference image features adapted to the image generation model through the feature adaptation module includes: Based on the feature scale of the image generation model, the image feature vector is scaled by the first feature mapping module to obtain the first image feature. The first image feature is injected into the image generation model.

11. The method according to claim 1, characterized in that, The step of mapping the image feature vector and text feature vector to reference image features adapted to the image generation model through the feature adaptation module includes: Based on the feature scale of the image generation model, the image feature vector is scaled by the second feature mapping module to obtain the second image feature. Based on the feature scales of each intermediate layer in the image generation model, the scale of the second image feature is adjusted by the third feature mapping module to obtain at least one third image feature; the third image feature corresponds to each of the intermediate layers. The third image features are injected into the corresponding intermediate layers of the image generation model.

12. The method according to claim 11, characterized in that, The step of mapping the image feature vector and text feature vector to reference image features adapted to the image generation model through the feature adaptation module further includes: The text feature is obtained by concatenating the at least one third image feature with the text feature vector; The text features are injected into the image generation model.

13. An image generation apparatus, characterized in that, include: The content extraction unit is used to extract content from the input reference image through the content extraction module to obtain image feature vectors and text feature vectors; The feature adaptation unit is used to map the image feature vector and text feature vector to reference image features that are adapted to the image generation model through the feature adaptation module. A reference image feature injection unit is used to inject the reference image features into the image generation model; An image generation instruction input unit is used to input image generation instructions into the image generation model; An image generation unit is used to perform image generation processing on the image generation instruction based on the features of the reference image and through the image generation model to obtain a target generated image.

14. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the image generation method according to any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform the image generation method of any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image generation method as described in any one of claims 1 to 12.

Images