Multi-modal image generation method and device

Abstract

The invention provides a multi-modal image generation method and device. The method comprises the following steps: inputting a reference image and text description into a multi-modal image generation model; extracting image features of the reference image through an image feature extraction network; extracting text features of the text description through a text encoder; randomly generating a noise map through a noise map generation network; processing the noise map through an attention layer to obtain noise features; processing the text features and the noise features through a first cross attention layer to obtain attention features; and processing the image features and the attention features through the second cross attention layer to obtain a target image. By adopting the technical means, the problem that in the prior art, the generated image cannot meet the personalized effect of text description on the basis of keeping the key features of the reference image is solved.

Description

Technical Field

[0001] The present application relates to the field of image generation technology, and in particular to a multimodal image generation method and device. Background Art

[0002] With the rapid advancement of image generation technology, generating personalized images has attracted great attention due to its wide application potential in applications such as AI portrait photos, image animation, and virtual try-on. In this field, image customization technology is facing a core challenge: how to cleverly combine the key features of the reference image with the customization requirements of the text description during the generation process to create a personalized face image that is both faithful to the reference image and meets the text description. This requires the algorithm to not only accurately capture and retain the key features of the reference image, but also be able to understand and execute the customization instructions in the text description, while ensuring that the generated image is highly consistent with the reference image visually, achieving a natural and user-desirable personalized effect. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a multimodal image generation method, device, electronic device and computer-readable storage medium to solve the problem in the prior art that the generated image cannot meet the personalized effect of the text description while retaining the key features of the reference image.

[0004] According to a first aspect of an embodiment of the present application, a multimodal image generation method is provided, including: constructing an image feature extraction network, and constructing a multimodal image generation model using the image feature extraction network, a text encoder, a noise map generation network, an attention layer, and two cross-attention layers; obtaining a reference image and a text description of a target image to be generated, and inputting the reference image and the text description into the multimodal image generation model: extracting image features of the reference image through the image feature extraction network; extracting text features of the text description through the text encoder; randomly generating a noise map through a noise map generation network; processing the noise map through an attention layer to obtain noise features; processing text features and noise features through a first cross-attention layer to obtain attention features; processing image features and attention features through a second cross-attention layer to obtain a target image.

[0005] According to a second aspect of an embodiment of the present application, a multimodal image generation device is provided, comprising: a construction module, configured to construct an image feature extraction network, and construct a multimodal image generation model using the image feature extraction network, a text encoder, a noise map generation network, an attention layer, and two cross-attention layers; an acquisition module, configured to acquire a reference image and a text description of a target image to be generated, and input the reference image and the text description into the multimodal image generation model: a first extraction module, configured to extract image features of a reference image through an image feature extraction network; a second extraction module, configured to extract text features of a text description through a text encoder; a generation module, configured to randomly generate a noise map through a noise map generation network; a first processing module, configured to process the noise map through an attention layer to obtain noise features; a second processing module, configured to process text features and noise features through a first cross-attention layer to obtain attention features; and a third processing module, configured to process image features and attention features through a second cross-attention layer to obtain a target image.

[0006] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0007] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0008] Compared with the prior art, the embodiments of the present application have the following beneficial effects: construct an image feature extraction network, and construct a multimodal image generation model using the image feature extraction network, a text encoder, a noise map generation network, an attention layer, and two cross-attention layers; obtain a reference image and a text description of the target image to be generated, and input the reference image and the text description into the multimodal image generation model: extract the image features of the reference image through the image feature extraction network; extract the text features of the text description through the text encoder; randomly generate a noise map through the noise map generation network; process the noise map through the attention layer to obtain the noise features; process the text features and the noise features through the first cross-attention layer to obtain the attention features; process the image features and the attention features through the second cross-attention layer to obtain the target image. The above-mentioned technical means can solve the problem in the prior art that the generated image cannot meet the personalized effect of the text description while retaining the key features of the reference image, and then generate an image that is both faithful to the reference image and can reflect the personalized effect of the text description, thereby improving the quality of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 It is a flowchart of a multimodal image generation method provided in an embodiment of the present application;

[0011] Figure 2 It is a flowchart of a training method for a multimodal image generation model provided in an embodiment of the present application;

[0012] Figure 3 is a structural schematic diagram of a multimodal image generation device provided in an embodiment of the present application;

[0013] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0015] A multimodal image generation method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0016] Figure 1 It is a flowchart of a multimodal image generation method provided in an embodiment of the present application. Figure 1 The multimodal image generation method can be executed by a computer or a server, or software on a computer or a server. Figure 1 As shown, the multimodal image generation method includes:

[0017] S101, construct an image feature extraction network, and use the image feature extraction network, text encoder, noise map generation network, attention layer and two cross attention layers to build a multimodal image generation model;

[0018] S102, obtaining a reference image and a text description of a target image to be generated, and inputting the reference image and the text description into a multimodal image generation model:

[0019] S103, extracting image features of the reference image through an image feature extraction network;

[0020] S104, extracting text features of the text description through a text encoder;

[0021] S105, randomly generating a noise map through a noise map generation network;

[0022] S106, processing the noise map through the attention layer to obtain noise features;

[0023] S107, processing the text feature and the noise feature through the first cross attention layer to obtain the attention feature;

[0024] S108, process the image features and attention features through the second cross-attention layer to obtain the target image.

[0025] The noise map generation network is a network that randomly generates noise maps using a specific function. The text encoder, attention layer and cross-attention layer are all existing networks. The text encoder can use the CLIP (Contrastive Language-Image Pre-training) text encoder. The embodiment of the present application specifically constructs an image feature extraction network, and then uses the image feature extraction network, text encoder, noise map generation network, attention layer and two cross-attention layers to construct a multimodal image generation model. The reference image and text description are the user's input. The multimodal image generation model needs to generate a target image that meets the personalized effect of the text description on the basis of ensuring the key features of the reference image. The text description describes the effect of generating the target image that the user expects in the form of text.

[0026] Inside the multimodal image generation model: the image feature extraction network extracts the image features of the reference image, the text encoder extracts the text features of the text description, and the noise map generation network randomly generates a noise map, where the generated noise map has the same dimension as the reference image. The attention layer processes the noise map to obtain noise features, and then the first cross-attention layer in the multimodal image generation model processes the text features and the noise features to obtain attention features. Finally, the second cross-attention layer in the multimodal image generation model processes the image features and the attention features to obtain the target image. It is easy to understand that the image itself is a feature, so the processing result of the second cross-attention layer can be directly used as the target image.

[0027] According to the technical solution provided in the embodiment of the present application, an image feature extraction network is constructed, and a multimodal image generation model is constructed using the image feature extraction network, a text encoder, a noise map generation network, an attention layer, and two cross-attention layers; a reference image and a text description of the target image to be generated are obtained, and the reference image and the text description are input into the multimodal image generation model: the image features of the reference image are extracted by the image feature extraction network; the text features of the text description are extracted by the text encoder; a noise map is randomly generated by the noise map generation network; the noise map is processed by the attention layer to obtain noise features; the text features and noise features are processed by the first cross-attention layer to obtain attention features; the image features and attention features are processed by the second cross-attention layer to obtain the target image. The above-mentioned technical means can solve the problem in the prior art that the generated image cannot meet the personalized effect of the text description on the basis of retaining the key features of the reference image, and then an image that is faithful to the reference image and can reflect the personalized effect of the text description is generated, thereby improving the quality of the generated image.

[0028] Furthermore, an image feature extraction network is constructed, including: constructing a first image encoding network using a first image encoder, a multi-layer perceptron and a cross-attention layer, wherein the first image encoder can encode all types of images; constructing a second image encoding network using a second image encoder, a multi-layer perceptron and a cross-attention layer, wherein the second image encoder can only encode images of the same type as the reference image; and constructing an image feature extraction network using the first image encoding network, the second image encoding network and a feedforward network.

[0029] The first image encoder, the multi-layer perceptron, and the cross-attention layer are connected in sequence to obtain the first image encoding network, and the second image encoder, the multi-layer perceptron, and the cross-attention layer are connected in sequence to obtain the second image encoding network. Assuming that the reference image is a face image, the second image encoding network is a network that specifically encodes face-type images. The first image encoder can use the CLIP image encoder, and the second image encoder can use FaceNet. The first image encoder extracts general features of face images, and the second image encoder is a network that specifically encodes face-type images, so the second image encoder extracts more detailed features of face images. The multi-layer perceptron, the cross-attention layer, and the feedforward network are all existing networks and will not be repeated here.

[0030] Furthermore, image features of a reference image are extracted through an image feature extraction network, including: within the image feature extraction network: processing the reference image and a query vector through a first image coding network to obtain a first image coding feature, wherein the query vector is determined during a training phase of a multimodal image generation model; processing the reference image and the first image coding feature through a second image coding network to obtain a second image coding feature; and processing the second image coding feature through a feedforward network to obtain an image feature.

[0031] The training phase of the multimodal image generation model is the phase of training the multimodal image generation model. The embodiments of the present application and the above text are both the inference phase of the multimodal image generation model. The inference phase is the phase after the training phase when the model is formally used.

[0032] Furthermore, the reference image and the query vector are processed by the first image coding network to obtain a first image coding feature, including: within the first image coding network: encoding the reference image by the first image encoder to obtain a first coding feature; processing the first coding feature by a multi-layer perceptron to obtain a first perceptual feature; processing the first perceptual feature and the query vector by a cross-attention layer to obtain the first image coding feature.

[0033] The first image encoder and the multi-layer perceptron process the reference image in sequence to obtain the first perceptual feature; the cross-attention layer performs cross-attention calculation on the first perceptual feature and the query vector to obtain the first image coding feature.

[0034] Furthermore, the reference image and the first image coding features are processed by the second image coding network to obtain the second image coding features, including: within the second image coding network: encoding the reference image by the second image encoder to obtain the second coding features; processing the second coding features by a multi-layer perceptron to obtain the second perceptual features; processing the second perceptual features and the first image coding features by a cross-attention layer to obtain the second image coding features.

[0035] The second image encoder and the multi-layer perceptron process the reference image in sequence to obtain a second perceptual feature; the cross-attention layer performs cross-attention calculation on the second perceptual feature and the first image coding feature to obtain a second image coding feature.

[0036] Figure 2 FIG. 1 is a flow chart of a method for training a multimodal image generation model provided in an embodiment of the present application. Figure 2 As shown, the method includes:

[0037] S201, obtaining training data, wherein the training data includes training reference images and training text descriptions, and inputting the training reference images and training text descriptions into a multimodal image generation model:

[0038] S202, extracting training image features of a training reference image through an image feature extraction network;

[0039] S203, extracting training text features described in the training text through a text encoder;

[0040] S204, randomly generating a training noise map through a noise map generation network;

[0041] S205, processing the training noise image through the attention layer to obtain training noise features;

[0042] S206, processing the training text feature and the training noise feature through a first cross attention layer to obtain a training attention feature;

[0043] S207, processing the training image features and the training attention features through a second cross attention layer to obtain a generated image;

[0044] S208, calculating the mean square error loss of the generated image and the training reference image, and optimizing the model parameters of the multimodal image generation model based on the mean square error loss to complete the training phase of the multimodal image generation model.

[0045] The embodiment of the present application is the training phase of the multimodal image generation model. The embodiments of the present application are all the reasoning phase of the multimodal image generation model. The training reference image is similar to the reference image, and the training text description is similar to the text description. The training phase and the reasoning phase of the model are similar, and will not be repeated here. The mean square error loss function is used to calculate the mean square error loss of the generated image and the training reference image. The training reference image can also be manually adjusted to obtain a label (similar to the annotation in the training of the classification regression type model), and the mean square error loss of the generated image and the label is calculated.

[0046] According to the technical solution provided in the embodiment of the present application, training data is obtained, wherein the training data includes a training reference image and a training text description, and the training reference image and the training text description are input into the multimodal image generation model: the training image features of the training reference image are extracted through the image feature extraction network; the training text features of the training text description are extracted through the text encoder; the training noise map is randomly generated through the noise map generation network; the training noise map is processed through the attention layer to obtain the training noise features; the training text features and the training noise features are processed through the first cross-attention layer to obtain the training attention features; the training image features and the training attention features are processed through the second cross-attention layer to obtain the generated image; the mean square error loss of the generated image and the training reference image is calculated, and the model parameters of the multimodal image generation model are optimized according to the mean square error loss to complete the training stage of the multimodal image generation model. The above technical means can solve the problem in the prior art that the generated image cannot meet the personalized effect of the text description on the basis of retaining the key features of the reference image, and then generate an image that is both faithful to the reference image and can reflect the personalized effect of the text description, thereby improving the quality of the generated image.

[0047] Furthermore, the training image features of the training reference image are extracted through the image feature extraction network, including: within the image feature extraction network: processing the training reference image and the initialized query vector through the first image coding network to obtain the first training image coding features; processing the training reference image and the first training image coding features through the second image coding network to obtain the second training image coding features; processing the second training image coding features through the feedforward network to obtain the training image features; wherein, optimizing the model parameters includes optimizing the query vector, and the query vector after the training phase is the query vector used by the subsequent multimodal image generation model.

[0048] The query vector is usually randomly initialized before the model starts training. Random initialization can be performed using probability distributions such as normal distribution or uniform distribution. During the model training process, the query vector is continuously updated through the back propagation algorithm as part of the model parameters. The goal of optimization is to enable the query vector to effectively capture important information in the input data, thereby improving the performance of the model on specific tasks.

[0049] The image feature extraction network extracts the training image features of the training reference image in a similar manner as the image feature extraction network extracts the image features of the reference image, which will not be repeated here. The query vector after initialization is a zero vector. The query vector after the training phase is the query vector in the first image coding feature obtained by the first image coding network.

[0050] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0051] The following is an embodiment of the device of the present application, which can be used to implement the embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the present application.

[0052] Figure 3 is a schematic diagram of a multimodal image generation device provided in an embodiment of the present application. Figure 3 As shown, the multimodal image generating device comprises:

[0053] A construction module 301 is configured to construct an image feature extraction network, and construct a multimodal image generation model using the image feature extraction network, the text encoder, the noise map generation network, the attention layer, and the two cross attention layers;

[0054] The acquisition module 302 is configured to acquire a reference image and a text description of a target image to be generated, and input the reference image and the text description into a multimodal image generation model:

[0055] A first extraction module 303 is configured to extract image features of a reference image through an image feature extraction network;

[0056] A second extraction module 304 is configured to extract text features of the text description through a text encoder;

[0057] A generating module 305 is configured to randomly generate a noise map through a noise map generating network;

[0058] A first processing module 306 is configured to process the noise map through an attention layer to obtain a noise feature;

[0059] A second processing module 307 is configured to process the text feature and the noise feature through a first cross attention layer to obtain an attention feature;

[0060] The third processing module 308 is configured to process the image features and the attention features through a second cross-attention layer to obtain a target image.

[0061] According to the technical solution provided in the embodiment of the present application, an image feature extraction network is constructed, and a multimodal image generation model is constructed using the image feature extraction network, a text encoder, a noise map generation network, an attention layer, and two cross-attention layers; a reference image and a text description of the target image to be generated are obtained, and the reference image and the text description are input into the multimodal image generation model: the image features of the reference image are extracted by the image feature extraction network; the text features of the text description are extracted by the text encoder; a noise map is randomly generated by the noise map generation network; the noise map is processed by the attention layer to obtain noise features; the text features and noise features are processed by the first cross-attention layer to obtain attention features; the image features and attention features are processed by the second cross-attention layer to obtain the target image. The above-mentioned technical means can solve the problem in the prior art that the generated image cannot meet the personalized effect of the text description on the basis of retaining the key features of the reference image, and then an image that is faithful to the reference image and can reflect the personalized effect of the text description is generated, thereby improving the quality of the generated image.

[0062] In some embodiments, the construction module 301 is also configured to construct a first image coding network using a first image encoder, a multi-layer perceptron, and a cross-attention layer, wherein the first image encoder can encode all types of images; construct a second image coding network using a second image encoder, a multi-layer perceptron, and a cross-attention layer, wherein the second image encoder can only encode images of the same type as the reference image; and construct an image feature extraction network using the first image coding network, the second image coding network, and the feedforward network.

[0063] In some embodiments, the first extraction module 303 is also configured to: process the reference image and the query vector through the first image coding network to obtain the first image coding feature, wherein the query vector is determined in the training phase of the multimodal image generation model; process the reference image and the first image coding feature through the second image coding network to obtain the second image coding feature; and process the second image coding feature through the feedforward network to obtain the image feature.

[0064] In some embodiments, the first extraction module 303 is also configured to: encode the reference image through the first image encoder to obtain a first encoding feature; process the first encoding feature through a multi-layer perceptron to obtain a first perceptual feature; and process the first perceptual feature and the query vector through a cross-attention layer to obtain a first image encoding feature within the first image encoding network.

[0065] In some embodiments, the first extraction module 303 is also configured to: encode the reference image through the second image encoder to obtain a second encoding feature; process the second encoding feature through a multi-layer perceptron to obtain a second perceptual feature; and process the second perceptual feature and the first image encoding feature through a cross-attention layer to obtain a second image encoding feature.

[0066] In some embodiments, the acquisition module 302 is further configured to acquire training data, wherein the training data includes a training reference image and a training text description, and the training reference image and the training text description are input into the multimodal image generation model: the training image features of the training reference image are extracted through an image feature extraction network; the training text features of the training text description are extracted through a text encoder; the training noise image is randomly generated through a noise image generation network; the training noise image is processed through an attention layer to obtain a training noise feature; the training text features and the training noise features are processed through a first cross-attention layer to obtain a training attention feature; the training image features and the training attention features are processed through a second cross-attention layer to obtain a generated image; the mean square error loss of the generated image and the training reference image is calculated, and the model parameters of the multimodal image generation model are optimized based on the mean square error loss to complete the training phase of the multimodal image generation model.

[0067] In some embodiments, the acquisition module 302 is also configured to, within the image feature extraction network: process the training reference image and the initialized query vector through the first image coding network to obtain the first training image coding feature; process the training reference image and the first training image coding feature through the second image coding network to obtain the second training image coding feature; process the second training image coding feature through the feedforward network to obtain the training image feature; wherein, optimizing the model parameters includes optimizing the query vector, and the query vector after the training phase is the query vector used by the subsequent multimodal image generation model.

[0068] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0069] Figure 4 Schematic diagram of an electronic device 4 provided in an embodiment of the present application. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 401 executes the computer program 403, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0070] The electronic device 4 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4 The electronic device 4 is merely an example and does not limit the electronic device 4 , and may include more or less components than those shown in the figure, or different components.

[0071] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0072] The memory 402 may be an internal storage unit of the electronic device 4, for example, a hard disk or memory of the electronic device 4. The memory 402 may also be an external storage device of the electronic device 4, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 4. The memory 402 may also include both an internal storage unit and an external storage device of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0073] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0074] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0075] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multimodal image generation method, characterized in that: include: Build an image feature extraction network, and use the image feature extraction network, text encoder, noise map generation network, attention layer and two cross attention layers to build a multimodal image generation model; Get the reference image and text description of the target image to be generated, and input the reference image and text description into the multimodal image generation model: Extracting image features of the reference image through an image feature extraction network; Extract text features of text description through a text encoder; Randomly generate noise images through a noise image generation network; Process the noise map through the attention layer to obtain the noise features; Process the text features and noise features through the first cross-attention layer to obtain the attention features; The image features and attention features are processed by the second cross-attention layer to obtain the target image.

2. The method according to claim 1, characterized in that Construct an image feature extraction network, including: Constructing a first image encoding network using a first image encoder, a multi-layer perceptron, and a cross-attention layer, wherein the first image encoder can encode all types of images; A second image encoding network is constructed using a second image encoder, a multi-layer perceptron, and a cross-attention layer, wherein the second image encoder can only encode images of the same type as the reference image; An image feature extraction network is constructed using the first image encoding network, the second image encoding network and the feedforward network.

3. The method according to claim 2, characterized in that The image features of the reference image are extracted through the image feature extraction network, including: Inside the image feature extraction network: Processing the reference image and the query vector through a first image coding network to obtain a first image coding feature, wherein the query vector is determined during a training phase of the multimodal image generation model; Processing the reference image and the first image coding feature through a second image coding network to obtain a second image coding feature; The second image encoding feature is processed through a feedforward network to obtain an image feature.

4. The method according to claim 3, characterized in that The reference image and the query vector are processed by the first image coding network to obtain a first image coding feature, including: Inside the first image encoding network: Encoding the reference image by a first image encoder to obtain a first encoding feature; Processing the first encoding feature through a multi-layer perceptron to obtain a first perceptual feature; The first perceptual feature and the query vector are processed through a cross-attention layer to obtain the first image encoding feature.

5. The method according to claim 3, characterized in that Processing the reference image and the first image coding feature through the second image coding network to obtain the second image coding feature includes: Inside the second image encoding network: Encoding the reference image by a second image encoder to obtain a second encoding feature; Processing the second encoding feature through a multi-layer perceptron to obtain a second perceptual feature; The second perceptual feature and the first image encoding feature are processed through a cross attention layer to obtain a second image encoding feature.

6. The method according to claim 1, characterized in that Before inputting the reference image and the text description into the multimodal image generation model, the method further includes: Obtain training data, where the training data includes training reference images and training text descriptions, and input the training reference images and training text descriptions into the multimodal image generation model: Extracting training image features of a training reference image through an image feature extraction network; Extracting training text features described by the training text through a text encoder; Randomly generate training noise images through the noise image generation network; Process the training noise map through the attention layer to obtain the training noise features; Process the training text features and the training noise features through the first cross attention layer to obtain the training attention features; The training image features and the training attention features are processed through the second cross-attention layer to obtain the generated image; The mean square error loss of the generated image and the training reference image is calculated, and the model parameters of the multimodal image generation model are optimized based on the mean square error loss to complete the training phase of the multimodal image generation model.

7. The method according to claim 2, characterized in that The training image features of the training reference image are extracted through the image feature extraction network, including: Inside the image feature extraction network: Processing the training reference image and the initialized query vector through the first image coding network to obtain a first training image coding feature; Processing the training reference image and the first training image coding features through a second image coding network to obtain second training image coding features; Processing the encoded features of the second training image through a feedforward network to obtain training image features; Optimizing the model parameters includes optimizing the query vector, and the query vector after the training phase is the query vector used by the subsequent multimodal image generation model.

8. A multimodal image generation device, characterized in that: include: A construction module, configured to construct an image feature extraction network, and construct a multimodal image generation model using the image feature extraction network, the text encoder, the noise map generation network, the attention layer, and two cross attention layers; The acquisition module is configured to acquire a reference image and a text description of a target image to be generated, and input the reference image and the text description into the multimodal image generation model: A first extraction module is configured to extract image features of a reference image through an image feature extraction network; A second extraction module is configured to extract text features of the text description through a text encoder; A generation module, configured to randomly generate a noise map through a noise map generation network; A first processing module is configured to process the noise map through an attention layer to obtain a noise feature; A second processing module is configured to process the text feature and the noise feature through a first cross-attention layer to obtain an attention feature; The third processing module is configured to process the image features and the attention features through a second cross-attention layer to obtain a target image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When a processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images