Image generation method and device, computer equipment and storage medium

Abstract

The invention discloses an image generation method and device, computer equipment and a storage medium, and belongs to the field of image generation. The method comprises the following steps: acquiring content information and a plurality of reference images, wherein the plurality of reference images have the same reference style; extracting a content feature corresponding to the content information, and extracting a reference style feature corresponding to each reference image in the plurality of reference images; fusing the reference style features corresponding to the plurality of reference images to obtain a first fused style feature; executing a self-attention coding operation on the first fusion style feature to obtain a second fusion style feature; a target image is generated based on the content features with the reference style represented by the second fusion style features as guidance, and the content of the target image is matched with the content indicated by the content information. By providing a plurality of reference images of any one style, the image of the style can be generated, and the flexibility and the universality of the image generation method are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image generation, and in particular to an image generation method, apparatus, computer equipment and storage medium. Background Art

[0002] With the continuous development of computer technology, image processing technology has become more and more abundant. In some image processing scenarios, it is necessary to generate images of a specified style.

[0003] In the related art, if one wants to generate an image of a specified style, a large number of sample images of the specified style are used to train a model for generating images of the specified style.

[0004] In this method, only the trained model can be used to generate images of a specified style, and images of other styles cannot be generated, resulting in significant limitations in image generation. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, apparatus, computer device and storage medium, which can improve the efficiency of image generation. The technical solution includes the following aspects.

[0006] In one aspect, a method for generating an image is provided, the method comprising: Acquire content information and a plurality of reference images, wherein the plurality of reference images have the same reference style, and the content information is used to indicate the content of the image to be generated; Extracting content features corresponding to the content information, and extracting reference style features corresponding to each reference image in the plurality of reference images; Fusing the reference style features respectively corresponding to the multiple reference images to obtain a first fused style feature; Performing a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, where the second fused style feature is used to characterize the reference style; Guided by the reference style represented by the second fused style feature and based on the content feature, a target image is generated, wherein the content of the target image matches the content indicated by the content information.

[0007] In another aspect, an image generating device is provided, the device comprising: An acquisition module, used to acquire content information and a plurality of reference images, wherein the plurality of reference images have the same reference style, and the content information is used to indicate the content of the image to be generated; A feature extraction module, used to extract content features corresponding to the content information, and extract reference style features corresponding to each reference image in the plurality of reference images; A feature fusion module, used to fuse the reference style features respectively corresponding to the multiple reference images to obtain a first fused style feature; A self-attention encoding module, configured to perform a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, wherein the second fused style feature is used to characterize the reference style; An image generation module is used to generate a target image based on the content feature and guided by the reference style represented by the second fusion style feature, wherein the content of the target image matches the content indicated by the content information.

[0008] Optionally, the first image generation model includes an encoding network and a first downsampling network; the feature extraction module includes: An encoding unit, configured to perform an encoding operation on each of the reference images through the encoding network to obtain image encoding features corresponding to each of the reference images; A downsampling unit is used to perform a downsampling operation on the image coding features corresponding to each reference image through the first downsampling network to obtain the reference style features corresponding to each reference image.

[0009] Optionally, the first downsampling network includes a first residual layer and a feature encoding layer; the downsampling unit is used to: Mapping the image coding features corresponding to each of the reference images to first residual features corresponding to each of the reference images through the first residual layer; Through the feature encoding layer, an attention encoding operation is performed on the first residual feature corresponding to each reference image to obtain a reference style feature corresponding to each reference image.

[0010] Optionally, the feature encoding layer includes a first self-attention layer and a second self-attention layer; and the downsampling unit is used to: Performing a self-attention encoding operation on the first residual feature corresponding to each reference image through the first self-attention layer to obtain a first intermediate feature corresponding to each reference image; Through the second self-attention layer, a self-attention encoding operation is performed on the first intermediate features corresponding to each reference image to obtain the reference style features corresponding to each reference image.

[0011] Optionally, the first downsampling network includes k first downsampling layers connected in sequence, where k is an integer greater than 1; and the downsampling unit is used to: Performing a downsampling operation on the image coding features corresponding to each of the reference images through a first first downsampling layer among the k first downsampling layers, so as to obtain a first reference style feature corresponding to each of the reference images; A downsampling operation is performed on the m-1th first reference style feature corresponding to each reference image through the mth first downsampling layer among the k first downsampling layers to obtain the mth reference style feature corresponding to each reference image, where m is an integer greater than 1 and not greater than k.

[0012] Optionally, the first image generation model includes a feature processing network, the feature processing network includes k feature fusion layers; the feature fusion module is used to: fusing the first reference style features respectively corresponding to the multiple reference images through the first feature fusion layer among the k feature fusion layers to obtain a first first fused style feature; The mth reference style features respectively corresponding to the multiple reference images are fused through the mth feature fusion layer among the k feature fusion layers to obtain the mth first fused style feature.

[0013] Optionally, the feature processing network further includes k third self-attention layers; the self-attention encoding module is used to: Performing a self-attention encoding operation on the first first fused style feature through a first third self-attention layer among the k third self-attention layers to obtain a first second fused style feature; A self-attention encoding operation is performed on the mth first fusion style feature through the mth third self-attention layer among the k third self-attention layers to obtain the mth second fusion style feature.

[0014] Optionally, the first image generation model includes a decoding network and a first upsampling network; the image generation module includes: an upsampling unit, configured to perform an upsampling operation on the content feature and the second fused style feature through the first upsampling network to obtain a target image feature; A decoding unit is used to perform a decoding operation on the target image feature through the decoding network to obtain the target image.

[0015] Optionally, the first upsampling network includes a second residual layer and an attention layer; the upsampling unit is used to: Mapping the first noise feature to a second residual feature through the second residual layer, where the first noise feature is used to characterize random noise; Through the attention layer, an attention encoding operation is performed on the content feature, the second fusion style feature and the second residual feature to obtain the target image feature.

[0016] Optionally, the attention layer includes a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer; and the upsampling unit is used to: Performing a self-attention encoding operation on the second residual feature through the fourth self-attention layer to obtain a third residual feature; Performing a cross-attention encoding operation on the third residual feature and the second fused style feature through the first cross-attention layer to obtain a second intermediate feature; Through the second cross-attention layer, a cross-attention encoding operation is performed on the content feature and the second intermediate feature to obtain the target image feature.

[0017] Optionally, the first image generation model further includes a first downsampling network, the first downsampling network includes k first downsampling layers connected in sequence, the first upsampling network includes k first upsampling layers connected in sequence, the number of the second fusion style features is k, and k is an integer greater than 1; the upsampling unit is used to: performing an upsampling operation on the first noise feature, the content feature, and the first second fused style feature through the first first upsampling layer among the k first upsampling layers to obtain a first target image feature, wherein the first second fused style feature is determined based on the feature output by the first first downsampling layer, and the first noise feature is used to characterize random noise; An upsampling operation is performed on the m-1th target image feature, the content feature and the mth second fused style feature through the mth first upsampling layer among the k first upsampling layers to obtain the mth target image feature, wherein the mth second fused style feature is determined based on the feature output by the mth first downsampling layer, and m is an integer greater than 1 and not greater than k.

[0018] Optionally, the decoding unit is used to: Through the decoding network, a decoding operation is performed on the kth target image feature to obtain the target image.

[0019] Optionally, the device further comprises a model training module, which is used to: Acquire sample content information and a sample label image, wherein the content of the sample label image matches the content indicated by the sample content information; Extracting, by means of a second image generation model, a sample content feature corresponding to the sample content information, extracting a first sample reference feature corresponding to the sample label image, adding a second noise feature to the first sample reference feature to obtain a second sample reference feature, wherein the second noise feature is used to characterize random noise; Generate a sample target image based on the sample content feature by using the second image generation model and guided by the second sample reference feature; Based on the sample target image and the sample label image, training the second image generation model to obtain a trained second image generation model; The device also includes a model building module, which is used to: Based on the trained second image generation model, the first image generation model is constructed.

[0020] Optionally, the trained first image generation model includes an encoding network, a second downsampling network, a second upsampling network and a decoding network connected in sequence, the second downsampling network includes a first residual layer, a first self-attention layer and a third cross-attention layer connected in sequence, and the second upsampling network includes a second residual layer, a fourth self-attention layer and a first cross-attention layer connected in sequence; The model building module is used to: Copy the first self-attention layer to obtain a second self-attention layer; copy the first cross-attention layer to obtain a second cross-attention layer; Copying the fourth self-attention layer to obtain a third self-attention layer; constructing a feature processing network based on the feature fusion layer and the third self-attention layer; In the trained second image generation model, the third cross-attention layer of the second down-sampling network is replaced with the second self-attention layer to obtain a first down-sampling network, the second cross-attention layer is added after the first cross-attention layer of the second up-sampling network to obtain a first up-sampling network, and the feature processing network is added between the first down-sampling network and the first up-sampling network to obtain the first image generation model.

[0021] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the image generation method described in the above aspects.

[0022] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the image generation method described in the above aspects.

[0023] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is loaded and executed by a processor to implement the operations performed by the image generation method as described in the above aspects.

[0024] The solution provided by the embodiment of the present application provides multiple reference images of the specified style when it is necessary to generate an image of a specified style. First, the style features of multiple different reference images are extracted in parallel, and then the style features of multiple different reference images are fused and self-attention encoded, so as to capture the common and key style features in multiple different reference images, avoid being affected by the semantic features of the reference image itself, and ensure the global relevance of the extracted style features. Then, the target image is generated using the content features of the content information and the extracted style features, so that the content of the target image can be guaranteed to be the same as the content indicated by the content information, and the style of the target image is the same as the specified style. Therefore, the present application implements a zero-sample style transfer image generation solution, and only needs to provide multiple reference images of any specified style to generate such a specified style image, which is not limited to generating only one specific style image, and improves the flexibility and versatility of the image generation method. Moreover, under the guidance of the style features extracted from multiple reference images, the accuracy of the generated image can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 is a schematic diagram of a computer system provided in an embodiment of the present application; Figure 2 is a flow chart of an image generation method provided in an embodiment of the present application; Figure 3 is a structural schematic diagram of a first image generation model provided in an embodiment of the present application; Figure 4 is a flow chart of another image generation method provided in an embodiment of the present application; Figure 5 is a flow chart of a method for determining reference style features provided in an embodiment of the present application; Figure 6 is a schematic diagram of the structure of a downsampling network provided in an embodiment of the present application; Figure 7 It is a structural diagram of an upsampling network and a feature processing network provided in an embodiment of the present application; Figure 8 is a flow chart of a method for determining target image features provided by an embodiment of the present application; Fig. 9 is a comparison diagram of a reference image and a target image provided in an embodiment of the present application; Fig.10 is a flowchart of another image generation method provided in an embodiment of the present application; Fig.11 is a structural schematic diagram of another first image generation model provided in an embodiment of the present application; Fig.12 is a flow chart of a model training method provided in an embodiment of the present application; Fig.13 is a structural schematic diagram of a second image generation model provided in an embodiment of the present application; Fig.14 is a structural schematic diagram of a third image generation model provided in an embodiment of the present application; Fig.15 is a flowchart of a method for constructing a first image generation model provided in an embodiment of the present application; Fig.16 is a flowchart of another image generation method provided in an embodiment of the present application; Fig.17 is a structural schematic diagram of an image generating device provided in an embodiment of the present application; Fig.18 is a structural schematic diagram of another image generating device provided in an embodiment of the present application; Fig.19 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application; Fig. 20 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0028] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first fusion style feature may be referred to as a second fusion style feature, and similarly, a second fusion style feature may be referred to as a first fusion style feature.

[0029] Among them, at least one refers to one or more than one, for example, at least one reference image can be one reference image, two reference images, three reference images, or any other reference image that is an integer greater than or equal to one. Multiple refers to two or more than two, for example, multiple reference images can be two reference images, three reference images, or any other reference images that is an integer greater than or equal to two. Each refers to each of at least one, for example, each reference image refers to each reference image in multiple reference images, and if the multiple reference images are 3 reference images, each reference image refers to each reference image in the 3 reference images.

[0030] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the content information and reference images involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0031] To facilitate understanding, the concepts involved in the embodiments of the present application are explained below.

[0032] Stable Diffusion: is a variant of the Diffusion Model, called the Latent Diffusion Model (LDM). The purpose of the Diffusion Model is to eliminate the Gaussian noise that is continuously applied to the training images, which can be regarded as a series of denoising autoencoders. Stable Diffusion consists of 3 parts: VAE (Variational Auto-Encoder), U-Net (U-Net), and text encoder. VAE is used to transform the image into a low-dimensional latent space, and the process of adding and removing Gaussian noise is applied to the low-dimensional latent space, and then the denoised features are decoded into the pixel space. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent features. Each denoising step is completed by the U-Net architecture containing a residual neural network, and the latent features are obtained by denoising in the reverse direction from forward diffusion. Finally, the VAE decoder generates the image by transforming the latent features back to the pixel space.

[0033] U-Net: It is one of the algorithms that uses a fully convolutional network for semantic segmentation. It uses a symmetrical U-shaped structure that includes a compression path and an expansion path for processing. At the same time, U-Net is also the basic backbone model of the Stable Diffusion model. Self attention and cross attention modules are added to U-Net, so that the model noise reduction function can be built. U-Net is divided into down block and up block. In fact, U-Net in the Stable Diffusion model is divided into three parts, namely down block, mid block and up block. Each block in these three parts has the same composition, but the feature size and the direction of the feature dimension are different, thus distinguishing the down block, mid block and up block.

[0034] Clip (Contrastive Language-Image Pre-Training) model: This is a deep learning model that can understand text and images. It is a multimodal model that can match text and images and learn to recognize the content in the image and the text describing the image. The main goal is to learn to match images and text through contrastive learning. During the training process, the Clip model learned to encode images and text into a unified vector space, which enables it to understand the relationship between them both linguistically and visually. In this way, the Clip model can recognize elements such as objects, scenes, actions in the image, while also understanding the text associated with the image, such as tags, descriptions, titles, etc.

[0035] latent: In the Stable Diffusion model, "latent" usually refers to latent features, that is, features that are not directly observed in the model, which are the most basic features in the entire model processing process. These latent features are very important for explaining the observed data and the behavior of the model. In the Stable Diffusion model, latent features may represent unobserved factors, such as individual characteristics, behaviors, styles, and compositions, which may affect the diffusion process.

[0036] Cross attention: The "cross attention" in the Stable Diffusion model refers to the cross attention mechanism in the model. This cross attention mechanism is usually used to process the association between multiple input sequences in order to capture important information interactions between different sequences in the model. In the Stable Diffusion model, the cross attention mechanism can help the model better understand the association between different parts, so as to more accurately predict the changes and impacts in the diffusion process.

[0037] Self attention: The "self attention" in the Stable Diffusion model refers to the self-attention mechanism in the model. The self-attention mechanism is an important supplement to the U-Net of the Stable Diffusion model. The input of its calculation mechanism is the backbone feature latent of the Stable Diffusion model. The role of self attention is to further integrate all semantic representations in the latent feature, and at the same time make the representation of the generated object clearer, thereby improving the U-Net's ability to reduce latent noise.

[0038] Zero-shot: refers to the zero-sample learning mode, which allows the model to perform well on data of this category even if the model has not seen the data of this category in the training phase. In the embodiment of the present application, the zero-shot capability means that under the scheme of the present application, the model can achieve good performance in the style transfer function when facing unknown style images.

[0039] The image generation method provided in the embodiment of the present application can be used in a computer device. Optionally, the computer device is a terminal or a server. The embodiment of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0040] Figure 1 is a schematic diagram of a computer system provided in an embodiment of the present application, see Figure 1 , the computer system includes: a first terminal 101 and a server 102. The first terminal 101 and the server 102 are connected via a wireless or wired network.

[0041] The first terminal 101 is installed and runs a client 111, which may be a social application client, an online payment client, an online shopping client, a game client, a medical service client, an application store client, a video client, etc. When the first terminal 101 runs the client 111, the user interface of the client 111 is displayed on the screen of the first terminal 101. The first terminal 101 is a terminal used by the user 121.

[0042] Optionally, the first terminal 101 may generally refer to one of multiple terminals, and the first terminal 101 includes: smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, smart voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, VR (Virtual Reality) devices, AR (Augmented Reality) devices, etc., but is not limited to these.

[0043] Those skilled in the art will appreciate that the number of the above terminals may be more or less. For example, the above terminal may be only one, or the above terminals may be 6, 8, or more. The embodiment of the present application does not limit the number of terminals and device types.

[0044] Figure 1Only one terminal is shown in the figure, but in different embodiments, there are multiple other second terminals 103 that can access the server 102. Optionally, there are one or more second terminals 103 that are terminals corresponding to developers, and a client development and editing platform is installed on the second terminal 103. The developer can edit and update the client on the second terminal 103, and transmit the updated client installation package to the server 102 via a wired or wireless network. The first terminal 101 can download the client installation package from the server 102 to update the client.

[0045] The first terminal 101 and other second terminals 103 are connected to the server 102 via a wired network or a wireless network.

[0046] The server 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 is used to provide background services for the client. Optionally, the server 102 undertakes the main computing work, and the first terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the first terminal 101 undertakes the main computing work; or, the server 102 and the first terminal 101 adopt a distributed computing architecture for collaborative computing.

[0047] In the embodiment of the present application, the first terminal 101 sends an image generation request to the server 102, and the image generation request carries content information and multiple reference images. After receiving the image generation request, the server 102 uses the method provided in the embodiment of the present application to generate a target image based on the content information and multiple reference images. The target image has the same content as the content information, and the target image has the same style as the multiple reference images. The server 102 returns the target image to the first terminal 101.

[0048] It should be noted that the above implementation environment is only an example. The method provided in the embodiment of the present application can also be executed by the first terminal 101 or the server 102 alone, or by the first terminal 101 and the server 102 together, or by other computer devices. The embodiment of the present application does not limit this.

[0049] Figure 2 is a flowchart of an image generation method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Figure 2 , the method comprises the following steps.

[0050] 201. A computer device obtains content information and multiple reference images, where the multiple reference images have the same reference style, and the content information is used to indicate the content of an image to be generated.

[0051] The computer device obtains content information, and the content information is used to indicate the content of the image to be generated. The content information can be any type of information. For example, the content information can be a content image, and the image to be generated needs to include the content in the content image. For another example, the content information can be a description text, and the image to be generated needs to include the content described by the description text. For another example, the content information can be a description voice, and the image to be generated needs to include the content described by the description voice.

[0052] The computer device obtains multiple reference images, wherein the multiple reference images are different images, but the multiple reference images have the same reference style, that is, the multiple reference images correspond to the same style label, and the style label of the reference image is used to indicate the reference style of the reference image. For example, the reference styles of the multiple reference images are all anime style, pixel style, stick figure style, picture book style, etc.

[0053] The multiple reference images refer to at least two reference images, such as 3 reference images, 5 reference images or more reference images. The embodiment of the present application does not limit the number of reference images.

[0054] 202. The computer device extracts content features corresponding to the content information, and extracts reference style features corresponding to each reference image in the plurality of reference images.

[0055] For each of the multiple reference images, the computer device extracts features from the reference image to obtain reference style features corresponding to the reference image, thereby obtaining reference style features corresponding to each reference image. It should be noted that although the multiple reference images have the same reference style, it is difficult to directly and accurately extract features in the reference image that are only related to the reference style. The extracted reference style features may contain some features that are not related to the reference style. Therefore, the reference style features of the multiple reference images are not necessarily exactly the same.

[0056] The computer device extracts content features corresponding to the content information, where the content features are used to represent the semantics of the content information and can reflect the content of the content information.

[0057] 203. The computer device fuses the reference style features corresponding to the multiple reference images to obtain a first fused style feature.

[0058] After the computer device obtains the reference style features corresponding to each of the multiple reference images, it fuses the reference style features corresponding to the multiple reference images to obtain a first fused style feature. The first fused style feature includes the style features of the multiple reference images and is a global expression of the reference styles of the multiple reference images.

[0059] 204. The computer device performs a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, and the second fused style feature is used to represent the reference style.

[0060] After the computer device obtains the first fusion style feature, it uses a self-attention mechanism to perform a self-attention encoding operation on the first fusion style feature to obtain a second fusion style feature, thereby capturing the key part of the first fusion style feature and reducing the influence of features irrelevant to the style, so that the obtained second fusion style feature pays more attention to the part related to the style, so the second fusion style feature can more accurately represent the reference style.

[0061] In an embodiment of the present application, by fusing the style features of multiple different reference images with consistent styles, the style features of the multiple different reference images can be balanced, common style features can be extracted from the multiple different reference images, and then the extracted style features are further purified through self-attention encoding, thereby accurately extracting the style features of the multiple different reference images, thereby improving the expressive ability of the second fused style feature in image style.

[0062] 205. The computer device generates a target image based on the content feature and guided by the reference style represented by the second fused style feature, wherein the content of the target image matches the content indicated by the content information.

[0063] After acquiring the content features and the second fused style features, the computer device generates a target image based on the content features of the content dimension, guided by the second fused style features of the style dimension.

[0064] Since the content feature can reflect the content of the content information, the image generation process is guided in the content dimension through the content feature, so that the content of the generated target image can be guaranteed to match the content indicated by the content information. For example, if the content information is a running child, then the target image includes a running child. Since the second fused style feature can reflect the reference style of multiple reference images, the image generation process is guided in the style dimension through the second fused style feature, so that the generated target image can be guaranteed to have the same reference style as the multiple reference images. For example, if the style of multiple reference images is anime style, then the style of the target image is anime style. Therefore, under the joint action of the content feature and the second fused style feature, it can be guaranteed that the content and style of the generated target image are consistent with expectations.

[0065] It should be noted that the content information used in the embodiments of the present application may include any content, and the reference style of the multiple reference images may be any style. Therefore, by using the method of the embodiments of the present application, images of different styles can be flexibly generated, and it is only necessary to provide multiple reference images of a specified style, without the need to train different models for different styles, thus realizing a zero-shot style transfer image generation process.

[0066] The method provided by the embodiment of the present application provides multiple reference images of the specified style when it is necessary to generate an image of a specified style. First, the style features of multiple different reference images are extracted in parallel, and then the style features of multiple different reference images are fused and self-attention encoded, so as to capture the common and key style features in multiple different reference images, avoid being affected by the semantic features of the reference image itself, and ensure the global relevance of the extracted style features. Then, the target image is generated using the content features of the content information and the extracted style features, so that the content of the target image can be guaranteed to be the same as the content indicated by the content information, and the style of the target image is the same as the specified style. Therefore, the present application implements a zero-sample style transfer image generation scheme, and only needs to provide multiple reference images of any specified style to generate such a specified style image, which is not limited to generating only one specific style image, and improves the flexibility and versatility of the image generation method. Moreover, under the guidance of the style features extracted from multiple reference images, the accuracy of the generated image can be guaranteed.

[0067] Above Figure 2 The embodiment is only a brief description of the image generation method. In some embodiments, the computer device can also perform the image generation process through the first image generation model.

[0068] Figure 3 is a structural diagram of a first image generation model shown in an embodiment of the present application, such as Figure 3 As shown, the first image generation model includes an encoding network 301, a first downsampling network 302, a feature processing network 303, a first upsampling network 304, a decoding network 305, and a feature extraction network 306. The encoding network 301 is connected to the first downsampling network 302, the first downsampling network 302 is connected to the feature processing network 303, the feature processing network 303 is connected to the first upsampling network 304, the feature extraction network 306 is connected to the first upsampling network 304, and the first upsampling network 304 is connected to the decoding network 305.

[0069] Among them, the encoding network 301 is used to extract the first latent feature 12 from multiple reference images 11. The first downsampling network 302 is used to downsample the first latent feature 12 (that is, the image encoding feature) to obtain the latent feature 13 (that is, the reference style feature). The latent features 13 of multiple reference images 11 can be stored in the storage stack. The storage stack is a temporary storage container, not an actual network structure. The feature processing network 303 is used to balance the latent features 13 of multiple reference images 11, that is, the process of realizing feature fusion and self-attention encoding. The feature extraction network 306 is used to extract the content feature 15 of the content information. The first upsampling network 304 is used to upsample the first noise feature 14, the content feature 15 and the latent feature output by the feature processing network 303 to obtain the second latent feature 16 (that is, the target image feature). The decoding network 305 is used to decode the second latent feature 16 to obtain the target image 17.

[0070] Exemplarily, the encoding network 301 and the decoding network 305 are the encoding network and the decoding network in the VAE model architecture. VAE consists of two main parts: an encoder and a decoder. The encoder converts the image into a low-dimensional latent feature, which will be used as the input of the next network structure (U-Net in the embodiment of the present application). The decoder converts the latent feature back to the image. During the training process, the encoder is used to obtain the latent features of the input image in the forward diffusion process. During the inference process, the VAE encoder is no longer needed, and the input to the U-Net is a pure noise latent feature. The VAE decoder will convert the latent features output by the U-Net back to the image.

[0071] Exemplarily, the first downsampling network 302 and the first upsampling network 304 are the downsampling network and upsampling network in the U-Net model architecture. U-Net consists of two main parts: an encoder (downsampling part) and a decoder (upsampling part), both of which are composed of multiple unet blocks, and each unet block is composed of ResNet (residual network) and attention (attention network). The encoder compresses the high-resolution image into low-resolution features, and the decoder decodes the low-resolution features back to the high-resolution image. In order to prevent U-Net from losing important information when the encoder downsamples, an intermediate connection, such as a skip-connect, is usually added between the ResNet of the encoder and the upsampling ResNet of the decoder. In the Stable Diffusion model, U-Net adds a cross attention layer to adjust the potential features. The cross attention layer is added between the ResNet of the encoder and the ResNet of the decoder. At the same time, a self-attention layer is added between the ResNet and the cross attention layer. Therefore, in the network structure of U-Net, each unet block is divided into three parts, namely ResNet, self-attention layer and cross-attention layer.

[0072] Exemplarily, the feature extraction network 306 can be an image encoder or a text encoder. Taking the text encoder as an example, the function of the text encoder is to convert the input text into an embedding feature that can be understood by U-Net. The text encoder can be the text encoder module in the Clip model, which is a transformer-based encoder for mapping text into potential text embedding features.

[0073] In one possible implementation, Figure 3 As shown, the first downsampling network 302 includes k first downsampling layers connected in sequence, and the first upsampling network 304 includes k first upsampling layers connected in sequence, wherein the k first downsampling layers correspond one to one to the k first upsampling layers.

[0074] The image generation method provided in the embodiment of the present application is as follows: Figure 3 The first image generation model shown is implemented, and the detailed process is shown below Figure 4 Embodiment of the invention. Figure 4 is a flowchart of another image generation method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Figure 4 , the method comprises the following steps.

[0075] 401. A computer device obtains content information and multiple reference images, where the multiple reference images have the same reference style, and the content information is used to indicate the content of an image to be generated.

[0076] The computer device obtains content information, where the content information is used to indicate the content of the image to be generated. The content information can be any type of information. The computer device obtains multiple reference images, where the multiple reference images are different images, and the multiple reference images are used to indicate the reference style of the image to be generated. The embodiment of the present application does not limit the number of the multiple reference images.

[0077] In a possible implementation, a computer device displays an image generation interface, the image generation interface including a content upload entry and a style upload entry. Based on the content upload entry, the uploaded content information is obtained. Based on the style upload entry, the uploaded multiple reference images are obtained.

[0078] In a possible implementation, a computer device displays an image generation interface, which includes a content upload entry and a style selection area. Based on the content upload entry, the uploaded content information is obtained. The style selection area displays multiple style tags, and the multiple style tags are used to represent different styles. In response to a selection operation of any style tag, multiple reference images with the style tag are obtained. The computer device stores multiple reference images corresponding to any style tag.

[0079] 402. The computer device extracts content features corresponding to the content information through a feature extraction network of the first image generation model.

[0080] like Figure 3 As shown, the first image generation model includes a feature extraction network, the computer device inputs the content information into the feature extraction network, and the feature extraction network performs a feature extraction operation on the content information and outputs a content feature. The content feature is used to represent the semantics of the content information and can reflect the content of the content information.

[0081] Exemplarily, the content information is a content image, and the feature extraction network is used to perform feature extraction operations on any image, for example, the feature extraction network is an image encoder. The computer device inputs the content image into the image encoder, and the image encoder outputs content features.

[0082] Exemplarily, the content information is a description text, and the feature extraction network is used to perform feature extraction operations on any text, for example, the feature extraction network is a text encoder. The computer device inputs the description text into the text encoder, and the text encoder outputs content features.

[0083] Exemplarily, the content feature may be in the form of a feature matrix or a feature vector, etc., which is not limited in the embodiments of the present application.

[0084] 403. The computer device performs a coding operation on each reference image through the coding network of the first image generation model to obtain image coding features corresponding to each reference image.

[0085] like Figure 3 As shown, the first image generation model includes a coding network. The computer device inputs each of the multiple reference images into the coding network, and the coding network outputs the image coding features corresponding to each reference image. The image coding features are also Figure 3 The first potential feature 12 output by the encoding network 301 in the embodiment. The encoding process of each reference image in the plurality of reference images is independent from each other and does not interfere with each other.

[0086] Exemplarily, the encoding network is an encoder in a VAE model architecture.

[0087] 404. The computer device performs a downsampling operation on the image coding features corresponding to each reference image through a first downsampling network of the first image generation model to obtain a reference style feature corresponding to each reference image.

[0088] like Figure 3 As shown, the first image generation model includes a first downsampling network. For each reference image among the multiple reference images, the computer device inputs the image encoding feature corresponding to the reference image into the first downsampling network, and the first downsampling network outputs the reference style feature corresponding to the reference image. Therefore, the computer device can obtain the reference style feature corresponding to each reference image.

[0089] The reference style features of the reference image extracted by the first downsampling network include not only the style semantics of the reference image, but also other semantics that are irrelevant to the style semantics. That is, the reference style features cannot simply reflect the semantics, so the following steps 405 and 406 are required to purify the reference style features so that the purified features have higher significance and prominence in the style dimension.

[0090] In this implementation, the encoding network extracts the latent features of the reference image, and the first downsampling network further extracts the style features based on the latent features. By encoding and downsampling the reference image, the style features of the reference image can be effectively extracted, which is beneficial to improving the effect of style transfer in the image generation process.

[0091] In one possible implementation, the first downsampling network includes a first residual layer and a feature encoding layer, the first residual layer is connected to the feature encoding layer, and the output of the first residual layer is the input of the feature encoding layer. The first residual layer is used to perform a residual processing operation, and the residual processing operation is used to retain the features before the residual. The feature encoding layer is used to perform an attention encoding operation, which refers to the process of encoding input data using an attention mechanism. In this process, the model dynamically generates a set of encoding vectors based on the relevance of different parts of the input data to the current task. These encoding vectors not only contain the original information of the input data, but also incorporate the model's understanding and attention to the input data.

[0092] So, if Figure 5 As shown, step 404 includes the following steps 4041 and 4042.

[0093] 4041. Map the image coding features corresponding to each reference image to the first residual features corresponding to each reference image through the first residual layer of the first downsampling network.

[0094] The first residual layer can be a ResNet network structure, and the basic unit of ResNet is a residual block, which is composed of two or more layers of convolution, and jump connections are added between these layers. These jump connections directly add the input to the output, thereby realizing the transfer of residuals.

[0095] For each reference image among the multiple reference images, the computer device inputs the image coding feature corresponding to the reference image into the first residual layer, and the first residual layer outputs the first residual feature.

[0096] 4042. Perform an attention encoding operation on the first residual feature corresponding to each reference image through the feature encoding layer of the first down-sampling network to obtain a reference style feature corresponding to each reference image.

[0097] For each reference image among the multiple reference images, the computer device inputs a first residual feature corresponding to the reference image into a feature encoding layer, and the feature encoding layer outputs a reference style feature.

[0098] Optionally, the feature encoding layer includes a first self-attention layer and a second self-attention layer. The first self-attention layer is connected to the second self-attention layer, and the output of the first self-attention layer is the input of the second self-attention layer.

[0099] Then, step 4042 includes: performing a self-attention encoding operation on the first residual features corresponding to each reference image through the first self-attention layer, so as to obtain the first intermediate features corresponding to each reference image; performing a self-attention encoding operation on the first intermediate features corresponding to each reference image through the second self-attention layer, so as to obtain the reference style features corresponding to each reference image.

[0100] Among them, the self-attention mechanism is a special attention mechanism that allows the model to directly consider the relationship between the positions of the input data when processing the input data without relying on external information. The core of this mechanism is to determine the degree of attention between different positions by calculating the correlation (or attention weight) between each position in the input data and other positions.

[0101] Exemplarily, the network structure of the first self-attention layer is the same as the network structure of the second self-attention layer, the model parameters of the first self-attention layer are different from the model parameters of the second self-attention layer, or the model parameters of the first self-attention layer and the model parameters of the second self-attention layer may also be the same.

[0102] Exemplarily, taking the first self-attention layer as an example, the process of self-attention encoding includes: using a first mapping matrix to map the first residual feature of the reference image into a first query feature, a first key feature and a first value feature; performing a normalization operation on the first query feature, the product of the transposition of the first key feature and the scaling factor to obtain a normalized feature, and multiplying the normalized feature with the first value feature to obtain a first intermediate feature. The key feature (key), value feature (value) and query feature (query) in the embodiment of the present application belong to different feature spaces respectively.

[0103] Among them, the first mapping matrix is ​​used to perform a spatial transformation operation on the first residual feature, and the first mapping matrix is ​​a model parameter of the first self-attention layer, and the first mapping matrix is ​​obtained in the model training stage. The computer device multiplies the first residual feature with the first mapping matrix to obtain a first spatial matrix, and splits out the first query feature, the first key feature and the first value feature in the first spatial matrix. For example, the first mapping matrix is ​​a 3-dimensional mapping matrix, and the first spatial matrix obtained by multiplying the first residual feature with the first mapping matrix is ​​a 3-dimensional first spatial matrix. The electronic device uses the features on each dimension in the first spatial matrix as the first query feature, the first key feature and the first value feature, respectively.

[0104] Among them, the process of performing the self-attention encoding operation on the first intermediate feature is the same as the process of performing the self-attention encoding operation on the image encoding feature, which will not be repeated here. Figure 6is a schematic diagram of a downsampling network structure provided in an embodiment of the present application, such as Figure 6 As shown in (1), the first downsampling network 302 in the embodiment of the present application includes multiple first downsampling layers 31, and each first downsampling layer 31 is composed of a first residual layer, a first self-attention layer and a second self-attention layer. Figure 6 (1) is only illustrated by taking the example that the first downsampling network 302 includes multiple first downsampling layers 31. In fact, the first downsampling network 302 may also include only one first downsampling layer 31, or it can be understood that the first downsampling network 302 only includes a set of first residual layer, first self-attention layer and second self-attention layer.

[0105] In this implementation, by introducing the residual layer, the gradient vanishing problem in the deep network can be avoided, so that deeper features can be extracted. By introducing attention encoding, it is possible to focus on the important parts of the style features, which is conducive to extracting the key style features in the image and reducing the interference of style-irrelevant features in the image, thereby improving the effect of style transfer in the image generation process.

[0106] In addition, by using a two-layer self-attention mechanism, the model can pay more attention to the key style features in the image, making the extracted style features more significant and prominent, ignoring the interference of redundant information, improving the ability to express the details of the image style, and facilitating capturing the subtle differences in style features in images of different styles, thereby improving the effect of style transfer in the image generation process.

[0107] 405. The computer device fuses the reference style features corresponding to the plurality of reference images through the feature processing network of the first image generation model to obtain a first fused style feature.

[0108] like Figure 3 As shown, the first image generation model includes a feature processing network. The computer device inputs the reference style features of multiple reference images into the feature processing network, and the feature processing network fuses the reference style features of the multiple reference images to obtain a first fused style feature.

[0109] In the embodiment of the present application, the generation of an image of a specified style mainly depends on the reference style features of multiple reference images with the specified style. Therefore, the first fused style features of multiple reference images are fused to balance the features of the multiple reference images in the style dimension, and the first fused style features that can characterize the common style attributes of the multiple reference images are obtained to avoid biasing the style features of a single reference image, which is conducive to improving the accuracy of style transfer in image generation tasks.

[0110] In a possible implementation, the reference style features of multiple reference images are averaged in the channel dimension to obtain the first fused style feature. The common attributes between the multiple reference style features can be automatically extracted during the averaging process. Alternatively, the reference style features of multiple reference images are concatenated to obtain the first fused style feature.

[0111] Figure 7 is a schematic diagram of the structure of an upsampling network and a feature processing network provided in an embodiment of the present application, such as Figure 7 As shown in (1), the first image generation model in the embodiment of the present application includes a feature processing network 303. Taking the number of multiple reference images as 3 as an example, the computer device obtains the reference style feature a, reference style feature b and reference style feature c of the three reference images. The reference style feature a, reference style feature b and reference style feature c are input into the feature processing network 303, and the reference style feature a, reference style feature b and reference style feature c are concat (connected) in the channel dimension in the feature processing network 303, and then the average value is calculated in the channel dimension to obtain the first fusion style feature.

[0112] In a possible implementation, generating an image of a specified style mainly depends on the reference style features of multiple reference images of the specified style. Once the reference style features of the multiple reference images have different degrees of feature expression, the style features of different reference images will be affected. Therefore, when balancing the reference style features of multiple reference images, directly taking the average value may affect the final purification effect. Therefore, performing a weighted average operation on the reference style features of multiple reference images to obtain the first fused style feature can automatically weaken the role of the reference image with poor style representation in purification, so that the style can be transferred more accurately.

[0113] The weight of the reference style feature of the reference image may be determined according to the style representation degree of the reference image. The higher the style representation degree of the reference image, the greater the weight of the reference style feature of the reference image. The lower the style representation degree of the reference image, the smaller the weight of the reference style feature of the reference image.

[0114] 406. The computer device performs a self-attention encoding operation on the first fused style feature through a feature processing network of the first image generation model to obtain a second fused style feature, and the second fused style feature is used to characterize the reference style.

[0115] The feature processing network performs a self-attention encoding operation on the first fused style feature to obtain a second fused style feature. The process of performing the self-attention encoding operation on the first fused style feature is the same as the process of performing the self-attention encoding operation on the image coding feature, which will not be repeated here.

[0116] In an embodiment of the present application, a self-attention mechanism is adopted to perform a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, thereby capturing the key part of the first fused style feature and reducing the influence of features unrelated to the style, so that the obtained second fused style feature pays more attention to the part related to the style.

[0117] In one possible implementation, the feature processing network includes a fourth self-attention layer, and the first fused style feature is input into the fourth self-attention layer to obtain the second fused style feature output by the fourth self-attention layer. Figure 7 As shown, the feature processing network 303 includes a fourth self-attention layer 32.

[0118] 407. The computer device performs an upsampling operation on the content feature and the second fused style feature through a first upsampling network of the first image generation model to obtain a target image feature.

[0119] like Figure 3 As shown, the first image generation model includes a first upsampling network. The computer device inputs the content feature and the second fused style feature into the first upsampling network, and the first upsampling network outputs the target image feature.

[0120] In one possible implementation, the first upsampling network includes a second residual layer and an attention layer. The second residual layer is connected to the attention layer, and the output of the second residual layer is the input of the attention layer. The second residual layer is used to perform a residual processing operation, and the residual processing operation is used to retain the features before the residual. The attention layer is used to perform an attention encoding operation, which refers to the process of encoding input data using an attention mechanism. In this process, the model dynamically generates a set of encoding vectors based on the relevance of different parts of the input data to the current task. These encoding vectors not only contain the original information of the input data, but also incorporate the model's understanding and attention to the input data.

[0121] So, if Figure 8 As shown, step 407 includes the following steps 4071 and 4072.

[0122] 4071. Map the first noise feature to a second residual feature through a second residual layer of the first upsampling network, where the first noise feature is used to characterize random noise.

[0123] The second residual layer can be a ResNet network structure, and the basic unit of ResNet is a residual block, which is composed of two or more layers of convolution, and jump connections are added between these layers. These jump connections directly add the input to the output, thereby realizing the transfer of residuals.

[0124] The computer device obtains a preset or randomly generated first noise feature, inputs the first noise feature to the second residual layer, and the second residual layer outputs a second residual feature. Optionally, the first noise feature can be a noise feature of random Gaussian noise, etc.

[0125] 4072. Perform an attention encoding operation on the content feature, the second fused style feature, and the second residual feature through the attention layer of the first upsampling network to obtain a target image feature.

[0126] The computer device inputs the content feature, the second fused style feature and the second residual feature into the attention layer, and the attention layer outputs the target image feature.

[0127] Optionally, the attention layer includes a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer. The fourth self-attention layer is connected to the first cross-attention layer, and the output of the fourth self-attention layer is the input of the first cross-attention layer. The first cross-attention layer is connected to the second cross-attention layer, and the output of the first cross-attention layer is the input of the second cross-attention layer.

[0128] Then, step 4072 includes: performing a self-attention encoding operation on the second residual feature through the fourth self-attention layer to obtain a third residual feature; performing a cross-attention encoding operation on the third residual feature and the second fusion style feature through the first cross-attention layer to obtain a second intermediate feature; performing a cross-attention encoding operation on the content feature and the second intermediate feature through the second cross-attention layer to obtain a target image feature.

[0129] Among them, the process of performing the self-attention encoding operation on the second residual feature is the same as the process of performing the self-attention encoding operation on the image coding feature mentioned above, and will not be repeated here.

[0130] Among them, the cross attention mechanism is a variant of the attention mechanism, which is used to establish associations between two different input data and calculate attention weights. This mechanism enables the model to dynamically focus on the part of one input data that is related to another input data, thereby effectively fusing the information of the two input data.

[0131] Exemplarily, the network structure of the first cross-attention layer and the network structure of the second cross-attention layer are the same, the model parameters of the first cross-attention layer and the model parameters of the second cross-attention layer are different, or the model parameters of the first cross-attention layer and the model parameters of the second cross-attention layer may also be the same.

[0132] Exemplarily, the process of cross-attention encoding in the first cross-attention layer includes: using a second mapping matrix to map the third residual feature to a second query feature; using a third mapping matrix to map the second fusion style feature to a second key feature and a second value feature; performing a normalization operation on the product of the transpose of the second query feature and the second key feature and a scaling factor to obtain a normalized feature, and multiplying the normalized feature by the second value feature to obtain a second intermediate feature.

[0133] Exemplarily, the process of cross-attention encoding in the second cross-attention layer includes: using a fourth mapping matrix to map the second intermediate feature to a third query feature; using a fifth mapping matrix to map the content feature to a third key feature and a third value feature; performing a normalization operation on the product of the transpose of the third query feature and the third key feature and a scaling factor to obtain a normalized feature, and multiplying the normalized feature by the third value feature to obtain a target image feature.

[0134] like Figure 7 As shown in (1), the first upsampling network 304 in the embodiment of the present application includes multiple first upsampling layers 33, each of which is composed of a second residual layer, a fourth self-attention layer, a first cross-attention layer and a second cross-attention layer. Figure 7 (1) in the figure only takes the example that the first upsampling network 304 includes multiple first upsampling layers 33. In practice, the first upsampling network 304 may also include only one first upsampling layer 33, or it can be understood that the first upsampling network 304 only includes a set of second residual layer, fourth self-attention layer, first cross-attention layer and second cross-attention layer.

[0135] In this implementation, by introducing the residual layer, the gradient vanishing problem that occurs in deep networks can be avoided, thereby increasing the expressiveness of features. In addition, by encoding content features, style features, and residual features through the attention layer, the importance of each feature in the image generation process can be adaptively adjusted, thereby enhancing the expressiveness of target image features and improving the accuracy of the generated target image.

[0136] In addition, the cross-attention mechanism is used to cross-attention encode the style features and content features, which effectively realizes the deep fusion between content and style, enables more refined collaborative processing of content features and style features in the image generation process, and improves the adaptability of the image generation process based on style transfer.

[0137] 408. The computer device performs a decoding operation on the target image features through the decoding network of the first image generation model to obtain the target image.

[0138] like Figure 3 As shown, the first image generation model includes a decoding network. The computer device inputs the target image features into the decoding network, and the decoding network outputs the target image. The target image features include features of the content dimension and features of the style dimension, and can guide the image generation process in the content dimension to ensure that the generated target image has the same content as the content information, and can guide the image generation process in the style dimension to ensure that the generated target image has the same style as multiple reference images.

[0139] Exemplarily, the decoding network is a decoder in a VAE model architecture.

[0140] In this implementation, the first upsampling network performs upsampling operations on content features and style features, which can gradually restore image details and reconstruct image structural information during the image generation process. In addition, the target image features include content features and style features, realizing the combination of content and style. Then, performing a decoding operation on the target image features to obtain the target image can ensure that the content and style of the target image are more in line with the expected generation target.

[0141] The embodiment of the present application provides an image generation method, which performs balanced calculation on the style features of multiple reference images input in parallel to realize a zero-shot-based image style transfer scheme. By inputting an indefinite number of reference images with the same style, the style features of the multiple reference images can be automatically extracted, and under the guidance of the style features, a specified content image with the same style as the multiple reference images is generated.

[0142] Fig. 9 is a comparison diagram of a reference image and a target image provided in an embodiment of the present application, such as Fig. 9As shown, the first row is picture book style, the first three columns of the first row are reference images, and the fourth column is the target image generated based on the three picture book style reference images, and the target image is picture book style. The second row is pixel style, the first three columns of the second row are reference images, and the fourth column is the target image generated based on the three pixel style reference images, and the target image is pixel style. The third row is pixel style, the first three columns of the third row are reference images, and the fourth column is the target image generated based on the three pixel style reference images, and the target image is pixel style. The fourth row is toy style, the first three columns of the fourth row are reference images, and the fourth column is the target image generated based on the three toy style reference images, and the target image is toy style. The fifth row is stick figure style, the first three columns of the fifth row are reference images, and the fourth column is the target image generated based on the three stick figure style reference images, and the target image is stick figure style.

[0143] The method provided by the embodiment of the present application provides multiple reference images of the specified style when it is necessary to generate an image of a specified style. First, the style features of multiple different reference images are extracted in parallel, and then the style features of multiple different reference images are fused and self-attention encoded, so as to capture the common and key style features in multiple different reference images, avoid being affected by the semantic features of the reference image itself, and ensure the global relevance of the extracted style features. Then, the target image is generated using the content features of the content information and the extracted style features, so that the content of the target image can be guaranteed to be the same as the content indicated by the content information, and the style of the target image is the same as the specified style. Therefore, the present application implements a zero-sample style transfer image generation scheme, and only needs to provide multiple reference images of any specified style to generate such a specified style image, which is not limited to generating only one specific style image, and improves the flexibility and versatility of the image generation method. Moreover, under the guidance of the style features extracted from multiple reference images, the accuracy of the generated image can be guaranteed.

[0144] Moreover, the present application can control the style of the generated target image by providing multiple reference images with the same style, without the need for additional stylization training of the model. The method of the invention can be mainly applied to the process of generating stylized images, and the style representation of all stylized images to be generated can be controlled by the input style image. At the same time, no additional stylization training is required, and fast style transfer can be achieved.

[0145] In addition, this solution constructs a processing mechanism for style transfer using multiple reference images. This processing mechanism allows receiving an indefinite number of reference images as input to the first image generation model. This indefinite number of inputs can make the first image generation model more extensible. The style features of multiple reference images can accurately capture the specified style. The more reference images provided, the more accurate the extracted style features, and the more accurate the generated target image.

[0146] In one embodiment, in the above Figure 4 Based on the example, Figure 3 As shown, the first downsampling network of the first image generation model includes multiple first downsampling layers, and the first upsampling network includes multiple first upsampling layers. The image generation process can refer to the following Fig.10 Embodiment of the invention. Fig.10 is a flowchart of another image generation method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Fig.10 , the method comprises the following steps.

[0147] 1001. A computer device obtains content information and multiple reference images, where the multiple reference images have the same reference style, and the content information is used to indicate the content of an image to be generated.

[0148] The process of obtaining content information and multiple reference images in step 1001 is the same as the process of obtaining content information and multiple reference images in step 401 above, and will not be repeated here.

[0149] 1002. The computer device extracts content features corresponding to the content information through a feature extraction network of the first image generation model.

[0150] The process of extracting content features in step 1002 is the same as the process of extracting content features in step 402 above, and will not be repeated here.

[0151] 1003. The computer device extracts reference style features corresponding to each reference image of a plurality of reference images through an encoding network of a first image generation model.

[0152] The process of extracting image coding features in step 1003 is the same as the process of extracting image coding features in step 403 above, and will not be repeated here.

[0153] like Figure 3 As shown, the first image generation model in the embodiment of the present application includes a first downsampling network, and the first downsampling network includes k first downsampling layers connected in sequence, where k is an integer greater than 1. The process of extracting the reference style feature through the first downsampling network is as follows step 1004.

[0154] 1004. The computer device performs a downsampling operation on the image coding features corresponding to each reference image through the first first downsampling layer among the k first downsampling layers to obtain the first reference style features corresponding to each reference image, and performs a downsampling operation on the m-1th first reference style features corresponding to each reference image through the mth first downsampling layer among the k first downsampling layers to obtain the mth reference style features corresponding to each reference image.

[0155] Here, m is an integer greater than 1 and not greater than k.

[0156] Taking a certain reference image as an example, the computer device inputs the image coding feature of the reference image into the first first downsampling layer to obtain the first reference style feature of the reference image. The first reference style feature of the reference image is input into the second first downsampling layer to obtain the second reference style feature of the reference image. The second reference style feature of the reference image is input into the third first downsampling layer to obtain the third reference style feature of the reference image. And so on, until the last first downsampling layer outputs the last reference style feature. That is, until the k-1th reference style feature of the reference image is input into the kth first downsampling layer to obtain the kth reference style feature of the reference image.

[0157] The computer device executes the above steps for each reference image, thereby obtaining k reference style features of each reference image.

[0158] In this implementation, multiple first downsampling layers are introduced to gradually extract the style features of the image at different scales, which improves the diversity and richness of the style features and is conducive to capturing the details and global style in the image. In addition, by downsampling layer by layer, each layer gradually compresses the feature representation of the image, removes unimportant information, and retains key features related to the style, thereby improving the ability to capture style features and processing efficiency, thereby generating richer and more refined style features in the image generation task based on style transfer.

[0159] In a possible implementation, each first downsampling layer includes a first residual layer and a feature encoding layer. Each first downsampling layer downsamples the reference style feature to obtain the reference style feature in the same manner as the above steps 4041-4042. Exemplarily, the feature encoding layer includes a first self-attention layer and a second self-attention layer.

[0160] That is, if Figure 6 As shown in (1), the first downsampling network 302 includes multiple first downsampling layers 31, each of which includes a first residual layer, a first self-attention layer and a second self-attention layer.

[0161] In one possible implementation, in order to make the model scalable and allow the model to support input of an indefinite number of reference images, a storage stack of reference style features is designed. Figure 3 As shown, the reference style features 13 output by the first downsampling network 302 are stored in the storage stack. The first downsampling network 302 includes multiple first downsampling layers. According to the network structure, the storage stack is stacked and stored according to the differences of the multiple first downsampling layers. The reference style features 13 output by different first downsampling layers are respectively stored in different layers of the storage stack, so as to facilitate the subsequent one-to-one processing according to the network layers in the feature processing network and the first upsampling network.

[0162] Fig.11 is a structural diagram of another first image generation model provided in an embodiment of the present application, such as Fig.11 As shown, multiple reference images including reference image A, reference image B and reference image C are taken as an example. The first latent features 12 of the three reference images are respectively extracted through the encoding network 301, and the first latent features 12 are also the above-mentioned image encoding features. The first latent features 12 of the three reference images are respectively input into the first downsampling network 302, and the first downsampling network 302 outputs k reference style features 13 of each reference image respectively, and the reference style features 13 output by different first downsampling layers are respectively stored in different layers of the storage stack.

[0163] like Figure 3 As shown, the first image generation model in the embodiment of the present application includes a feature processing network, and the feature processing network includes k processing layers, each processing layer includes a feature fusion layer and a third self-attention layer. It can be understood that the feature processing network includes k feature fusion layers and k third self-attention layers. The process of feature fusion and self-attention encoding through the feature processing network is as follows Step 1004 and Step 1005.

[0164] 1005. The computer device fuses the first reference style features corresponding to the multiple reference images respectively through the first feature fusion layer among the k feature fusion layers to obtain the first first fused style feature, and fuses the mth reference style features corresponding to the multiple reference images respectively through the mth feature fusion layer among the k feature fusion layers to obtain the mth first fused style feature.

[0165] Here, m is an integer greater than 1 and not greater than k.

[0166] The computer device inputs the first reference style feature of the multiple reference images into the first feature fusion layer to obtain the first first fused style feature. The second reference style feature of the multiple reference images is input into the second feature fusion layer to obtain the second first fused style feature. The third reference style feature of the multiple reference images is input into the third feature fusion layer to obtain the third first fused style feature. And so on, until the last feature fusion layer outputs the last first fused style feature. That is, until the kth reference style feature of the multiple reference images is input into the kth feature fusion layer to obtain the kth first fused style feature.

[0167] In this implementation, the style features of multiple reference images are fused, which can more accurately capture the features of the same style corresponding to multiple reference images, avoid the semantic interference of the image itself when using a single reference image, and cause style bias or distortion, thereby improving the accuracy of the extracted style features.

[0168] In a possible implementation, as in step 1004, the k reference style features of the plurality of reference images are stored in different layers of the storage stack according to the network structure. The computer device sequentially reads the first reference style feature to the kth reference style feature of the reference image in the storage stack according to the hierarchical structure of the storage stack.

[0169] In a possible implementation, the process of fusing the reference style features of multiple reference images in each feature fusion layer is the same as the above step 405, which will not be described in detail here.

[0170] 1006. The computer device performs a self-attention encoding operation on the first first fusion style feature through the first third self-attention layer among the k third self-attention layers to obtain the first second fusion style feature, and performs a self-attention encoding operation on the mth first fusion style feature through the mth third self-attention layer among the k third self-attention layers to obtain the mth second fusion style feature.

[0171] Here, m is an integer greater than 1 and not greater than k.

[0172] The computer device inputs the first first fusion style feature into the first third self-attention layer to obtain the first second fusion style feature. The second first fusion style feature is input into the second third self-attention layer to obtain the second second fusion style feature. The first first fusion style feature is input into the first third self-attention layer to obtain the first second fusion style feature. And so on, until the last third self-attention layer outputs the last second fusion style feature. That is, until the kth first fusion style feature is input into the kth third self-attention layer to obtain the kth second fusion style feature.

[0173] In this implementation, the self-attention mechanism is used to process the fused style features, which is conducive to capturing the common and key style features in multiple different reference images, making the extracted style features more significant and prominent, avoiding the influence of the semantic features of the reference image itself, and ensuring the accuracy of the extracted style features.

[0174] In a possible implementation, the process of each third self-attention layer performing self-attention encoding on the first fusion style feature is the same as the above step 406, which will not be repeated here.

[0175] In this implementation, a feature balance purification process based on the self-attention mechanism is added to each processing layer of the feature processing network, so that the model pays more attention to the common style features in multiple reference images, which is conducive to improving the model's ability to extract style features.

[0176] like Figure 3 As shown, the first image generation model in the embodiment of the present application includes a first upsampling network, and the first upsampling network includes k first upsampling layers connected in sequence. The number of second fusion style features is k, and k is an integer greater than 1. The process of extracting target image features through the first upsampling network is as follows step 1007.

[0177] 1007. The computer device performs an upsampling operation on the first noise feature, the content feature, and the first second fusion style feature through the first first upsampling layer among the k first upsampling layers to obtain the first target image feature, and performs an upsampling operation on the m-1th target image feature, the content feature, and the mth second fusion style feature through the mth first upsampling layer among the k first upsampling layers to obtain the mth target image feature.

[0178] Wherein, m is an integer greater than 1 and not greater than k. The first second fusion style feature is determined based on the feature output by the first first downsampling layer, and the mth second fusion style feature is determined based on the feature output by the mth first downsampling layer. It can be understood that the k first downsampling layers correspond one-to-one to the k first upsampling layers, the feature extracted based on the first first downsampling layer is the input of the first first upsampling layer, and the feature extracted based on the mth first downsampling layer is the input of the mth first upsampling layer.

[0179] The computer device inputs the first noise feature, the content feature, and the first second fused style feature into the first first upsampling layer to obtain the first target image feature. The first target image feature, the content feature, and the second second fused style feature are input into the second first upsampling layer to obtain the second target image feature. The second target image feature, the content feature, and the third second fused style feature are input into the third first upsampling layer to obtain the third target image feature. And so on, until the last first upsampling layer outputs the last target image feature. That is, until the k-1th target image feature, the content feature, and the kth second fused style feature are input into the kth first upsampling layer to obtain the kth target image feature.

[0180] In this implementation, multiple first upsampling layers are introduced to restore image details layer by layer, reconstruct image structural information, and gradually restore low-resolution features to high-resolution image features, which helps to improve the detail quality of the generated image. In addition, each first upsampling layer can combine content features with corresponding second fused style features for upsampling, which can ensure that content features and style features are gradually merged at different scales, further improving the accuracy of the generated image.

[0181] In one possible implementation, each first upsampling layer includes a second residual layer and an attention layer. The process of extracting target image features in each first downsampling layer is the same as the process of steps 4071-4072 above. Exemplarily, the attention layer includes a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer.

[0182] That is, if Figure 7 As shown in (1), the first upsampling network 304 includes multiple first upsampling layers 33, each of which includes a second residual layer, a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer.

[0183] 1008. The computer device performs a decoding operation on the kth target image feature through the decoding network of the first image generation model to obtain a target image.

[0184] The computer device inputs the kth target image feature into the decoding network, and the decoding network outputs the target image.

[0185] The method provided in the embodiment of the present application provides a mechanism for performing multi-layer style feature extraction on multiple reference images. In the multiple first downsampling layers and the multiple first upsampling layers in the U-Net model architecture, the potential features of each layer are extracted in turn. After multi-layer parallel feature extraction, the model can capture more accurate style features at multiple levels, which is conducive to improving the accuracy of the generated target image.

[0186] In one embodiment, the construction process of the first image generation model is as follows: Fig.12 Embodiment of the invention. Fig.12 is a flow chart of a model training method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device. Fig.12 , the method comprises the following steps.

[0187] 1201. A computer device obtains sample content information and a sample label image, wherein the content of the sample label image matches the content indicated by the sample content information.

[0188] The sample content information is the same as above Figure 4 The content information in the embodiments is the same and will not be repeated here.

[0189] The content of the sample label image matches the content indicated by the sample content information. For example, if the sample content information is a running child, then the sample label image includes a running child. The sample label image can be an image of any style, and the embodiment of the present application does not limit the style of the sample label image.

[0190] 1202. The computer device extracts the sample content feature corresponding to the sample content information through the second image generation model, extracts the first sample reference feature corresponding to the sample label image, adds the second noise feature to the first sample reference feature, and obtains the second sample reference feature, where the second noise feature is used to characterize random noise.

[0191] Fig.13 is a structural diagram of a second image generation model provided in an embodiment of the present application, such as Fig.13 As shown, the second image generation model includes an encoding network 301, a feature extraction network 306, a second downsampling network 307, a connection network 308, a second upsampling network 309 and a decoding network 305.

[0192] The computer device inputs the sample content information into the feature extraction network 306 to obtain the sample content feature. The sample label image 41 and the second noise feature are input into the encoding network 301, and the encoding network 301 extracts the first sample reference feature of the sample label image, and adds the second noise feature to the first sample reference feature, thereby obtaining the second sample reference feature 42.

[0193] Among them, the encoding network and feature extraction network of the second image generation model are the same as the encoding network and feature extraction network of the first image generation model, and will not be repeated here.

[0194] 1203. The computer device generates a sample target image based on the sample content features and using the second image generation model and guided by the second sample reference features.

[0195] like Fig.13 As shown, the computer device inputs the sample content feature and the second sample reference feature 42 into the second downsampling network 307, the potential feature output by the second downsampling network 307 is input into the connection network 308, the potential feature output by the connection network 308 is input into the second upsampling network 309, and the second upsampling network 309 outputs the sample target image feature 43. The computer device inputs the sample target image feature 43 into the decoding network 305, and the decoding network 305 decodes the sample target image feature 43 and outputs the sample target image 44.

[0196] Among them, the second sample reference feature 42 is a feature after noise is added. The second sample reference feature 42 is processed by the second downsampling network 307, the connection network 308 and the second upsampling network 309 to reduce noise, so as to obtain a clean sample target image feature 43. The sample target image feature 43 contains both the semantics of the sample content information and the semantics of the sample label image.

[0197] 1204. The computer device trains a second image generation model based on the sample target image and the sample label image to obtain a trained second image generation model.

[0198] The sample label image is a real image indicated by the sample content information, and the sample target image is obtained based on the second image generation model. The smaller the difference between the sample target image generated by the second image generation model and the sample label image, the more accurate the second image generation model is. Therefore, the computer device adjusts the model parameters of the second image generation model based on the sample target image and the sample label image to obtain a more accurate second image generation model. The training goal is to reduce the difference between the sample target image and the sample label image, thereby improving the image generation ability of the second image generation model.

[0199] In one possible implementation, the encoding network in the second image generation model is removed to obtain a third image generation model, where the third image generation model is used to generate a corresponding image based on any content information.

[0200] Among them, in the model training stage, an encoding network is needed to encode the sample label image, while in the model inference stage, there is no sample label image, and the sample label image does not need to be encoded, so the encoding network is no longer needed.

[0201] Fig.14 is a schematic diagram of the structure of a third image generation model provided in an embodiment of the present application, such as Fig.14As shown, the third image generation model includes a feature extraction network 306, a second downsampling network 307, a connection network 308, a second upsampling network 309 and a decoding network 305. In the task of generating an image matching the content information, the computer device inputs the content information into the feature extraction network 306 to obtain the content feature. The computer device inputs the third noise feature 81 into the second downsampling network 307, and inputs the content feature into the second downsampling network 307, the connection network 308 and the second upsampling network 309. After being processed by the second downsampling network 307, the connection network 308 and the second upsampling network 309, a denoised image feature 82 is obtained. The image feature 82 is input into the decoding network 305 to obtain a content image 83 matching the content information.

[0202] 1205. The computer device constructs the first image generation model based on the trained second image generation model.

[0203] After acquiring the trained second image generation model, the computer device adjusts the model structure of the second image generation model to obtain a first image generation model suitable for style transfer.

[0204] In one possible implementation, the trained first image generation model includes an encoding network, a feature extraction network, a second downsampling network, a second upsampling network, and a decoding network connected in sequence. The second downsampling network includes a first residual layer, a first self-attention layer, and a third cross-attention layer connected in sequence, and the second upsampling network includes a second residual layer, a fourth self-attention layer, and a first cross-attention layer connected in sequence. Fig.15 As shown, step 1205 includes steps 1215 to 1245.

[0205] 1215. Copy the first self-attention layer to obtain the second self-attention layer.

[0206] 1225. Copy the first cross attention layer to obtain the second cross attention layer.

[0207] 1235. Copy the fourth self-attention layer to obtain the third self-attention layer, and build a feature processing network based on the feature fusion layer and the third self-attention layer.

[0208] The feature fusion layer is used to perform an averaging operation or a weighted averaging operation. The feature fusion layer itself does not have model parameters and is only used to perform simple calculation operations, so no training is required. The computer device connects the third self-attention layer behind the feature fusion layer to obtain a feature processing network.

[0209] 1245. In the trained second image generation model, the third cross-attention layer of the second down-sampling network is replaced by the second self-attention layer to obtain the first down-sampling network, the second cross-attention layer is added after the first cross-attention layer of the second up-sampling network to obtain the first up-sampling network, and a feature processing network is added between the first down-sampling network and the first up-sampling network to obtain the first image generation model.

[0210] The computer device replaces the third cross-attention layer in the second down-sampling network with the second self-attention layer to obtain the first down-sampling network, and connects the second cross-attention layer behind the first cross-attention layer of the second up-sampling network to obtain the first up-sampling network. The feature processing network is added between the first down-sampling network and the first up-sampling network, and the output of the first down-sampling network is the input of the feature processing network, and the output of the feature processing network is the input of the first up-sampling network. The first image generation model is obtained by adjusting the model structure of the second image generation model without retraining the first image generation model.

[0211] For example, Figure 6 As shown in (2) in , the second downsampling network 307 of the second image generation model includes k second downsampling layers 61, each of which includes a first residual layer, a first self-attention layer, and a third cross-attention layer. For each second downsampling layer 61, the first self-attention layer of the second downsampling layer 61 is copied to obtain a second self-attention layer, and the third cross-attention layer of the second downsampling layer 61 is replaced with the second self-attention layer to obtain a first downsampling layer 31. The above adjustment operation is performed for each second downsampling layer 61, and the following can be obtained: Figure 6 The first downsampling network 302 in the first image generation model shown in (1).

[0212] For example, Figure 7 As shown in (2) in FIG. 1 , the second upsampling network 309 of the second image generation model includes k second upsampling layers 71, each of which includes a second residual layer, a fourth self-attention layer, and a first cross-attention layer. For each second upsampling layer 71, the first cross-attention layer of the second upsampling layer 71 is copied to obtain a second cross-attention layer, and the second cross-attention layer is added after the first cross-attention layer of the second upsampling layer 71 to obtain a first upsampling layer 33. The above adjustment operation is performed for each second upsampling layer 71, and the following can be obtained: Figure 7 The first upsampling network 304 in the first image generation model shown in (1).

[0213] Exemplarily, the second downsampling network 307 of the second image generation model includes k second downsampling layers 61, and the second upsampling network 309 of the second image generation model includes k second upsampling layers 71. The feature extraction network includes k processing layers. Taking the mth processing layer as an example, the computer device copies the fourth self-attention layer of the mth second upsampling layer 71 to obtain the mth third self-attention layer, and constructs the mth processing layer based on the mth feature fusion layer and the mth third self-attention layer. That is, the mth processing layer is composed of the mth feature fusion layer and the mth third self-attention layer. Using the above method, k processing layers can be constructed in sequence, thereby obtaining a feature processing network including k processing layers. Among them, the mth processing layer is added between the mth second downsampling layer and the mth second upsampling layer.

[0214] In this implementation, by copying the network layers in the trained second image generation model, a new first image generation model is constructed based on the trained second image generation model and the copied network layers, thereby reducing the time and resource costs required for training the new model and improving the efficiency of building the image generation model.

[0215] In the embodiment of the present application, the sample content information and the sample label image are used to train the second image generation model to improve the image generation capability of the second image generation model. Furthermore, based on the trained second image generation model, the first image generation model is constructed, which is equivalent to reusing the image generation capability of the second image generation model to construct a new first image generation model. The first image generation model can be used to generate images of any specified style. Therefore, there is no need to additionally train models for generating images of different styles for different styles, which saves the time and resource costs consumed by model training and improves processing efficiency and the versatility of the model.

[0216] The image generation method provided in the embodiment of the present application can be applied to any scenario of generating an image of a specified style. For example, in the scenario of generating a portrait photo of a specified style, the image generation process is described in detail below. Fig.16 Embodiment of the invention. Fig.16 is a flowchart of another image generation method provided in an embodiment of the present application. The embodiment of the present application is executed by a computer device, such as Fig.16 As shown, the method includes the following steps.

[0217] 1601. Obtain a reference image photo and multiple reference style photos.

[0218] The reference image photo includes an image, which may be an image of a person or an animal, etc. The plurality of reference style photos have the same reference style, for example, the reference style refers to an animation style, etc.

[0219] 1602. Extract image features corresponding to the reference image photo through the first image generation model, and extract image coding features corresponding to each reference style photo in the plurality of reference style photos.

[0220] 1603. Perform a downsampling operation on the image encoding features of each reference style photo through the first image generation model to obtain reference style features corresponding to each reference style photo.

[0221] 1604. Using the first image generation model, fuse the reference style features corresponding to the plurality of reference style photos to obtain a first fused style feature.

[0222] 1605. Perform a self-attention encoding operation on the first fused style feature through the first image generation model to obtain a second fused style feature.

[0223] 1606. Perform an upsampling operation on the second fused style feature and image feature through the first image generation model to obtain a target image feature.

[0224] 1607. Perform a decoding operation on the target image features through the first image generation model to obtain a target image photo, which has the same image as the reference image photo and the same style as the reference style photo.

[0225] In an embodiment of the present application, a user provides a reference image photo that needs to be stylized, and multiple reference style photos that can represent the desired style. The first image generation model can automatically extract style features from the input multiple reference style photos, and transfer the style to the input reference image photo, thereby generating a stylized target image photo. This solution can be applied to the function expansion part of the magic camera, which is convenient for users to customize stylized image photos and realize the expansion of photo gameplay.

[0226] In one embodiment, the image generation method of the embodiment of the present application can also be applied to the scenario of making DIY game characters. The user provides a description of the game character that needs to be stylized, as well as multiple game style photos that can represent the desired style. The first image generation model can automatically extract style features from the multiple game style photos input, and use the style features to guide the process of intelligently generating game characters based on the game character description, thereby realizing the custom design of the game character style. In electronic games, by opening the DIY mode, users can input pictures of the desired game style by themselves, so that users can design game characters by themselves, which increases the fun of the game and provides great help for the production of the game.

[0227] In one embodiment, the image generation method of the embodiment of the present application can also be applied to the scene of restoring the painting style of an artistic painter. For example, a user provides multiple painting photos and reference scene photos that need to be stylized, and the first image generation model can automatically extract style features from the input multiple painting photos, use the style features to guide the style transfer of the reference scene photos, and obtain the target scene photos that match the style of the multiple painting photos, which is conducive to quickly creating various paintings with the style.

[0228] In a related technology, if you want to generate an image of a specified style, you need to collect a batch of high-quality images of the specified style as training sample data in advance, and then input them into the Stable Diffusion model. Use the Stable Diffusion model of this batch of training sample data to inject the concept of this style into the Stable Diffusion model, and then use the Stable Diffusion model to generate images of this specified style. However, this method usually requires a large number of images as training sample data, and Stable Diffusion models for generating images of different styles need to be trained separately for different styles, which consumes a lot of time and processing resources and increases the processing burden.

[0229] The first image generation model provided in the embodiment of the present application can implement a zero-shot style transfer image generation method. For any image of a specified style that you want to generate, you only need to provide a few images of the specified style. There is no need to additionally train different models for different styles. The first image generation model can capture and balance the style features of multiple images of the specified style, so as to guide the first image generation model to accurately generate images of the specified style. Compared with the above-mentioned related technologies, it can save more time and processing resources and reduce processing costs.

[0230] In another related technology, a style image is provided, and a style image is encoded into image features, which are input into the model along with text features describing the text, thereby performing style guidance control on the image generation process. Although this method can pass the style semantics in the style image to the model, because the style semantics are not purified during the encoding process, the image features input into the model include other semantics (such as people, scenery, etc.) in addition to the style semantics, thereby affecting the model's accurate generation of stylized images. In addition, using only one style image for encoding can easily lead to inaccurate style capture, thereby affecting the style accuracy of the generated image.

[0231] The first image generation model provided in the embodiment of the present application constructs a feature balance mechanism for multiple style images, supports receiving an indefinite number of style images as model input, and this indefinite number of inputs can make the model more extensible. In addition, it can accurately capture common style features through multiple style images. The more style images are input, the more accurate the style features are extracted, which is conducive to guiding the process of style transfer to be more accurate.

[0232] In another related technology, multiple style images are provided, one of which is used as a basic style image, and other auxiliary style images are used to perform serial calculations with the basic style image to extract common style features in multiple style images. However, this method of combining the basic style image with the auxiliary style image depends to a large extent on the style representation of the basic style image. If the style representation of the basic style image is not strong, the final extracted style features will be inaccurate. In addition, this unbalanced serial extraction mechanism reduces the role of the auxiliary style image, thereby reducing the utilization rate of the style image.

[0233] The first image generation model provided in the embodiment of the present application performs balanced calculations on multiple style images input in parallel without distinction, directly extracts common style features from multiple style images through a balanced calculation mechanism, and purifies the style features of multiple style images in parallel, thereby avoiding the limitation of a single style image, being conducive to accurately capturing the style features of multiple style images, improving the accuracy of the extracted style features, and thereby ensuring the accuracy of image generation.

[0234] Fig.17 is a structural diagram of an image generating device provided in an embodiment of the present application. Fig.17 , the device comprises: An acquisition module 1701 is used to acquire content information and multiple reference images, where the multiple reference images have the same reference style, and the content information is used to indicate the content of the image to be generated; A feature extraction module 1702 is used to extract content features corresponding to the content information, and to extract reference style features corresponding to each reference image in the plurality of reference images; A feature fusion module 1703 is used to fuse the reference style features corresponding to the multiple reference images to obtain a first fused style feature; A self-attention encoding module 1704 is used to perform a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, where the second fused style feature is used to represent the reference style; The image generation module 1705 is used to generate a target image based on the content feature and guided by the reference style represented by the second fusion style feature, wherein the content of the target image matches the content indicated by the content information.

[0235] The image generation device provided by the embodiment of the present application provides multiple reference images of the specified style when it is necessary to generate an image of a specified style. First, the style features of multiple different reference images are extracted in parallel, and then the style features of the multiple different reference images are fused and self-attention encoded, so as to capture the common and key style features in multiple different reference images, avoid being affected by the semantic features of the reference image itself, and ensure the global relevance of the extracted style features. Then, the target image is generated by using the content features of the content information and the extracted style features, so as to ensure that the content of the target image is the same as the content indicated by the content information, and the style of the target image is the same as the specified style. Therefore, the present application realizes a zero-sample style transfer image generation scheme, and only needs to provide multiple reference images of any specified style to generate such a specified style image, which is not limited to generating only one specific style image, and improves the flexibility and versatility of the image generation method. Moreover, under the guidance of the style features extracted from multiple reference images, the accuracy of the generated image can be guaranteed.

[0236] Alternatively, see Fig.18 , the first image generation model includes an encoding network and a first downsampling network; the feature extraction module 1702 includes: The encoding unit 1712 is used to perform an encoding operation on each reference image through an encoding network to obtain image encoding features corresponding to each reference image; The down-sampling unit 1722 is used to perform a down-sampling operation on the image coding features corresponding to each reference image through a first down-sampling network to obtain a reference style feature corresponding to each reference image.

[0237] Alternatively, see Fig.18 , the first downsampling network includes a first residual layer and a feature encoding layer; the downsampling unit 1722 is used to: Through the first residual layer, the image coding features corresponding to each reference image are mapped to the first residual features corresponding to each reference image; Through the feature encoding layer, an attention encoding operation is performed on the first residual feature corresponding to each reference image to obtain a reference style feature corresponding to each reference image.

[0238] Alternatively, see Fig.18 , the feature encoding layer includes a first self-attention layer and a second self-attention layer; a downsampling unit 1722, for: Through the first self-attention layer, a self-attention encoding operation is performed on the first residual feature corresponding to each reference image to obtain a first intermediate feature corresponding to each reference image; Through the second self-attention layer, a self-attention encoding operation is performed on the first intermediate features corresponding to each reference image to obtain the reference style features corresponding to each reference image.

[0239] Alternatively, see Fig.18 , the first downsampling network includes k first downsampling layers connected in sequence, where k is an integer greater than 1; a downsampling unit 1722, configured to: Performing a downsampling operation on the image coding features corresponding to each reference image through the first first downsampling layer of the k first downsampling layers, so as to obtain the first reference style features corresponding to each reference image; A downsampling operation is performed on the m-1th first reference style feature corresponding to each reference image through the mth first downsampling layer among the k first downsampling layers to obtain the mth reference style feature corresponding to each reference image, where m is an integer greater than 1 and not greater than k.

[0240] Alternatively, see Fig.18 The first image generation model includes a feature processing network, the feature processing network includes k feature fusion layers; a feature fusion module 1703, used to: By fusing the first reference style features corresponding to the multiple reference images respectively through the first feature fusion layer among the k feature fusion layers, a first first fused style feature is obtained; The mth reference style features corresponding to multiple reference images are fused through the mth feature fusion layer among the k feature fusion layers to obtain the mth first fused style feature.

[0241] Alternatively, see Fig.18 , the feature processing network also includes k third self-attention layers; a self-attention encoding module 1704, which is used to: Performing a self-attention encoding operation on the first first fused style feature through the first third self-attention layer among the k third self-attention layers to obtain the first second fused style feature; Through the mth third self-attention layer among the k third self-attention layers, a self-attention encoding operation is performed on the mth first fusion style feature to obtain the mth second fusion style feature.

[0242] Alternatively, see Fig.18 , the first image generation model includes a decoding network and a first upsampling network; the image generation module 1705 is used to: An upsampling unit 1715 is used to perform an upsampling operation on the content feature and the second fused style feature through a first upsampling network to obtain a target image feature; The decoding unit 1725 is used to perform a decoding operation on the target image features through a decoding network to obtain a target image.

[0243] Alternatively, see Fig.18 , the first upsampling network includes a second residual layer and an attention layer; an upsampling unit 1715, for: Mapping the first noise feature to a second residual feature through a second residual layer, where the first noise feature is used to characterize random noise; Through the attention layer, attention encoding operations are performed on the content features, the second fused style features, and the second residual features to obtain the target image features.

[0244] Alternatively, see Fig.18 , the attention layer includes a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer; an upsampling unit 1715, for: Through the fourth self-attention layer, a self-attention encoding operation is performed on the second residual feature to obtain a third residual feature; Through the first cross attention layer, a cross attention encoding operation is performed on the third residual feature and the second fusion style feature to obtain a second intermediate feature; Through the second cross-attention layer, a cross-attention encoding operation is performed on the content features and the second intermediate features to obtain the target image features.

[0245] Alternatively, see Fig.18 The first image generation model further includes a first downsampling network, the first downsampling network includes k first downsampling layers connected in sequence, the first upsampling network includes k first upsampling layers connected in sequence, the number of the second fusion style features is k, and k is an integer greater than 1; the upsampling unit 1715 is used to: Performing an upsampling operation on the first noise feature, the content feature, and the first second fused style feature through the first first upsampling layer of the k first upsampling layers to obtain a first target image feature, wherein the first second fused style feature is determined based on the feature output by the first first downsampling layer, and the first noise feature is used to characterize random noise; An upsampling operation is performed on the m-1th target image feature, the content feature, and the mth second fused style feature through the mth first upsampling layer among the k first upsampling layers to obtain the mth target image feature, the mth second fused style feature is determined based on the feature output by the mth first downsampling layer, and m is an integer greater than 1 and not greater than k.

[0246] Alternatively, see Fig.18 , a decoding unit 1725 is used to: Through the decoding network, a decoding operation is performed on the kth target image feature to obtain the target image.

[0247] Alternatively, see Fig.18 The device further includes a model training module 1706, which is used to: Acquire sample content information and a sample label image, wherein the content of the sample label image matches the content indicated by the sample content information; Extracting sample content features corresponding to the sample content information through the second image generation model, extracting first sample reference features corresponding to the sample label image, adding second noise features to the first sample reference features to obtain second sample reference features, where the second noise features are used to characterize random noise; Generate a sample target image based on the sample content feature by using a second image generation model and guided by the second sample reference feature; Based on the sample target image and the sample label image, training a second image generation model to obtain a trained second image generation model; The apparatus further comprises a model building module 1707, which is used to: Based on the trained second image generation model, a first image generation model is constructed.

[0248] Alternatively, see Fig.18 The trained first image generation model includes an encoding network, a second downsampling network, a second upsampling network and a decoding network connected in sequence, the second downsampling network includes a first residual layer, a first self-attention layer and a third cross-attention layer connected in sequence, and the second upsampling network includes a second residual layer, a fourth self-attention layer and a first cross-attention layer connected in sequence; Model building module 1707, used to: Copy the first self-attention layer to get the second self-attention layer; copy the first cross-attention layer to get the second cross-attention layer; Copy the fourth self-attention layer to obtain the third self-attention layer; construct a feature processing network based on the feature fusion layer and the third self-attention layer; In the trained second image generation model, the third cross-attention layer of the second downsampling network is replaced by the second self-attention layer to obtain the first downsampling network, the second cross-attention layer is added after the first cross-attention layer of the second upsampling network to obtain the first upsampling network, and a feature processing network is added between the first downsampling network and the first upsampling network to obtain the first image generation model.

[0249] It should be noted that the image generation device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the image generation device provided in the above embodiment and the image generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0250] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the image generation method of the above embodiment.

[0251] Optionally, the computer device is provided as a terminal. Fig.19 A schematic structural diagram of a first terminal 101 provided by an exemplary embodiment of the present application is shown.

[0252] The first terminal 101 includes: a first processor 1901 and a first memory 1902 .

[0253] The first processor 1901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The first processor 1901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The first processor 1901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the first processor 1901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the first processor 1901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0254] The first memory 1902 may include one or more computer-readable storage media, which may be non-transitory. The first memory 1902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the first memory 1902 is used to store at least one computer program, which is used to be possessed by the first processor 1901 to implement the image generation method provided in the method embodiment of the present application.

[0255] In some embodiments, the first terminal 101 may further optionally include: a peripheral device interface 1903 and at least one peripheral device. The first processor 1901, the first memory 1902 and the peripheral device interface 1903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1903 via a bus, a signal line or a circuit board. Optionally, the peripheral device includes: at least one of a radio frequency circuit 1904, a display screen 1905, a camera assembly 1906, an audio circuit 1907 and a power supply 1908.

[0256] The peripheral device interface 1903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the first processor 1901 and the first memory 1902. In some embodiments, the first processor 1901, the first memory 1902, and the peripheral device interface 1903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the first processor 1901, the first memory 1902, and the peripheral device interface 1903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0257] The radio frequency circuit 1904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1904 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1904 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1904 can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1904 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0258] The display screen 1905 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1905 is a touch display screen, the display screen 1905 also has the ability to collect touch signals on the surface or above the surface of the display screen 1905. The touch signal can be input as a control signal to the first processor 1901 for processing. At this time, the display screen 1905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1905 can be one, which is set on the front panel of the first terminal 101; in other embodiments, the display screen 1905 can be at least two, which are respectively set on different surfaces of the first terminal 101 or are folded; in other embodiments, the display screen 1905 can be a flexible display screen, which is set on the curved surface or folded surface of the first terminal 101. Even, the display screen 1905 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 1905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0259] The camera component 1906 is used to capture images or videos. Optionally, the camera component 1906 includes a front camera and a rear camera. The front camera is arranged on the front panel of the first terminal 101, and the rear camera is arranged on the back of the first terminal 101. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera component 1906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0260] The audio circuit 1907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the first processor 1901 for processing, or input them into the radio frequency circuit 1904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the first terminal 101. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the first processor 1901 or the radio frequency circuit 1904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1907 may also include a headphone jack.

[0261] The power supply 1908 is used to power various components in the first terminal 101. The power supply 1908 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1908 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0262] Those skilled in the art will understand that Fig.19 The structure shown in the figure does not constitute a limitation on the first terminal 101, and may include more or less components than those shown in the figure, or combine certain components, or adopt a different component arrangement.

[0263] Optionally, the computer device is provided as a server. Fig. 20This is a structural diagram of a server provided in an embodiment of the present application. The server 102 may have relatively large differences due to different configurations or performances, and may include one or more second processors (Central Processing Units, CPU) 2001 and one or more second memories 2002, wherein at least one computer program is stored in the second memory 2002, and the at least one computer program is loaded and executed by the second processor 2001 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0264] An embodiment of the present application further provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the image generation method of the above embodiment.

[0265] An embodiment of the present application further provides a computer program product, including a computer program, wherein the computer program is loaded and executed by a processor to implement the operations performed by the image generation method of the above embodiment.

[0266] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0267] The above description is only an optional embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire content information and a plurality of reference images, wherein the plurality of reference images have the same reference style, and the content information is used to indicate the content of the image to be generated; Extracting content features corresponding to the content information, and extracting reference style features corresponding to each reference image in the plurality of reference images; Fusing the reference style features respectively corresponding to the multiple reference images to obtain a first fused style feature; Performing a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, where the second fused style feature is used to characterize the reference style; Guided by the reference style represented by the second fused style feature and based on the content feature, a target image is generated, wherein the content of the target image matches the content indicated by the content information.

2. The method according to claim 1, characterized in that: The method is performed by a first image generation model, wherein the first image generation model includes an encoding network and a first downsampling network; extracting a reference style feature corresponding to each reference image in the plurality of reference images includes: Performing encoding operation on each reference image through the encoding network to obtain image encoding features corresponding to each reference image; Through the first down-sampling network, a down-sampling operation is performed on the image encoding features corresponding to each reference image to obtain the reference style features corresponding to each reference image.

3. The method according to claim 2, characterized in that The first downsampling network includes a first residual layer and a feature encoding layer; the first downsampling network performs a downsampling operation on the image encoding features corresponding to each reference image to obtain the reference style features corresponding to each reference image, including: Mapping the image coding features corresponding to each of the reference images to first residual features corresponding to each of the reference images through the first residual layer; Through the feature encoding layer, an attention encoding operation is performed on the first residual feature corresponding to each reference image to obtain a reference style feature corresponding to each reference image.

4. The method according to claim 3, characterized in that The feature coding layer includes a first self-attention layer and a second self-attention layer; the feature coding layer performs an attention coding operation on the first residual feature corresponding to each reference image to obtain a reference style feature corresponding to each reference image, including: Performing a self-attention encoding operation on the first residual feature corresponding to each reference image through the first self-attention layer to obtain a first intermediate feature corresponding to each reference image; Through the second self-attention layer, a self-attention encoding operation is performed on the first intermediate features corresponding to each reference image to obtain the reference style features corresponding to each reference image.

5. The method according to claim 2, characterized in that: The first downsampling network includes k first downsampling layers connected in sequence, where k is an integer greater than 1; performing a downsampling operation on the image coding features corresponding to each reference image through the first downsampling network to obtain the reference style features corresponding to each reference image, including: Performing a downsampling operation on the image coding features corresponding to each of the reference images through a first first downsampling layer among the k first downsampling layers, so as to obtain a first reference style feature corresponding to each of the reference images; A downsampling operation is performed on the m-1th first reference style feature corresponding to each reference image through the mth first downsampling layer among the k first downsampling layers to obtain the mth reference style feature corresponding to each reference image, where m is an integer greater than 1 and not greater than k.

6. The method according to claim 5, characterized in that The first image generation model includes a feature processing network, and the feature processing network includes k feature fusion layers; the fusing the reference style features corresponding to the multiple reference images to obtain the first fused style feature includes: fusing the first reference style features respectively corresponding to the multiple reference images through the first feature fusion layer among the k feature fusion layers to obtain a first first fused style feature; The mth reference style features respectively corresponding to the multiple reference images are fused through the mth feature fusion layer among the k feature fusion layers to obtain the mth first fused style feature.

7. The method according to claim 6, characterized in that The feature processing network further includes k third self-attention layers; performing a self-attention encoding operation on the first fused style feature to obtain a second fused style feature includes: Performing a self-attention encoding operation on the first first fused style feature through a first third self-attention layer among the k third self-attention layers to obtain a first second fused style feature; A self-attention encoding operation is performed on the mth first fusion style feature through the mth third self-attention layer among the k third self-attention layers to obtain the mth second fusion style feature.

8. The method according to claim 1, characterized in that The method is performed by a first image generation model, wherein the first image generation model includes a decoding network and a first upsampling network; the method generates a target image based on the content feature and guided by the style represented by the second fusion style feature, including: Performing an upsampling operation on the content feature and the second fused style feature through the first upsampling network to obtain a target image feature; Through the decoding network, a decoding operation is performed on the target image features to obtain the target image.

9. The method according to claim 8, characterized in that The first upsampling network includes a second residual layer and an attention layer; the first upsampling network performs an upsampling operation on the content feature and the second fused style feature to obtain a target image feature, including: Mapping the first noise feature to a second residual feature through the second residual layer, where the first noise feature is used to characterize random noise; Through the attention layer, an attention encoding operation is performed on the content feature, the second fusion style feature and the second residual feature to obtain the target image feature.

10. The method according to claim 9, characterized in that The attention layer includes a fourth self-attention layer, a first cross-attention layer, and a second cross-attention layer; the attention layer performs an attention encoding operation on the content feature, the second fusion style feature, and the second residual feature to obtain the target image feature, including: Performing a self-attention encoding operation on the second residual feature through the fourth self-attention layer to obtain a third residual feature; Performing a cross-attention encoding operation on the third residual feature and the second fused style feature through the first cross-attention layer to obtain a second intermediate feature; Through the second cross-attention layer, a cross-attention encoding operation is performed on the content feature and the second intermediate feature to obtain the target image feature.

11. The method according to claim 8, characterized in that The first image generation model further includes a first downsampling network, the first downsampling network includes k first downsampling layers connected in sequence, the first upsampling network includes k first upsampling layers connected in sequence, the number of the second fused style features is k, and k is an integer greater than 1; performing an upsampling operation on the content features and the second fused style features through the first upsampling network to obtain target image features includes: performing an upsampling operation on the first noise feature, the content feature, and the first second fused style feature through the first first upsampling layer among the k first upsampling layers to obtain a first target image feature, wherein the first second fused style feature is determined based on the feature output by the first first downsampling layer, and the first noise feature is used to characterize random noise; An upsampling operation is performed on the m-1th target image feature, the content feature and the mth second fused style feature through the mth first upsampling layer among the k first upsampling layers to obtain the mth target image feature, wherein the mth second fused style feature is determined based on the feature output by the mth first downsampling layer, and m is an integer greater than 1 and not greater than k.

12. The method according to claim 11, characterized in that The step of performing a decoding operation on the target image feature through the decoding network to obtain the target image includes: Through the decoding network, a decoding operation is performed on the kth target image feature to obtain the target image.

13. The method according to any one of claims 1 to 12, characterized in that: The method is performed by a first image generation model, and the training process of the first image generation model includes: Acquire sample content information and a sample label image, wherein the content of the sample label image matches the content indicated by the sample content information; Extracting, by means of a second image generation model, a sample content feature corresponding to the sample content information, extracting a first sample reference feature corresponding to the sample label image, adding a second noise feature to the first sample reference feature to obtain a second sample reference feature, wherein the second noise feature is used to characterize random noise; Generate a sample target image based on the sample content feature by using the second image generation model and guided by the second sample reference feature; Based on the sample target image and the sample label image, training the second image generation model to obtain a trained second image generation model; Based on the trained second image generation model, the first image generation model is constructed.

14. The method according to claim 13, characterized in that The trained first image generation model includes an encoding network, a second downsampling network, a second upsampling network and a decoding network connected in sequence, the second downsampling network includes a first residual layer, a first self-attention layer and a third cross-attention layer connected in sequence, and the second upsampling network includes a second residual layer, a fourth self-attention layer and a first cross-attention layer connected in sequence; The step of constructing the first image generation model based on the trained second image generation model includes: Copy the first self-attention layer to obtain a second self-attention layer; copy the first cross-attention layer to obtain a second cross-attention layer; Copy the fourth self-attention layer to obtain a third self-attention layer; construct a feature processing network based on the feature fusion layer and the third self-attention layer; In the trained second image generation model, the third cross-attention layer of the second down-sampling network is replaced with the second self-attention layer to obtain a first down-sampling network, the second cross-attention layer is added after the first cross-attention layer of the second up-sampling network to obtain a first up-sampling network, and the feature processing network is added between the first down-sampling network and the first up-sampling network to obtain the first image generation model.

15. An image generating device, characterized in that: The device comprises: An acquisition module, used to acquire content information and a plurality of reference images, wherein the plurality of reference images have the same reference style, and the content information is used to indicate the content of the image to be generated; A feature extraction module, used to extract content features corresponding to the content information, and extract reference style features corresponding to each reference image in the plurality of reference images; A feature fusion module, used to fuse the reference style features respectively corresponding to the multiple reference images to obtain a first fused style feature; A self-attention encoding module, configured to perform a self-attention encoding operation on the first fused style feature to obtain a second fused style feature, wherein the second fused style feature is used to characterize the reference style; An image generation module is used to generate a target image based on the content feature and guided by the reference style represented by the second fusion style feature, wherein the content of the target image matches the content indicated by the content information.

16. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the image generation method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the image generation method according to any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that The computer program is loaded and executed by a processor to implement the operations performed by the image generating method according to any one of claims 1 to 14.

Images