Model training method, image generation method and device

Abstract

The embodiments of the present disclosure provide a model training method, an image generation method and a device. The model training method comprises: first obtaining a training sample set, the training sample set comprises a sample input image, a sample reference image and an expected image, then constructing an initial model, the initial model comprises a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture, and finally using a machine learning method, taking the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and taking the output of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, training the initial model to obtain an image generation model, training based on the sample input image and the sample reference image, and training to obtain an image generation model that can process the input image and the reference image to obtain the output image.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the fields of computer technology and Internet technology, specifically to the fields of artificial intelligence technology and image processing technology, and especially to model training methods, image generation methods and devices. Background Art

[0002] With the continuous development and widespread application of deep neural networks, artificial intelligence has developed rapidly in the field of computer vision. Among them, image editing based on artificial intelligence technology can not only allow the public without photo editing technology to edit photos conveniently and quickly, but also efficiently form some interesting special effects, such as generating photos of the opposite sex, photos of different ages, and cartoon faces. The commonly used technical means now is an image translation framework based on the inverse mapping of the generative network, which takes the semantic segmentation map or sketch map of the image as input, maps it to the hidden variables of the generative network, and uses the image generated by the hidden variables to fit the original image corresponding to the semantic segmentation map or sketch map.

[0003] However, the above training method is complex and time-consuming, and the network is highly dependent on data; although image editing based on the inverse mapping of the generative network can realize semantic editing of real images, the editing process requires detailed analysis and processing of the latent space of the generative network, which is relatively cumbersome and cannot be conveniently controlled. Summary of the invention

[0004] The embodiments of the present disclosure provide a model training method, an image generating method, an apparatus, an electronic device, and a computer-readable medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a model training method, the method comprising: obtaining a training sample set, wherein the training sample set comprises a sample input image, a sample reference image and an expected image; constructing an initial model, wherein the initial model comprises a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture; using a machine learning method, taking the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and taking the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, training the initial model to obtain an image generation model.

[0006] In some embodiments, each layer of generators in the multi-level generator is constructed based on a pre-trained multi-level generation network, and each layer of generator includes a pre-trained first module for processing feature vectors in a first space; and, using a machine learning method, taking a sample input image as an input of a first initial encoder, a sample reference image as an input of a second initial encoder, and the outputs of the first initial encoder and the second initial encoder as inputs of the multi-level generator, and an expected image as an expected output, the initial model is trained to obtain an image generation model, including: taking a sample input image as an input of the first initial encoder, and a sample reference image as an input of the second initial encoder; inputting the output results of the first initial encoder and the second initial encoder into the first module in each layer of the generator, and the expected image as the expected output, and training the first initial encoder and the second initial encoder using a machine learning method to obtain a first encoder and a second encoder; the first encoder, the second encoder, and the multi-level generator including the first module are determined as the image generation model.

[0007] In some embodiments, the preset layer generator in the multi-level generator also includes a second initial module for processing feature vectors in the second space; and the method also includes: obtaining a sample feature pyramid corresponding to the sample reference image, wherein the sample feature pyramid includes sample feature maps of multiple scales; using the sample input image as the input of the first encoder, and the sample reference image as the input of the second encoder; inputting the output results of the first encoder and the second encoder into the first module in each layer of the generator; inputting the output result of the previous layer generator of the preset layer generator and the sample feature pyramid into the second initial module, using the expected image as the expected output, and training the second initial module using a machine learning method to obtain a second module; and determining the first encoder, the second encoder, and the multi-level generator including the first module as an image generation model, including: determining the first encoder, the second encoder, and the multi-level generator including the first module and the second module as an image generation model.

[0008] In some embodiments, the second initial module includes a feature selection layer, a feature fusion layer and a Transformer; and the output result of the previous layer generator of the preset layer generator and the sample feature pyramid are input into the second initial module, the expected image is used as the expected output, and the second initial module is trained using a machine learning method to obtain a second module, including: using the sample feature pyramid corresponding to the sample reference image as the input of the feature fusion layer, the output result of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image as the input of the feature selection layer, and using the output of the feature fusion layer and the feature selection layer as the input of the Transformer; inputting the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator; using the expected image as the expected output, and training the second initial module in the preset layer generator using a machine learning method to obtain a second module.

[0009] In some embodiments, a machine learning method is used to train the initial model using a sample input image as the input of a first initial encoder, a sample reference image as the input of a second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of a multi-level generator, and an expected image as the expected output, to obtain an image generation model, including: using the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; determining a first loss function that characterizes image pixel features based on the generated image and the expected image; and adjusting the initial model based on the first loss function to obtain an image generation model.

[0010] In some embodiments, the method also includes: determining a second loss function based on the generated image and the expected image, wherein the second loss function includes at least one of the following: a perceptual loss function characterizing image perceptual features, a loss function characterizing image identification features, and a regularization loss function; and adjusting the initial model based on the first loss function to obtain an image generation model, including: adjusting the initial model based on the first loss function and the second loss function to obtain an image generation model.

[0011] In a second aspect, an embodiment of the present disclosure provides an image generation method, the method comprising: in response to acquiring an input image and a reference image, inputting the input image and the reference image into an image generation model, wherein the image generation model comprises a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture; encoding and mapping the input image by the first encoder to obtain a first spatial input vector corresponding to the input image; encoding and mapping the reference image by the second encoder to obtain a first spatial reference vector corresponding to the reference image; inputting the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is acquired based on the model training method in the first aspect.

[0012] In some embodiments, each layer of generators in the multi-level generator includes a pre-trained first module for processing feature vectors in the first space, and the preset layer generator also includes a second module for processing feature vectors in the second space; and the method also includes: obtaining a feature pyramid corresponding to a reference image, the feature pyramid including feature maps of multiple scales; inputting the first space input vector and the first space reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image, including: inputting the first space input vector and the first space reference vector into the first module in each layer of generators, and inputting the output result of the previous layer of generator of the preset layer generator and the feature pyramid into the second module; processing the first space input vector, the first space reference vector, the feature pyramid and the output result of the previous layer of generator of the preset layer generator through the first module and the second module to generate a target image corresponding to the input image and the reference image.

[0013] In some embodiments, the second module includes a feature selection layer, a feature fusion layer and a Transformer; and the first space input vector, the first space reference vector, the feature pyramid and the output result of the previous layer generator of the preset layer generator are processed by the first module and the second module to generate a target image corresponding to the input image and the reference image, including: inputting the feature pyramid into the feature fusion layer to obtain multiple fusion features; inputting the output result of the previous layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer, and outputting the expected features based on preset conditions; inputting multiple fusion features and the expected features into the Transformer for feature alignment to obtain alignment features; inputting the alignment features, the first space input vector and the first space reference vector into the first module, and inputting the processing results into the next layer generator of the preset layer generator until the target image corresponding to the first space input image and the first space reference image is generated.

[0014] In some embodiments, the Transformer also includes an attention mechanism; and the method also includes: selecting alignment features based on the attention mechanism of the Transformer to obtain a selected feature vector; and inputting the alignment features, the first spatial input vector and the first spatial reference vector into the first module, and inputting the processing results into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated, including: inputting the selected feature vector, the first spatial input vector and the first spatial reference vector into the first module, and inputting the processing results into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

[0015] In a third aspect, an embodiment of the present disclosure provides a model training device, which includes: an acquisition module, configured to acquire a training sample set, wherein the training sample set includes a sample input image, a sample reference image and an expected image; a construction module, configured to construct an initial model, wherein the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture; a training module, configured to use a machine learning method, take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and take the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to train the initial model to obtain an image generation model.

[0016] In some embodiments, each layer of generators in the multi-level generator is constructed based on a pre-trained multi-level generation network, and each layer of generator includes a pre-trained first module for processing feature vectors in a first space; and a training module, which is further configured to: use the sample input image as the input of the first initial encoder, and the sample reference image as the input of the second initial encoder; input the output results of the first initial encoder and the second initial encoder to the first module in each layer of the generator, and use the expected image as the expected output, and use a machine learning method to train the first initial encoder and the second initial encoder to obtain a first encoder and a second encoder; determine the first encoder, the second encoder and the multi-level generator including the first module as an image generation model.

[0017] In some embodiments, the preset layer generator in the multi-level generator also includes a second initial module for processing feature vectors in the second space; and the acquisition module is further configured to: acquire a sample feature pyramid corresponding to the sample reference image, wherein the sample feature pyramid includes sample feature maps of multiple scales; the training module is further configured to: use the sample input image as the input of the first encoder, and the sample reference image as the input of the second encoder; input the output results of the first encoder and the second encoder to the first module in each layer of the generator; input the output result of the previous layer generator of the preset layer generator and the sample feature pyramid to the second initial module, use the expected image as the expected output, and use a machine learning method to train the second initial module to obtain a second module; the first encoder, the second encoder, and the multi-level generator including the first module and the second module are determined as an image generation model.

[0018] In some embodiments, the second initial module includes a feature selection layer, a feature fusion layer and a Transformer; the training module is further configured to: use the sample feature pyramid corresponding to the sample reference image as the input of the feature fusion layer, the output result of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image as the input of the feature selection layer, and use the output of the feature fusion layer and the feature selection layer as the input of the Transformer; input the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator; use the expected image as the expected output, and use a machine learning method to train the second initial module in the preset layer generator to obtain a second module.

[0019] In some embodiments, the training module is further configured to: use the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and use the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; determine a first loss function that characterizes the pixel features of the image based on the generated image and the expected image; and adjust the initial model based on the first loss function to obtain an image generation model.

[0020] In some embodiments, the training module is further configured to: determine a second loss function based on the generated image and the expected image, wherein the second loss function includes at least one of the following: a perceptual loss function characterizing image perceptual features, a loss function characterizing image identification features, and a regularization loss function; adjust the initial model based on the first loss function and the second loss function to obtain an image generation model.

[0021] In a fourth aspect, an embodiment of the present disclosure provides an image generating device, which includes: an input module, configured to input the input image and the reference image into an image generation model in response to acquiring the input image and the reference image, wherein the image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture; a first encoding module, configured to encode and map the input image through the first encoder to obtain a first spatial input vector corresponding to the input image; a second encoding module, configured to encode and map the reference image through the second encoder to obtain a first spatial reference vector corresponding to the reference image; a generation module, configured to input the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is acquired based on the model training method in the first aspect.

[0022] In some embodiments, each layer of generators in the multi-level generator includes a pre-trained first module for processing feature vectors in the first space, and the preset layer generator also includes a second module for processing feature vectors in the second space; the device also includes an acquisition module; the acquisition module is configured to acquire a feature pyramid corresponding to a reference image, and the feature pyramid includes feature maps of multiple scales; the generation module is further configured to: input the first space input vector and the first space reference vector to the first module in each layer of generators, and input the output result of the previous layer generator of the preset layer generator and the feature pyramid to the second module; the first space input vector, the first space reference vector, the feature pyramid and the output result of the previous layer generator of the preset layer generator are processed by the first module and the second module to generate a target image corresponding to the input image and the reference image.

[0023] In some embodiments, the second module includes a feature selection layer, a feature fusion layer and a Transformer; the generation module is further configured to: input the feature pyramid into the feature fusion layer to obtain multiple fused features; input the output result of the previous layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer, and output the expected features based on preset conditions; input the multiple fused features and the expected features into the Transformer for feature alignment to obtain aligned features; input the aligned features, the first space input vector and the first space reference vector into the first module, and input the processing results into the next layer generator of the preset layer generator until a target image corresponding to the first space input image and the first space reference image is generated.

[0024] In some embodiments, the Transformer also includes an attention mechanism; and the generation module is further configured to: select alignment features based on the Transformer's attention mechanism to obtain a selected feature vector; input the selected feature vector, the first spatial input vector and the first spatial reference vector into the first module, and input the processing result into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

[0025] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by one or more processors, the one or more processors implement the model training method and image generation method described in any embodiment of the first aspect and the second aspect.

[0026] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the model training method and image generation method as described in any one of the embodiments of the first aspect and the second aspect.

[0027] The model training method provided by the embodiment of the present disclosure is as follows: the execution subject first obtains a training sample set, the training sample set includes a sample input image, a sample reference image and an expected image, and then constructs an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture; finally, a machine learning method is used to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output; the initial model is trained to obtain an image generation model; the training is performed based on the sample input image and the sample reference image; a kind of image generation model that can be processed based on the input image and the reference image to obtain the output image is obtained, so that the sample input image and the sample reference image can interact with each other, so that the image generation model is more accurate, and the output image is more in line with user needs, thereby improving the diversity and accuracy of model training, thereby improving the diversity of image generation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0030] Figure 2 is a flowchart of an embodiment of a model training method according to the present disclosure;

[0031] Figure 3 is a flowchart of another embodiment of the model training method according to the present disclosure;

[0032] Figure 4 is a flowchart of another embodiment of the model training method according to the present disclosure;

[0033] Figure 5 is a flow chart of an embodiment of an image generation method according to the present disclosure;

[0034] Figure 6 is a structural schematic diagram of another embodiment of the image generating method according to the present disclosure;

[0035] Figure 7 is a structural schematic diagram of another embodiment of the image generating method according to the present disclosure;

[0036] Figure 8 is a structural schematic diagram of an embodiment of a model training device according to the present disclosure;

[0037] Fig. 9 is a structural schematic diagram of an embodiment of an image generating device according to the present disclosure;

[0038] Fig.10 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the disclosure. It is also necessary to explain that, for ease of description, only the parts related to the relevant disclosure are shown in the accompanying drawings.

[0040] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] Figure 1 An exemplary system architecture 100 is shown to which the model training method, image generation method, and apparatus according to the embodiments of the present disclosure can be applied.

[0042] like Figure 1As shown, the system architecture 100 may include terminal devices 104, 105, 106, a network 107, and servers 101, 102, 103. The network 107 is used to provide a medium for communication links between the terminal devices 104, 105, 106 and the servers 101, 102, 103. The network 107 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] Users can interact with servers 101, 102, 103 belonging to the same server cluster through terminal devices 104, 105, 106 via network 107 to receive or send information, etc. Various applications can be installed on terminal devices 104, 105, 106, such as item display applications, data analysis applications, search applications, etc.

[0044] The terminal devices 104, 105, and 106 may be hardware or software. When the terminal device is hardware, it may be various electronic devices having a display screen and supporting communication with the server, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc. When the terminal device is software, it may be installed in the electronic devices listed above. It may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0045] Servers 101, 102, and 103 may be servers that provide various services, such as a backend server that receives a request sent by a terminal device that establishes a communication connection with the backend server. The backend server may receive and analyze the request sent by the terminal device and generate a processing result.

[0046] Servers 101, 102, and 103 can obtain a training sample set including a sample input image, a sample reference image, and an expected image, and then construct an initial model, which includes a first initial encoder, a second initial encoder, and a multi-level generator based on a hierarchical architecture. Finally, a machine learning method is used to train the initial model by taking the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to obtain an image generation model.

[0047] Alternatively, after obtaining the image generation model, the server 101, 102, 103 can obtain the input image and the reference image, and input the input image and the reference image into the image generation model, which includes a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture, and then encode and map the input image through the first encoder to obtain a first spatial input vector corresponding to the input image, and then encode and map the reference image through the second encoder to obtain a first spatial reference vector corresponding to the reference image, and finally input the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image. The server 101, 102, 103 can send the generated target image to the terminal device 104, 105, 106.

[0048] It should be noted that the server can be hardware or software. When the server is hardware, it can be various electronic devices that provide various services to the terminal device. When the server is software, it can be implemented as multiple software or software modules that provide various services to the terminal device, or it can be implemented as a single software or software module that provides various services to the terminal device. No specific limitation is made here.

[0049] It should be noted that the model training method and image generation method provided in the embodiments of the present disclosure can be executed by the servers 101, 102, and 103. Accordingly, the model training device and the image generation device are arranged in the servers 101, 102, and 103.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0051] Continue to refer Figure 2 , shows a process 200 of an embodiment of a model training method according to the present disclosure. The model training method comprises the following steps:

[0052] Step 210: Obtain a training sample set.

[0053] In this step, the execution entity (e.g. Figure 1The servers 101, 102, 103) in the video can read a character video from a network platform or from a local database, obtain different frame images from the video, and the characters in different frame images have different postures. The different frame images are used as sample reference images and expected images respectively, then the sample reference images and the expected images can be different images of the same character but with different character postures, wherein the sample reference image can represent an image with real feature information of the current character, and the expected image can represent an image with required posture information and real feature information of the current character.

[0054] The above-mentioned execution subject can extract key points of the characters in the expected image, form a key point image with the extracted key points, and use the key point image as a sample input image, so that the sample input image has the same posture information as the expected image. Alternatively, the above-mentioned execution subject performs semantic segmentation on the expected image to obtain a corresponding semantic segmentation image, and uses the semantic segmentation image as a sample input image, so that the sample input image has the same posture information as the expected image. Therefore, the above-mentioned execution subject can obtain the corresponding sample input image based on the expected image in any manner supported by the relevant technology.

[0055] The above-mentioned execution subject can repeat the above-mentioned processing on the character video to obtain multiple groups of sample input images, sample reference images and expected images, thereby forming a training sample set with the obtained multiple groups of sample input images, sample reference images and expected images.

[0056] Furthermore, the execution entity may also obtain the training sample set including the sample input image, the sample reference image and the expected image in any manner supported by any relevant technology, and the present disclosure does not make any specific limitation on this.

[0057] Step 220, constructing an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture.

[0058] In this step, after acquiring the training sample set, the execution subject can construct an initial model for generating a desired image based on a sample input image and a sample reference image, and the initial model may include a first initial encoder, a second initial encoder, and an initial model of a multi-level generator based on a hierarchical architecture. The first initial encoder and the second initial encoder are used to perform feature encoding on the input image; the multi-level generator based on the hierarchical architecture may include a multi-level generator, and the multi-level generator transmits data in a cascade manner, and the output of the previous layer of generator is used as the input of the next layer of generator, and the multi-level generator is used to generate a corresponding image according to the input image features.

[0059] Step 230, using a machine learning method, takes the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to train the initial model and obtain an image generation model.

[0060] In this step, after the above-mentioned execution entity obtains the training sample set and constructs the initial model, it can use machine learning methods to train the initial model based on the training sample set to obtain an image generation model for generating images with the required posture information and the real feature information of the current person.

[0061] Specifically, the above-mentioned execution entity can input the sample input image and the sample reference image into the initial model, wherein the sample input image is used as the input of the first initial encoder, and the sample reference image is used as the input of the second initial encoder, the first initial encoder encodes the sample input image to obtain a first output result, and the second initial encoder encodes the sample reference image to obtain a second output result.

[0062] The above-mentioned execution subject uses the output result of the first initial encoder and the output result of the second initial encoder as the input of the multi-level generator, and can input the output result of the first initial encoder into the first half of the generators in the multi-level generator, and input the output result of the second initial encoder into the latter half of the generators in the multi-level generator. Alternatively, the above-mentioned execution subject can also add and fuse the output result of the first initial encoder and the output result of the second initial encoder to obtain a fusion result, and then input the output result of the first initial encoder into the first preset number of generators in the multi-level generator, input the output result of the second initial encoder into the second preset number of generators in the latter of the multi-level generator, and input the fusion result into the remaining generators in the multi-level generator. And the above-mentioned execution subject uses the expected image as the expected output, trains the above-mentioned initial model, and obtains an image generation model.

[0063] As an example, the multi-level generator includes 18 cascaded generators, and each generator can be sorted according to the sorting position of each generator, and the 18 cascaded generators can be recorded as 1-18. The above-mentioned execution entity can use the output result of the first initial encoder as the input of generators No. 1-4, and the output result of the second initial encoder as the input of generators No. 9-18, add and fuse the output result of the first initial encoder and the output result of the second initial encoder to obtain a fusion result, and use the fusion result as the input of generators No. 5-8.

[0064] As an optional implementation method, the above step 230 can be implemented based on the following steps: using the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; based on the generated image and the expected image, determining a first loss function that characterizes the pixel features of the image; adjusting the initial model based on the first loss function to obtain an image generation model.

[0065] Specifically, the execution subject may use the sample input image as the input of the first initial encoder, and the sample reference image as the input of the second initial encoder. The first initial encoder encodes the sample input image to obtain a first output result, and the second initial encoder encodes the sample reference image to obtain a second output result. The execution subject may use the first output result and the second output result as the input of the multi-level generator. The multi-level generator processes the input first output result and the second output result to obtain the generated image of the multi-level generator.

[0066] The execution subject may calculate the first loss function according to the pixel features of the generated image and the expected image. The first loss function may be a loss function for characterizing the pixel features of the image. The first loss function may be expressed in the form of:

[0067] L2(I,E,Y)=‖YM(I,E)‖2

[0068] Wherein, M(·) represents the above initial model, I represents the sample input image, E represents the sample reference image, and Y represents the expected image.

[0069] After the execution subject determines the first loss function, it determines whether the first loss function exceeds the preset threshold. If the first loss function exceeds the preset threshold, the network parameters of the initial model are adjusted, and the sample input image and the sample reference image are input again to continue training. If the first loss function does not exceed the preset threshold, the model training is completed and the image generation model is obtained. Among them, the preset threshold can be pre-set based on experience, and the present disclosure does not make specific limitations on this.

[0070] In this implementation, by adjusting the initial model based on the first loss function that characterizes the image pixel features to obtain the final image generation model, the accuracy of model training can be improved, so that the image generation model can generate the required image more accurately.

[0071] As an optional implementation method, the above step 230 can also be implemented based on the following steps: using the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and using the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; based on the generated image and the expected image, determining a second loss function, the second loss function including at least one of the following: a perceptual loss function that characterizes the perceptual features of the image, a loss function that characterizes the identification features of the image, and a regularization loss function; adjusting the initial model based on the first loss function and the second loss function to obtain an image generation model.

[0072] Specifically, the execution subject may use the sample input image as the input of the first initial encoder, and the sample reference image as the input of the second initial encoder. The first initial encoder encodes the sample input image to obtain a first output result, and the second initial encoder encodes the sample reference image to obtain a second output result. The execution subject may use the first output result and the second output result as the input of the multi-level generator. The multi-level generator processes the input first output result and the second output result to obtain the generated image of the multi-level generator.

[0073] The execution subject may calculate the first loss function according to the pixel features of the generated image and the expected image. The first loss function may be a loss function for characterizing the pixel features of the image. The first loss function (L2(I, E, Y)) may be expressed as:

[0074] L2(I,E,Y)=‖YM(I,E)‖2

[0075] Wherein, M(·) represents the above initial model, I represents the sample input image, E represents the sample reference image, and Y represents the expected image.

[0076] After the above-mentioned execution entity determines the first loss function, it can also calculate the second loss function based on the generated image and the expected image. The second loss function may include at least one of the following: a perceptual loss function that characterizes image perceptual features, a loss function that characterizes image identification features, and a regularization loss function.

[0077] (1) Perceptual loss function (L) that characterizes image perceptual features LPIPS (I,E,Y)) can be expressed as:

[0078] L LPIPS (I,E,Y)=‖F(Y)-F(M(I,E))‖2

[0079] Wherein, M(·) represents the above initial model, I represents the sample input image, E represents the sample reference image, Y represents the expected image, F(·) represents a perceptual feature extraction network, and the above LPIPS loss function is a kind of perceptual loss function, which has been widely proved to have good performance in measuring and maintaining image quality.

[0080] (2) The loss function (L ID (I,E,Y)) can be expressed as:

[0081] L ID (I,E,Y)=1-<R(Y),R(M(I,E))>

[0082] Wherein, M(·) represents the above initial model, I represents the sample input image, E represents the sample reference image, Y represents the expected image, and R(·) is a pre-trained ArcFace (face recognition) network used to maintain the identity information of the face. When optimizing the model, it is required that the features of the generated image and the features of the expected image are as similar as possible.

[0083] (3) Regularization loss function (L reg (I,E)) can be expressed as:

[0084]

[0085] Where I represents the sample input image, E represents the sample reference image, E(·) is the encoder, It is the average value of multiple latent variables in the latent space. This regularization term can ensure that the latent variables do not deviate from the distribution learned in the pre-training stage.

[0086] The above execution entity can use these four loss functions to optimize the initial model at the same time, which can be expressed as:

[0087] L=λ1L2+λ2L LPIPS +λ3L ID +λ4L reg

[0088] Among them, λ1, λ2, λ3, and λ4 are weights of four loss functions respectively, which can be set according to experience, for example, λ1=1, λ2=0.8, λ3=0.1, and λ4=0.005, and the present disclosure does not make specific limitations on this.

[0089] In this implementation, by utilizing multiple loss functions to optimize the initial model, model training can be made more accurate, thereby improving the accuracy of the image generation model, so that the images generated based on the image generation model are more accurate and better meet user needs.

[0090] The model training method provided by the embodiment of the present disclosure is as follows: the execution subject first obtains a training sample set, the training sample set includes a sample input image, a sample reference image and an expected image, and then constructs an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture; finally, a machine learning method is used to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output; the initial model is trained to obtain an image generation model; the training is performed based on the sample input image and the sample reference image; a kind of image generation model that can be processed based on the input image and the reference image to obtain the output image is obtained, so that the sample input image and the sample reference image can interact with each other, so that the image generation model is more accurate, and the output image is more in line with user needs, thereby improving the diversity and accuracy of model training, thereby improving the diversity of image generation models.

[0091] refer to Figure 3 , Figure 3 A flowchart showing another embodiment of a model training method may include the following steps:

[0092] Step 310: Obtain a training sample set.

[0093] Step 310 of this embodiment can be performed according to Figure 2 Step 210 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0094] Step 320, constructing an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture.

[0095] Step 320 of this embodiment can be performed according to Figure 2 Step 220 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0096] Step 330: Use the sample input image as input to a first initial encoder, and use the sample reference image as input to a second initial encoder.

[0097] Step 330 of this embodiment can be performed according to Figure 2 Step 230 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0098] Step 340, input the output results of the first initial encoder and the second initial encoder to the first module in each layer generator, take the expected image as the expected output, and use the machine learning method to train the first initial encoder and the second initial encoder to obtain the first encoder and the second encoder.

[0099] Each layer of the generator in the above multi-level generator is constructed based on a pre-trained multi-level generative network, wherein each layer of the generator includes a pre-trained first module for processing feature vectors in the first space. The pre-trained multi-level generative network can be a plurality of cascaded generative networks that have completed model training, such as a StyleGAN network or other generative network for processing feature vectors in the first space. The first space can be the W+ space corresponding to the latent variable. If the first module is a pre-trained processing network, it does not need to be trained in this model training.

[0100] In this step, the above-mentioned execution entity uses the sample input image as the input of the first initial encoder and the sample reference image as the input of the second initial encoder. The first initial encoder encodes the sample input image to obtain a first output result, and the second initial encoder encodes the sample reference image to obtain a second output result.

[0101] The execution subject uses the output result of the first initial encoder and the output result of the second initial encoder as the input of the first module in each layer of the generator, and can input the output result of the first initial encoder into the first half of the generators in the multi-level generator, that is, input the output result of the first initial encoder into the first module of the first half of the generator, and input the output result of the second initial encoder into the second half of the generator in the multi-level generator, that is, input the output result of the second initial encoder into the first module of the second half of the generator. Alternatively, the execution subject can also add and fuse the output result of the first initial encoder and the output result of the second initial encoder to obtain a fusion result, and then input the output result of the first initial encoder into the first preset number of first modules in the front, input the output result of the second initial encoder into the second preset number of first modules in the back, and input the fusion result into the remaining first modules in the multi-level generator. And the execution subject uses the expected image as the expected output, trains the first initial encoder and the second initial encoder, and obtains the trained first encoder and second encoder.

[0102] Step 350, determining the first encoder, the second encoder and the multi-level generator including the first module as an image generation model.

[0103] In this step, the above-mentioned execution subject obtains the trained first encoder and second encoder through training, and the first encoder, the second encoder and the multi-level generator including the first module constitute an image generation model.

[0104] In this implementation, by utilizing a pre-trained generative network, there is no need to train the multi-level generator, thus getting rid of the dependence on adversarial training. At the same time, it can enhance the generalization of the image generation model and improve the efficiency of model training.

[0105] refer to Figure 4 , Figure 4 A flowchart showing another embodiment of a model training method may include the following steps:

[0106] Step 410: Obtain a training sample set.

[0107] Step 410 of this embodiment can be performed according to Figure 2 Step 210 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0108] Step 420, constructing an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture.

[0109] Step 420 of this embodiment can be performed according to Figure 2 Step 220 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0110] Step 430: Use the sample input image as input to a first initial encoder, and use the sample reference image as input to a second initial encoder.

[0111] Step 430 of this embodiment can be performed according to Figure 2 Step 230 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0112] Step 440, input the output results of the first initial encoder and the second initial encoder to the first module in each layer generator, take the expected image as the expected output, and use the machine learning method to train the first initial encoder and the second initial encoder to obtain the first encoder and the second encoder.

[0113] Step 440 of this embodiment can be performed according to Figure 3 Step 340 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0114] Step 450: Obtain a sample feature pyramid corresponding to the sample reference image.

[0115] Each layer of the generator in the above-mentioned multi-level generator includes a first module, and the preset layer generator also includes a second initial module for processing feature vectors in the second space. The second space can be the F space corresponding to the latent variable. The F space can realize the feature interaction in the spatial dimension (the "space" here refers to the three-dimensional feature, the length and width dimensions of the channel, length and width), and opens a green channel for reusing the reference image features. The reuse of the reference image features will greatly improve the model's retention of identity information, making the output image visually more similar to the reference image.

[0116] In this step, after the execution subject obtains the trained first encoder and second encoder, it can obtain a backbone network for extracting image features. The backbone network can be a deep neural feature extraction network, which can be a residual extraction block composed of different convolutional layers and pooling layers, and use different scale feature extraction modules to extract corresponding information. The execution subject inputs the sample reference image into the backbone network, and the backbone network extracts features from the sample reference image and outputs a sample feature pyramid corresponding to the sample reference image. The sample feature pyramid can include sample feature maps of multiple scales corresponding to the sample reference image.

[0117] Step 460: Use the sample input image as input to the first encoder, and use the sample reference image as input to the second encoder.

[0118] In this step, the above-mentioned execution entity can use the sample input image as the input of the first encoder and the sample reference image as the input of the second encoder. The first encoder encodes the sample input image to obtain the corresponding output result, and the second encoder encodes the sample reference image to obtain the corresponding output result.

[0119] Step 470, input the output results of the first encoder and the second encoder to the first module in each layer generator.

[0120] In this step, the execution subject may use the output result of the first encoder and the output result of the second encoder as the input of the first module in each layer of the generator, and may input the output result of the first encoder into the first half of the generators in the multi-level generator, that is, input the output result of the first encoder into the first module of the first half of the generator, and input the output result of the second encoder into the second half of the generator in the second half of the generator. Alternatively, the execution subject may also add and fuse the output result of the first encoder and the output result of the second encoder to obtain a fusion result, and then input the output result of the first encoder into the first preset number of first modules in the first order, input the output result of the second encoder into the second preset number of first modules in the back order, and input the fusion result into the remaining first modules in the multi-level generator.

[0121] Step 480, input the output result of the previous layer generator of the preset layer generator and the sample feature pyramid into the second initial module, take the expected image as the expected output, and use the machine learning method to train the second initial module to obtain the second module.

[0122] In this step, after the execution subject uses the output of the first encoder and the second encoder as the input of the first module in each layer generator, it can set the input of the second initial module in the preset layer generator.

[0123] The execution subject may input the output result of the previous layer generator of the preset layer generator and the sample feature pyramid into the second initial module, that is, the input of the second initial module in each preset layer generator may be the output result of the previous layer generator and the sample feature pyramid corresponding to the sample reference image. And the execution subject uses the expected image as the expected output, trains the second initial module, and obtains the trained second module.

[0124] Step 490, determine the first encoder, the second encoder, and the multi-level generator including the first module and the second module as the image generation model.

[0125] In this step, the above-mentioned execution subject obtains the trained first encoder and the second encoder, and obtains the trained second module through training, and the first encoder, the second encoder and the multi-level generator including the first module and the second module constitute an image generation model.

[0126] In this implementation, through the Dropout-based annealing training method, the first initial encoder and the second initial encoder corresponding to the first space are first trained, and the feature processing module of the first space is optimized, so that the role of the first space can be fully exerted. Then, the second initial module corresponding to the second space is trained, and the feature processing module of the second space is optimized, so that the role of the second space can be fully exerted, thereby making the feature processing modules corresponding to the first space and the second space in the image generation model more effective, thereby improving the accuracy of the image generation model.

[0127] As an optional implementation, the second initial module may include a feature selection layer, a feature fusion layer and a Transformer. Then the above step 480, inputting the output result of the previous layer generator of the preset layer generator and the sample feature pyramid into the second initial module, taking the expected image as the expected output, and training the second initial module using a machine learning method to obtain the second module, may include the following steps: taking the sample feature pyramid corresponding to the sample reference image as the input of the feature fusion layer, the output result of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image as the input of the feature selection layer, and taking the output of the feature fusion layer and the feature selection layer as the input of the Transformer; inputting the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator; taking the expected image as the expected output, and training the second initial module in the preset layer generator using a machine learning method to obtain the second module.

[0128] Specifically, the execution subject uses the sample feature pyramid corresponding to the sample reference image as the input of the feature fusion layer, and the feature fusion layer can process the sample feature pyramid to obtain multiple sample fusion features.

[0129] The above-mentioned execution entity can also perform feature extraction on the sample input image, obtain the input features corresponding to the sample input image, and use the output results of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image as inputs to the feature selection layer. The feature selection layer can select the output results and input features of the previous generator and set different selections according to different needs.

[0130] The above-mentioned execution entity can use the output of the feature fusion layer and the feature selection layer as the input of the Transformer, that is, the output of the feature fusion layer can be used as the input of the Key and Value in the Transformer, and the output of the feature selection layer can be used as the input of the Query. The Transformer can process the fusion features of multiple input samples and the selected features.

[0131] The above-mentioned preset layer generator includes a first module and a second module. The processing result of the second module needs to be input into the first module. The above-mentioned execution entity can input the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator, take the expected image as the expected output, and use the machine learning method to train the second initial module in the preset layer generator to obtain the trained second module.

[0132] In this implementation, by setting a feature selection layer, a feature fusion layer and a Transformer in the second module, the second module can better maintain the identity information in the sample reference image, and a feature selection layer is set to select the output result of the previous layer generator or the input feature corresponding to the sample input image as the input of the Transformer, which can meet the needs of different situations and can make the output image better maintain the identity information, or better maintain the input spatial information.

[0133] refer to Figure 5 , Figure 5 A flowchart 500 showing an embodiment of an image generation method may include the following steps:

[0134] Step 510, in response to acquiring the input image and the reference image, input the input image and the reference image into an image generation model, the image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture.

[0135] In this step, the execution subject (e.g. Figure 1 The servers 101, 102, 103 in the example can obtain an input image and a reference image, where the input image can be a key point image or a semantic segmentation image, etc., and the reference image can be an image with character feature information. The above-mentioned execution subject can input the input image and the reference image into an image generation model, where the image generation model can include a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture.

[0136] Step 520: Encode and map the input image using a first encoder to obtain a first spatial input vector corresponding to the input image.

[0137] In this step, the execution subject may encode and map the input image through the first encoder in the image generation model to obtain a first space input vector corresponding to the input image, and the first space may be the W+ space corresponding to the latent variable.

[0138] Step 530: Encode and map the reference image using a second encoder to obtain a first spatial reference vector corresponding to the reference image.

[0139] In this step, the execution subject may encode and map the reference image through the second encoder in the image generation model to obtain a first space reference vector corresponding to the reference image, and the first space may be the W+ space corresponding to the latent variable.

[0140] Step 540, input the first spatial input vector and the first spatial reference vector into a multi-level generator to generate a target image corresponding to the input image and the reference image.

[0141] In this step, after the above-mentioned execution entity obtains the first spatial input vector output by the first encoder and the first spatial reference vector output by the second encoder, the first spatial input vector and the first spatial reference vector are input into a multi-level generator, and the multi-level generator includes multiple cascaded generators.

[0142] The execution subject can input the first spatial input vector into the first half of the generators in the multi-level generator, and input the first spatial reference vector into the last half of the generators in the multi-level generator. Alternatively, the execution subject can also add and fuse the first spatial input vector and the first spatial reference vector to obtain a vector fusion result, and then input the first spatial input vector into the first preset number of generators in the multi-level generator, and input the first spatial reference vector into the second preset number of generators in the last half of the multi-level generator, and input the vector fusion result into the remaining generators in the multi-level generator. The multi-level generator processes the input first spatial input vector and the first spatial reference vector to generate a target image corresponding to the input image and the reference image. The target image can have the posture information of the input image and the real feature information of the person in the reference image.

[0143] As an example, a multi-level generator includes 18 cascaded generators, and each generator can be sorted according to the sorting position of each generator. The 18 cascaded generators are recorded as 1-18. The above-mentioned execution entity can use the first spatial input vector as the input of generators No. 1-4, and the first spatial reference vector as the input of generators No. 9-18, add and fuse the first spatial input vector and the first spatial reference vector to obtain a vector fusion result, and use the vector fusion result as the input of generators No. 5-8.

[0144] The above image generation model is obtained based on the above model training method, that is, it can be based on the above Figure 2-4 The image generation model can generate a target image corresponding to the input image and the reference image by obtaining the steps in .

[0145] The image generation method provided by the embodiment of the present disclosure, the above-mentioned execution subject first responds to the acquisition of the input image and the reference image, inputs the input image and the reference image into the image generation model, and the image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture, and then encodes and maps the input image through the first encoder to obtain a first spatial input vector corresponding to the input image, and then encodes and maps the reference image through the second encoder to obtain a first spatial reference vector corresponding to the reference image, and finally inputs the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image. The image generation model is obtained based on the above-mentioned model training method, and by adding the reference image as an additional input, the identity information of the generated face image is controlled to realize the editing of the real image, and can generate a target image with both the posture of the input image and the identity information of the reference image based on the input image and the reference image, thereby improving the authenticity and diversity of image generation.

[0146] refer to Figure 6 , Figure 6 A structural schematic diagram showing another embodiment of an image generation method may include the following steps:

[0147] Step 610, in response to acquiring the input image and the reference image, input the input image and the reference image into an image generation model, the image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture.

[0148] Step 610 of this embodiment can be performed according to Figure 5 Step 510 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0149] Step 620: Encode and map the input image using a first encoder to obtain a first spatial input vector corresponding to the input image.

[0150] Step 620 of this embodiment can be performed according to Figure 5 Step 520 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0151] Step 630: Encode and map the reference image using a second encoder to obtain a first spatial reference vector corresponding to the reference image.

[0152] Step 630 of this embodiment can be performed according to Figure 5 Step 530 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0153] Step 640: Obtain a feature pyramid corresponding to the reference image.

[0154] Each layer of generators in the multi-layer generators in the above-mentioned image generation model includes a pre-trained first module for processing feature vectors in a first space, and the preset layer generator also includes a second module for processing feature vectors in a second space.

[0155] In this step, the execution subject may obtain a backbone network for extracting image features, which may be a deep neural feature extraction network, which may be a residual extraction block composed of different convolutional layers and pooling layers, and use different scale feature extraction modules to extract corresponding information. The execution subject inputs the reference image into the backbone network, which extracts features from the reference image and outputs a feature pyramid corresponding to the reference image, which may include feature maps of multiple scales corresponding to the reference image.

[0156] Step 650, input the first spatial input vector and the first spatial reference vector to the first module in each layer generator, and input the output result of the previous layer generator of the preset layer generator and the feature pyramid to the second module.

[0157] In this step, the execution subject may input the obtained first spatial input vector and the first spatial reference vector into the first module in each layer of the generator, and input the output result and the feature pyramid of the previous layer of the preset layer generator into the second module. The multi-level generator may process the first spatial input vector and the first spatial reference vector through the first module, and process the output result and the feature pyramid of the previous layer of the generator through the second module.

[0158] Step 660, the first spatial input vector, the first spatial reference vector, the feature pyramid and the output result of the previous layer generator of the preset layer generator are processed by the first module and the second module to generate a target image corresponding to the input image and the reference image.

[0159] In this step, the above-mentioned execution entity inputs the first spatial input vector and the first spatial reference vector into the first module in each layer of the generator, and inputs the output result of the previous layer of the preset layer generator and the feature pyramid into the second module. The first module and the second module can then process the first spatial input vector, the first spatial reference vector, the feature pyramid and the output result of the previous layer of the preset layer generator, thereby realizing the multi-level generator processing the first spatial input vector, the first spatial reference vector and the feature pyramid to generate a target image corresponding to the input image and the reference image.

[0160] In this implementation, by setting a second module in the multi-level generator, the features of the second space can be further processed, and the reference image can be better integrated into the multi-level generator. The ability of the generative network inverse mapping to perform semantic editing on the image will not be affected. The real image can be simultaneously cross-domain controlled and semantically controlled, making the generated target image more realistic and accurate.

[0161] refer to Figure 7 , Figure 7 A schematic diagram showing the structure of another embodiment of an image generation method may include the following steps:

[0162] Step 710, in response to acquiring the input image and the reference image, input the input image and the reference image into an image generation model, the image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture.

[0163] Step 710 of this embodiment can be performed according to Figure 5 Step 510 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0164] Step 720: Encode and map the input image using a first encoder to obtain a first spatial input vector corresponding to the input image.

[0165] Step 720 of this embodiment can be performed according to Figure 5 Step 520 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0166] Step 730: Encode and map the reference image using a second encoder to obtain a first spatial reference vector corresponding to the reference image.

[0167] Step 730 of this embodiment can be performed according to Figure 5 Step 530 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0168] Step 740: Obtain a feature pyramid corresponding to the reference image.

[0169] Step 740 of this embodiment can be performed according to Figure 6 Step 640 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0170] Step 750, input the first spatial input vector and the first spatial reference vector to the first module in each layer generator, and input the output result of the previous layer generator of the preset layer generator and the feature pyramid to the second module.

[0171] Step 750 of this embodiment can be performed according to Figure 6 Step 650 in the illustrated embodiment is performed in a similar manner and will not be described in detail here.

[0172] Step 760, input the feature pyramid into the feature fusion layer to obtain multiple fusion features.

[0173] The above-mentioned second module may include a feature selection layer, a feature fusion layer and a Transformer.

[0174] In this step, in the feature pyramid, larger scale features can contain more high-frequency detail information, and smaller scale features can contain more structural information. The above execution subject can input the feature pyramid into the feature fusion layer, and can fuse each adjacent feature in the feature pyramid to obtain multiple fused features.

[0175] The above feature fusion layer can perform the following operations on the feature pyramid:

[0176]

[0177] The input matrix, f(·) represents the upsampling function, 1 represents the all-one vector, and is used to copy the feature map to C channels, “×” and denote cross product and element-wise multiplication respectively. And att(·) denotes the attention module:

[0178] att(X)=sigmoid(W a2 tanh(W a1 X))

[0179] Where X refers to the input, W a1 and W a2 is a parameter in the attention mechanism, and W a1 and W a2 is a fixed parameter that is not trainable.

[0180] The feature fusion layer can perform feature fusion on multiple feature maps in the feature pyramid according to the above operation to obtain multiple fused features.

[0181] Step 770, input the output result of the previous layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer, and output the desired features based on the preset conditions.

[0182] In this step, the execution subject can input the output result of the upper layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer. The feature selection layer can select and output the desired features of the input content according to the preset conditions. The preset conditions can be the desired image that the user wants, that is, if the user wants to generate a target image consistent with the reference image, the output result of the upper layer generator of the preset layer generator can be selected and output; if the user wants to generate a target image consistent with the input image, the input features corresponding to the input image can be selected and output.

[0183] In step 780, the multiple fused features and the expected features are input into the Transformer for feature alignment to obtain aligned features.

[0184] In this step, the above-mentioned execution entity can input multiple fused features and expected features into the Transformer for feature alignment to obtain aligned features, that is, the fused features are used as the Key and Value inputs in the Transformer, and the expected features are used as the Query input in the Transformer to perform feature alignment to obtain aligned features.

[0185] If the desired feature is the output result of the previous layer generator of the preset layer generator, the Transformer can perform the following operations on the input multiple fused features and the desired feature:

[0186] F′ e =W v F e softmax((W k F e ) T W q F g / t)

[0187] in, Represents the output result of the previous layer generator of the preset layer generator, represents the fusion feature, F′ e represents the aligned features, W q , W k , and W v Represent the embedding matrices of Query, Key and Value respectively, and t represents the hyperparameter (temperature parameter) of the softmax operation.

[0188] If the expected feature is the input feature corresponding to the input image, the Transformer can perform the following operations on the input multiple fused features and the expected feature:

[0189] F′ e =W v Fe softmax((W k F e ) T W q F i / t)

[0190] Among them, F i represents the input features corresponding to the input image, represents the fusion feature, F′ e represents the aligned features, W q , W k , and W v Represent the embedding matrices of Query, Key and Value respectively, and t represents the hyperparameter (temperature parameter) of the softmax operation.

[0191] Step 790, input the alignment features, the first spatial input vector and the first spatial reference vector into the first module, and input the processing results into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

[0192] In this step, after the above-mentioned execution entity obtains the alignment feature, the alignment feature, the first space input vector and the first space reference vector can be input into the first module, and the input alignment feature, the first space input vector and the first space reference vector can be processed by the first module, and the processing result can be input into the next layer generator of the preset layer generator, so that the next layer generator processes the input feature until the multi-level generator dual-outputs the target image corresponding to the first space input image and the first space reference image.

[0193] In this embodiment, by setting a feature selection layer, a feature fusion layer and a Transformer in the second module, the second module can better maintain the identity information in the sample reference image, and a feature selection layer is set to select the output result of the previous layer generator or the input feature corresponding to the input image as the input of the Transformer, which can meet the needs of different situations and can make the output image better maintain the identity information, or better maintain the input spatial information.

[0194] As an optional implementation, Transformer may also include an attention mechanism. The above-mentioned image generation method also includes: selecting alignment features based on the attention mechanism of Transformer to obtain a selected feature vector; inputting the selected feature vector, the first spatial input vector and the first spatial reference vector into the first module, and inputting the processing result into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

[0195] Specifically, since there may be occlusion when changing the facial posture, such as teeth being blocked when closing the mouth, not all features of the reference image can be aligned and effectively used. Therefore, the above-mentioned execution subject can set an attention mechanism in the Transformer, further select the aligned features based on the attention mechanism, and obtain the selected feature vector that can be used to generate the target image. The selection process of the above-mentioned Transformer based on the attention mechanism can be:

[0196]

[0197] Among them, F e represents the selected feature vector, F′ e represents the alignment feature, and att(·) represents the attention module:

[0198] att(X)=sigmoid(W a4 tanh(W a3 X))

[0199] X refers to input, W a3 and W a4 is a parameter in the attention mechanism.

[0200] After the above-mentioned execution entity obtains the selected feature vector, it can input the selected feature vector, the first space input vector and the first space reference vector into the first module, process the selected feature vector, the first space input vector and the first space reference vector through the first module, and input the processing result to the next layer generator of the preset layer generator, so that the next layer generator processes the input features until the multi-level generator dual-outputs the target image corresponding to the first space input image and the first space reference image.

[0201] In this implementation, by setting the attention mechanism, the accuracy of the features of the reference image can be improved, making the generated target image more accurate and realistic.

[0202] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a model training device. Figure 2 The method embodiments shown correspond.

[0203] like Figure 8 As shown, the model training device 800 of this embodiment may include: an acquisition module 810, a construction module 820 and a training module 830.

[0204] The acquisition module 810 is configured to acquire a training sample set, wherein the training sample set includes a sample input image, a sample reference image, and a desired image;

[0205] A construction module 820 is configured to construct an initial model, wherein the initial model includes a first initial encoder, a second initial encoder, and a multi-level generator based on a hierarchical architecture;

[0206] The training module 830 is configured to use a machine learning method to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to train the initial model and obtain an image generation model.

[0207] In some optional implementations of the present embodiment, each layer of generators in the multi-level generator is constructed based on a pre-trained multi-level generation network, and each layer of generator includes a pre-trained first module for processing feature vectors in a first space; and a training module, which is further configured to: use the sample input image as the input of the first initial encoder, and the sample reference image as the input of the second initial encoder; input the output results of the first initial encoder and the second initial encoder into the first module in each layer of the generator, and use the expected image as the expected output, and use a machine learning method to train the first initial encoder and the second initial encoder to obtain a first encoder and a second encoder; and determine the first encoder, the second encoder and the multi-level generator including the first module as an image generation model.

[0208] In some optional implementations of the present embodiment, the preset layer generator in the multi-level generator also includes a second initial module for processing feature vectors in the second space; and the acquisition module is further configured to: acquire a sample feature pyramid corresponding to the sample reference image, wherein the sample feature pyramid includes sample feature maps of multiple scales; the training module is further configured to: use the sample input image as the input of the first encoder, and the sample reference image as the input of the second encoder; input the output results of the first encoder and the second encoder to the first module in each layer of the generator; input the output result of the previous layer generator of the preset layer generator and the sample feature pyramid to the second initial module, use the expected image as the expected output, and use a machine learning method to train the second initial module to obtain a second module; the first encoder, the second encoder, and the multi-level generator including the first module and the second module are determined as an image generation model.

[0209] In some optional implementations of the present embodiment, the second initial module includes a feature selection layer, a feature fusion layer and a Transformer; the training module is further configured to: use the sample feature pyramid corresponding to the sample reference image as the input of the feature fusion layer, the output result of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image as the input of the feature selection layer, and use the output of the feature fusion layer and the feature selection layer as the input of the Transformer; input the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator; use the expected image as the expected output, and use a machine learning method to train the second initial module in the preset layer generator to obtain a second module.

[0210] In some optional implementations of the present embodiment, the training module is further configured to: use the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and use the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; determine a first loss function that characterizes the pixel features of the image based on the generated image and the expected image; adjust the initial model based on the first loss function to obtain an image generation model.

[0211] In some optional implementations of the present embodiment, the training module is further configured to: determine a second loss function based on the generated image and the expected image, wherein the second loss function includes at least one of the following: a perceptual loss function characterizing image perceptual features, a loss function characterizing image identification features, and a regularization loss function; adjust the initial model based on the first loss function and the second loss function to obtain an image generation model.

[0212] The model training device provided by the above-mentioned embodiment of the present disclosure, the above-mentioned execution body first obtains a training sample set, the training sample set includes a sample input image, a sample reference image and an expected image, and then constructs an initial model, the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture, and finally uses a machine learning method to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to train the initial model to obtain an image generation model, and train based on the sample input image and the sample reference image to obtain an image generation model that can be processed based on the input image and the reference image to obtain an output image, so that the sample input image and the sample reference image can interact with each other, so that the image generation model is more accurate, and the output image is more in line with user needs, thereby improving the diversity and accuracy of model training, thereby improving the diversity of image generation models.

[0213] Those skilled in the art will appreciate that the above device also includes some other well-known structures, such as a processor, a memory, etc. In order to unnecessarily obscure the embodiments of the present disclosure, these well-known structures are described in detail. Figure 8 Not shown.

[0214] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image generating device. Figure 5 The method embodiments shown correspond.

[0215] like Fig. 9 As shown, the image generating device 900 of this embodiment may include: an input module 910 , a first encoding module 920 , a second encoding module 930 and a generating module 940 .

[0216] The input module 910 is configured to input the input image and the reference image into the image generation model in response to obtaining the input image and the reference image, wherein the image generation model includes a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture;

[0217] A first encoding module 920 is configured to perform encoding mapping on the input image through a first encoder to obtain a first spatial input vector corresponding to the input image;

[0218] The second encoding module 930 is configured to perform encoding mapping on the reference image through a second encoder to obtain a first spatial reference vector corresponding to the reference image;

[0219] The generation module 940 is configured to input the first spatial input vector and the first spatial reference vector into a multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is obtained based on the above-mentioned model training method.

[0220] In some optional implementations of the present embodiment, each layer of generator in the multi-level generator includes a pre-trained first module for processing feature vectors in the first space, and the preset layer generator also includes a second module for processing feature vectors in the second space; the device also includes an acquisition module; the acquisition module is configured to acquire a feature pyramid corresponding to a reference image, and the feature pyramid includes feature maps of multiple scales; the generation module is further configured to: input the first space input vector and the first space reference vector into the first module in each layer of the generator, and input the output result of the previous layer of the preset layer generator and the feature pyramid into the second module; the first space input vector, the first space reference vector, the feature pyramid and the output result of the previous layer of the preset layer generator are processed by the first module and the second module to generate a target image corresponding to the input image and the reference image.

[0221] In some optional implementations of the present embodiment, the second module includes a feature selection layer, a feature fusion layer and a Transformer; the generation module is further configured to: input the feature pyramid into the feature fusion layer to obtain multiple fused features; input the output results of the previous layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer, and output the expected features based on preset conditions; input the multiple fused features and the expected features into the Transformer for feature alignment to obtain aligned features; input the aligned features, the first space input vector and the first space reference vector into the first module, and input the processing results into the next layer generator of the preset layer generator until a target image corresponding to the first space input image and the first space reference image is generated.

[0222] In some optional implementations of the present embodiment, the Transformer also includes an attention mechanism; and the generation module is further configured to: select alignment features based on the Transformer's attention mechanism to obtain a selected feature vector; input the selected feature vector, the first spatial input vector and the first spatial reference vector into the first module, and input the processing result into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

[0223] The above-mentioned embodiment of the present disclosure provides an image generation device. The above-mentioned execution subject first responds to the acquisition of an input image and a reference image, and inputs the input image and the reference image into an image generation model. The image generation model includes a first encoder, a second encoder and a multi-level generator based on a hierarchical architecture. The input image is then encoded and mapped by the first encoder to obtain a first spatial input vector corresponding to the input image. The reference image is then encoded and mapped by the second encoder to obtain a first spatial reference vector corresponding to the reference image. Finally, the first spatial input vector and the first spatial reference vector are input into the multi-level generator to generate a target image corresponding to the input image and the reference image. The image generation model is obtained based on the above-mentioned model training method. By adding a reference image as an additional input, the identity information of the generated face image is controlled to realize the editing of the real image. The target image with the posture of the input image and the identity information of the reference image can be generated based on the input image and the reference image, thereby improving the authenticity and diversity of image generation.

[0224] Those skilled in the art will appreciate that the above device also includes some other well-known structures, such as a processor, a memory, etc. In order to unnecessarily obscure the embodiments of the present disclosure, these well-known structures are described in detail. Fig. 9 Not shown.

[0225] Reference below Fig.10 , which shows a schematic diagram of the structure of an electronic device 1000 suitable for implementing an embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as smart screens, notebook computers, PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0226] like Fig.10 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 are also stored. The processing device 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0227] Typically, the following devices may be connected to the I / O interface 1005: an input device 1006 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1007 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1008 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the electronic device 1000 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 1000 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Fig.10 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0228] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1008, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium of the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0229] Computer program code for performing the operations of the embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet).

[0230] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0231] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, a construction module and a training module, or a processor includes an input module, a first encoding module, a second encoding module and a generation module, wherein the names of these modules do not constitute a limitation on the modules themselves in some cases.

[0232] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: obtains a training sample set, wherein the training sample set includes a sample input image, a sample reference image, and an expected image; constructs an initial model, wherein the initial model includes a first initial encoder, a second initial encoder, and a multi-level generator based on a hierarchical architecture; uses a machine learning method to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the output of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, trains the initial model, and obtains an image generation model. Alternatively, the electronic device: in response to acquiring an input image and a reference image, inputs the input image and the reference image into an image generation model, wherein the image generation model includes a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture; encodes and maps the input image through the first encoder to obtain a first spatial input vector corresponding to the input image; encodes and maps the reference image through the second encoder to obtain a first spatial reference vector corresponding to the reference image; inputs the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is acquired based on a model training method.

[0233] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A model training method, the method comprising: Acquire a training sample set, wherein the training sample set includes a sample input image, a sample reference image, and an expected image; Constructing an initial model, wherein the initial model includes a first initial encoder, a second initial encoder, and a multi-level generator based on a hierarchical architecture, each layer of the generator in the multi-level generator is constructed based on a pre-trained multi-level generation network, each layer of the generator includes a pre-trained first module for processing feature vectors in a first space, and a preset layer generator in the multi-level generator also includes a second initial module for processing feature vectors in a second space; Using a machine learning method, the sample input image is used as the input of the first initial encoder, the sample reference image is used as the input of the second initial encoder, the outputs of the first initial encoder and the second initial encoder are used as the input of the multi-level generator, and the expected image is used as the expected output. The initial model is trained to obtain an image generation model.

2. The method according to claim 1, wherein: The method of using a machine learning method, taking the sample input image as the input of the first initial encoder, taking the sample reference image as the input of the second initial encoder, taking the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, taking the expected image as the expected output, training the initial model, and obtaining an image generation model, comprises: Using the sample input image as input of the first initial encoder and the sample reference image as input of the second initial encoder; Inputting the output results of the first initial encoder and the second initial encoder into the first module in each layer generator, taking the expected image as the expected output, and training the first initial encoder and the second initial encoder by using a machine learning method to obtain a first encoder and a second encoder; The first encoder, the second encoder, and a multi-level generator including the first module are determined as the image generation model.

3. The method according to claim 2, further comprising: Acquire a sample feature pyramid corresponding to the sample reference image, wherein the sample feature pyramid includes sample feature maps of multiple scales; Using the sample input image as input to the first encoder and the sample reference image as input to the second encoder; Input the output results of the first encoder and the second encoder to the first module in each layer generator; Inputting the output result of the previous layer generator of the preset layer generator and the sample feature pyramid into the second initial module, taking the expected image as the expected output, and training the second initial module using a machine learning method to obtain a second module; and The step of determining the first encoder, the second encoder, and a multi-level generator including the first module as the image generation model comprises: The first encoder, the second encoder, and a multi-level generator including the first module and the second module are determined as the image generation model.

4. The method according to claim 3, wherein: The second initial module includes a feature selection layer, a feature fusion layer and a Transformer; and The output result of the previous layer generator of the preset layer generator and the sample feature pyramid are input into the second initial module, the expected image is used as the expected output, and the second initial module is trained by a machine learning method to obtain a second module, including: The sample feature pyramid corresponding to the sample reference image is used as the input of the feature fusion layer, the output result of the previous layer generator of the preset layer generator and the input features corresponding to the sample input image are used as the input of the feature selection layer, and the outputs of the feature fusion layer and the feature selection layer are used as the input of the Transformer; Input the output of the Transformer and the output results of the first encoder and the second encoder into the first module in the preset layer generator; The expected image is used as the expected output, and a machine learning method is used to train the second initial module in the preset layer generator to obtain the second module.

5. The method according to claim 1, wherein: The method of using a machine learning method, taking the sample input image as the input of the first initial encoder, taking the sample reference image as the input of the second initial encoder, taking the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, taking the expected image as the expected output, training the initial model, and obtaining an image generation model, comprises: Using the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and using the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator to obtain a generated image of the multi-level generator; Determine a first loss function characterizing image pixel features based on the generated image and the desired image; The initial model is adjusted based on the first loss function to obtain the image generation model.

6. The method according to claim 5, wherein: The method further comprises: Determining a second loss function based on the generated image and the expected image, wherein the second loss function includes at least one of the following: a perceptual loss function characterizing image perceptual features, a loss function characterizing image identification features, and a regularization loss function; and The step of adjusting the initial model based on the first loss function to obtain the image generation model includes: The initial model is adjusted based on the first loss function and the second loss function to obtain the image generation model.

7. A method for generating an image, the method comprising: In response to acquiring an input image and a reference image, the input image and the reference image are input into an image generation model, wherein the image generation model comprises a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture, each layer of the multi-level generator is constructed based on a pre-trained multi-level generation network, each layer of the generator comprises a pre-trained first module for processing feature vectors in a first space, and the preset layer generator further comprises a second module for processing feature vectors in a second space; Performing encoding mapping on the input image by the first encoder to obtain a first spatial input vector corresponding to the input image; Performing encoding mapping on the reference image by the second encoder to obtain a first spatial reference vector corresponding to the reference image; The first spatial input vector and the first spatial reference vector are input into the multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is obtained based on any one of the methods of claims 1-6.

8. The method according to claim 7, further comprising: Acquire a feature pyramid corresponding to the reference image, wherein the feature pyramid includes feature maps of multiple scales; The step of inputting the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image includes: Input the first spatial input vector and the first spatial reference vector into the first module in each layer generator, and input the output result of the previous layer generator of the preset layer generator and the feature pyramid into the second module; The first spatial input vector, the first spatial reference vector, the feature pyramid and the output result of the previous layer generator of the preset layer generator are processed by the first module and the second module to generate a target image corresponding to the input image and the reference image.

9. The method according to claim 8, wherein: The second module includes a feature selection layer, a feature fusion layer and a Transformer; and The first module and the second module are used to process the first spatial input vector, the first spatial reference vector, the feature pyramid, and the output result of the upper layer generator of the preset layer generator to generate a target image corresponding to the input image and the reference image, including: Inputting the feature pyramid into the feature fusion layer to obtain multiple fusion features; Inputting the output result of the previous layer generator of the preset layer generator and the input features corresponding to the input image into the feature selection layer, and outputting the expected features based on preset conditions; Inputting the multiple fused features and the expected features into the Transformer for feature alignment to obtain aligned features; The alignment features, the first spatial input vector and the first spatial reference vector are input into the first module, and the processing results are input into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

10. The method according to claim 9, wherein: The Transformer also includes an attention mechanism; and the method further includes: Selecting the alignment features based on the attention mechanism of the Transformer to obtain a selected feature vector; and The step of inputting the alignment feature, the first spatial input vector, and the first spatial reference vector into the first module, and inputting the processing result into a next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated includes: The selected feature vector, the first spatial input vector and the first spatial reference vector are input into the first module, and the processing result is input into the next layer generator of the preset layer generator until a target image corresponding to the first spatial input image and the first spatial reference image is generated.

11. A model training device, comprising: An acquisition module is configured to acquire a training sample set, wherein the training sample set includes a sample input image, a sample reference image, and an expected image; A construction module is configured to construct an initial model, wherein the initial model includes a first initial encoder, a second initial encoder and a multi-level generator based on a hierarchical architecture, each layer of the multi-level generator is constructed based on a pre-trained multi-level generation network, each layer of the generator includes a pre-trained first module for processing feature vectors in a first space, and a preset layer generator in the multi-level generator also includes a second initial module for processing feature vectors in a second space; The training module is configured to use a machine learning method to take the sample input image as the input of the first initial encoder, the sample reference image as the input of the second initial encoder, and the outputs of the first initial encoder and the second initial encoder as the input of the multi-level generator, and the expected image as the expected output, to train the initial model and obtain an image generation model.

12. An image generating device, the device comprising: An input module is configured to input the input image and the reference image into an image generation model in response to acquiring the input image and the reference image, wherein the image generation model includes a first encoder, a second encoder, and a multi-level generator based on a hierarchical architecture, each layer of the multi-level generator is constructed based on a pre-trained multi-level generation network, each layer of the generator includes a pre-trained first module for processing feature vectors in a first space, and the preset layer generator also includes a second module for processing feature vectors in a second space; A first encoding module is configured to perform encoding mapping on the input image through the first encoder to obtain a first spatial input vector corresponding to the input image; A second encoding module is configured to perform encoding mapping on the reference image through the second encoder to obtain a first spatial reference vector corresponding to the reference image; A generation module is configured to input the first spatial input vector and the first spatial reference vector into the multi-level generator to generate a target image corresponding to the input image and the reference image, wherein the image generation model is obtained based on any one of the methods of claims 1-6 above.

13. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 10.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images