Image generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image generation method, device, electronic device and storage medium, wherein the image generation method includes: obtaining an original image and a reference image, wherein the original image and the reference image contain target objects of different styles, obtaining low-frequency information of the reference image, obtaining high-frequency information of the original image, and inputting the low-frequency information and the high-frequency information into an image generation model so that the style of the target object in the reference image is transferred to the target object in the original image to obtain a synthesized image. In this way, the high- and low-frequency information of the reference image and the original image can be processed in advance, different feature information can be decoupled, and the desired information can be obtained from different images respectively. Based on this, an image with a strong sense of reality that meets the requirements can be synthesized.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an image generation method, device, electronic device and storage medium. Background Art

[0002] Hair generation technology belongs to 3D head reconstruction technology and is an important part of virtual human technology. Head and facial features are the most important features of virtual humans. Hair is an external feature of the face. Hair reconstruction is essential in virtual human technology and is a research hotspot in the field of computer graphics.

[0003] For example, hair generation technology can refer to a technology that can generate a uniform target color based on requirements for any hair color of an input original image. However, in the prior art, although the final generated hair color can be very close to the target color, the details of the hair are difficult to meet the requirements.

[0004] Based on the above problems, the present invention proposes an image generation technology, which ensures that the color of the generated object (such as hair) meets the requirements while other details can also meet the requirements. Summary of the invention

[0005] The present disclosure provides an image generation method, device, electronic device and storage medium to at least solve the problems of poor image realism and roughness caused by some details not meeting the requirements in the images generated in the related art. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

[0007] Acquire an original image and a reference image; the original image and the reference image contain target objects of different styles;

[0008] Obtain low-frequency information of the reference image;

[0009] Obtain high-frequency information of the original image;

[0010] The low-frequency information and the high-frequency information are input into an image generation model so that the style of the target object in the reference image is transferred to the target object in the original image to obtain a composite image.

[0011] In an optional embodiment, the step of obtaining the low-frequency information of the reference image includes:

[0012] Gaussian blur processing is performed on the reference image to obtain low-frequency information of the reference image.

[0013] In an optional embodiment, the step of obtaining the high-frequency information of the original image includes:

[0014] Performing Gaussian blur processing on the original image to obtain low-frequency information of the original image;

[0015] The high-frequency information of the original image is determined according to the difference between the original image and the low-frequency information of the original image.

[0016] In an optional embodiment, inputting the low-frequency information and the high-frequency information into an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image, and obtaining a synthesized image includes:

[0017] Inputting the low-frequency information into the feature extraction network in the image generation model to extract features, thereby obtaining first feature information;

[0018] The first feature information and the high-frequency information are input into a generator in an image generation model so that the style of the target object in the reference image is transferred to the target object in the original image to obtain a composite image.

[0019] In an optional embodiment, the generator includes an encoder and a decoder;

[0020] The step of inputting the first feature information and the high-frequency information into a generator in an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image includes:

[0021] Using the encoder to extract features from the high-frequency information to obtain second feature information;

[0022] The decoder is used to perform image synthesis according to the first feature information and the second feature information, so that the style of the target object in the reference image is transferred to the target object in the original image, thereby obtaining the synthesized image.

[0023] In an alternative embodiment,

[0024] The first characteristic information includes color characteristic information:

[0025] The second feature information includes texture feature information and / or illumination feature information.

[0026] In an optional embodiment, the training method of the above-mentioned image generation model includes:

[0027] Obtaining a sample original image and a sample reference image; the sample original image and the sample reference image contain sample target objects of different styles;

[0028] Obtaining low-frequency information of the sample reference image and high-frequency information of the sample original image;

[0029] Inputting the low-frequency information of the sample reference image and the high-frequency information of the sample original image into a neural network to obtain a sample synthetic image;

[0030] Inputting the sample original image and the sample composite image into the discriminator of the neural network to obtain verification result information;

[0031] The above-mentioned verification result information is used to train the above-mentioned neural network to obtain a trained image generation model.

[0032] In an optional embodiment, the above-mentioned using the above-mentioned verification result information to train the above-mentioned neural network to obtain the trained above-mentioned image generation model includes:

[0033] Determine true and false label information of the sample original image and the sample composite image;

[0034] Determine the loss value based on the verification result information and the true and false label information;

[0035] The above neural network is trained according to the above loss value to obtain a trained image generation model.

[0036] In an optional embodiment, the determining of the loss value based on the verification result information and the true and false label information includes:

[0037] Determine a first loss value between the verification result information and the true and false label information based on a first loss function;

[0038] Determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain;

[0039] The sum of the first loss value and the second loss value is determined as the loss value.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

[0041] An image acquisition module is configured to acquire an original image and a reference image; the original image and the reference image contain target objects of different styles;

[0042] A low-frequency information acquisition module is configured to acquire the low-frequency information of the reference image;

[0043] A high-frequency information acquisition module is configured to acquire the high-frequency information of the original image;

[0044] The synthesis module is configured to input the low-frequency information and the high-frequency information into an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0045] In an optional embodiment, the low-frequency information acquisition module is configured to execute:

[0046] Gaussian blur processing is performed on the reference image to obtain low-frequency information of the reference image.

[0047] In an optional embodiment, the high-frequency information acquisition module is configured to execute:

[0048] Performing Gaussian blur processing on the original image to obtain low-frequency information of the original image;

[0049] The high-frequency information of the original image is determined according to the difference between the original image and the low-frequency information of the original image.

[0050] In an optional embodiment, the synthesis module includes:

[0051] A feature extraction submodule is configured to perform feature extraction by inputting the low-frequency information into a feature extraction network in the image generation model to obtain first feature information;

[0052] The image synthesis submodule is configured to execute the input of the first feature information and the high-frequency information into the generator in the image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0053] In an optional embodiment, the generator includes an encoder and a decoder;

[0054] The above-mentioned image synthesis submodule is configured to perform:

[0055] Using the encoder to extract features from the high-frequency information to obtain second feature information;

[0056] The decoder is used to perform image synthesis according to the first feature information and the second feature information, so that the style of the target object in the reference image is transferred to the target object in the original image, thereby obtaining the synthesized image.

[0057] In an alternative embodiment,

[0058] The first characteristic information includes color characteristic information:

[0059] The second feature information includes texture feature information and / or illumination feature information.

[0060] In an optional embodiment, the image generating device further includes:

[0061] A sample acquisition module is configured to acquire a sample original image and a sample reference image; the sample original image and the sample reference image contain sample target objects of different styles;

[0062] An information acquisition module, configured to acquire low-frequency information of the sample reference image and high-frequency information of the sample original image;

[0063] A sample synthesis module is configured to input the low-frequency information of the sample reference image and the high-frequency information of the sample original image into a neural network to obtain a sample synthesized image;

[0064] A result output module is configured to execute the discriminator of the neural network by inputting the sample original image and the sample composite image to obtain verification result information;

[0065] The training module is configured to train the neural network using the verification result information to obtain the trained image generation model.

[0066] In an optional embodiment, the training module includes:

[0067] A label determination module is configured to determine true and false label information of the sample original image and the sample composite image;

[0068] A loss determination module, configured to determine a loss value based on the verification result information and the true or false label information;

[0069] The model acquisition module is configured to execute training of the above-mentioned neural network according to the above-mentioned loss value to obtain a trained image generation model.

[0070] In an optional embodiment, the above-mentioned loss determination module includes:

[0071] A first loss determination module is configured to determine a first loss value between the verification result information and the true and false label information based on a first loss function;

[0072] A second loss determination module is configured to determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain;

[0073] The total loss determination module is configured to determine the sum of the first loss value and the second loss value as the loss value.

[0074] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement a method as described in any one of the first aspects above.

[0075] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any method in the first aspect of the embodiment of the present disclosure.

[0076] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any one of the methods in the first aspect of the embodiment of the present disclosure.

[0077] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0078] The original image and the reference image are obtained. The original image and the reference image contain target objects with different styles. The low-frequency information of the reference image is obtained. The high-frequency information of the original image is obtained. The low-frequency information and the high-frequency information are input into the image generation model so that the style of the target object in the reference image is transferred to the target object in the original image to obtain a synthesized image. In this way, the high- and low-frequency information of the reference image and the original image can be processed in advance, different feature information can be decoupled, and the desired information can be obtained from different images. Based on this, a highly realistic image that meets the requirements can be synthesized.

[0079] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0081] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment;

[0082] Figure 2 is a flow chart of an image generating method according to an exemplary embodiment;

[0083] Figure 3 is a structural diagram of an image generation model according to an exemplary embodiment;

[0084] Figure 4 is a structural diagram of an image generation model according to an exemplary embodiment;

[0085] Figure 5 is a flow chart of a method for generating a composite image according to an exemplary embodiment;

[0086] Figure 6 is a block diagram of a first feature extraction network according to an exemplary embodiment;

[0087] Figure 7 is a flowchart of a training process of an image generation model according to an exemplary embodiment;

[0088] Figure 8 is a neural network structure diagram shown according to an exemplary embodiment;

[0089] Fig. 9 is a flowchart of a training process of an image generation model according to an exemplary embodiment;

[0090] Fig.10 is a block diagram of an image generating device according to an exemplary embodiment;

[0091] Fig.11 It is a block diagram of an electronic device for image generation according to an exemplary embodiment. DETAILED DESCRIPTION

[0092] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0093] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0094] In the related art, the original image can extract the object color information of the reference image in the RGB space based on the reference image. However, there is no way to clearly separate the texture, lighting and color information of the object in the RGB space. Therefore, it is impossible to complete hair coloring while maintaining the lighting information and texture information of the original image.

[0095] See also Figure 1 , Figure 1 A schematic diagram of an application environment according to an exemplary embodiment is shown as follows: Figure 1 As shown, it includes a server 01 and a terminal 02. Optionally, the server 01 and the terminal 02 may be connected via a wireless link or a wired link, which is not limited in the present disclosure.

[0096] In an optional embodiment, the server 01 can be used to generate a composite image that meets the requirements based on the acquired original image and reference image. Specifically, the server 01 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the operating system running on the server 01 may include but is not limited to IOS, Linux, Windows, Unix, Android systems, etc.

[0097] In an optional embodiment, the terminal 02 can assist the server 01 in obtaining a device for synthesizing an image. The terminal can be a provider of original images and reference images, and is used to send the original images and reference images to the server 01. Optionally, after the server 01 generates a synthesized image, the synthesized image can be sent back to the terminal 02 for display on the terminal 02. Specifically, the terminal 02 can include but is not limited to electronic devices such as smart phones, desktop computers, tablet computers, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, etc. Optionally, the operating system running on the electronic device can include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0098] In addition, it should be noted that Figure 1 What is shown is only one application environment of the image processing method provided by the present disclosure. In practical applications, other application environments may also be included, such as the generation of a synthetic image, which may also be implemented on the terminal 02.

[0099] Figure 2 is a flowchart of an image generating method according to an exemplary embodiment. Figure 2 As shown, the image generation method is used in electronic devices such as servers and terminals, and includes the following steps:

[0100] In step S201, an original image and a reference image are obtained; the original image and the reference image contain target objects of different styles.

[0101] In an optional embodiment, the original image and the reference image may be obtained by the server from the terminal. In a specific embodiment, the terminal may be an image acquisition device, for example, the original image and the reference image may be acquired by a camera arranged on the terminal. In another optional embodiment, the original image and the reference image may also be obtained by the server through searching from an Internet gallery.

[0102] The purpose of the disclosed embodiment is to combine certain features of the object in the original image with other features of the object in the reference image to generate a realistic object containing the combined features. For example, the texture and lighting of the hair in the original image and the color of the hair in the reference image are combined to produce new hair. In other words, the original image and the reference image contain target objects of different styles. Taking the above example, the styles of the hair color in the original image and the hair color in the reference image are different.

[0103] Optionally, the target object in the original image and the target object in the reference image may be the same thing, such as a face, hair, top, head, etc. Optionally, the target object in the original image and the target object in the reference image may not be the same thing, such as the target object in the original image is hair, and the target object in the reference image is clothing.

[0104] In order to better explain the present disclosure, the target object in both the original image and the reference image is hair. In an optional embodiment, the server obtains the original image and the reference image, wherein the color of the hair in the original image is different from the color of the hair in the reference image. For example, the hair color in the original image is golden, and the hair color in the reference image is black, and the styles of the two are inconsistent.

[0105] In step S203, low-frequency information of the reference image is obtained.

[0106] Since in the embodiment of the present disclosure, the server needs the color information in the reference image, and the color information is located in or exists more in the low-frequency space of the image, therefore, in an optional embodiment, the server can first obtain the low-frequency information of the reference image, and then obtain the color information from the low-frequency information.

[0107] In an optional embodiment, the server may filter the reference image by filtering to obtain the low-frequency information of the reference image. Optionally, the filtering method includes Gaussian blur processing using a Gaussian blur function. Specifically, the server calls the Gaussian blur function to apply a filter function to the reference image, and the output is the low-frequency information of the reference image.

[0108] In step S205, high-frequency information of the original image is obtained.

[0109] In the embodiment of the present disclosure, the server needs the illumination information and texture information in the original image, and the illumination information and texture information are located or exist more in the high-frequency space of the image. Therefore, in an optional embodiment, the server can first obtain the high-frequency information of the original image, and then obtain the illumination information and texture information from the high-frequency information.

[0110] In an optional embodiment, the server may filter the original image by filtering to obtain the low-frequency information of the original image. Optionally, the filtering method includes Gaussian blur processing using a Gaussian blur function. Specifically, the server calls the Gaussian blur function to apply a filter function to the original image, and obtains the low-frequency information of the original image. The server may determine the high-frequency information of the original image by using the difference between the original image and the low-frequency information of the original image.

[0111] In step S207, the low-frequency information and the high-frequency information are input into the image generation model so that the style of the target object in the reference image is transferred to the target object in the original image to obtain a composite image.

[0112] In an optional embodiment, the image generation model includes a feature extraction network, and the feature extraction network is used to extract features of low-frequency information to obtain first feature information.

[0113] Figure 3 1 is a structural diagram of an image generation model according to an exemplary embodiment, including an image generation model 300, and a feature extraction network 301 and a generator 302 included in the image generation model 300. Based on the above image generation model, the server can transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0114] In the disclosed embodiment, the server may input the low-frequency information into the feature extraction network 301 in the image generation model for feature extraction to obtain the first feature information. The first feature information and the high-frequency information are input into the generator in the image generation model to transfer the style of the target object in the reference image to the target object in the original image to obtain a composite image.

[0115] Based on the target object being hair, in an optional embodiment, the image generation model is trained in advance before it is actually used to obtain a synthetic image, so the generator in the image generation model is also trained. Therefore, we can know that in theory, the color of the hair in the synthetic image generated by the generator should be the color of the hair in the reference image, and the texture and lighting of the hair in the synthetic object should be the texture and lighting of the hair in the original image. However, the trained image generation model is necessarily not perfect, so it is sufficient to meet the following conditions: the matching degree value of the color of the hair in the synthetic image and the color of the hair in the reference image is greater than the preset first degree value; the matching degree value of the texture of the hair in the synthetic image and the texture of the hair in the original image is greater than the preset second degree value; the matching degree value of the lighting of the hair in the synthetic image and the lighting of the hair in the original image is greater than the preset third degree value. Optionally, the above-mentioned preset first degree value, preset second degree value and preset third degree value can all be considered to be set, or can be determined based on the trained image generation model.

[0116] In an alternative embodiment, Figure 3 The generator 302 shown may also include an encoder and a decoder, Figure 4 It is a structural diagram of an image generation model according to an exemplary embodiment, including an image generation model 300, and a feature extraction network 301 and a generator 302 included in the image generation model 300; wherein the generator 302 may include an encoder 3021 and a decoder 3022.

[0117] Figure 5 is a flowchart of a method for generating a synthetic image according to an exemplary embodiment. Figure 4 As shown in the structural diagram, inputting the first feature information and the high-frequency information into the generator in the image generation model to obtain the synthesized image may include:

[0118] In step S501, an encoder is used to extract features from high-frequency information to obtain second feature information.

[0119] In an optional embodiment, the above-mentioned feature extraction network and encoder can be constructed based on a neural convolutional network in a deep learning network. In the embodiment of this specification, taking the feature extraction network as an example, the feature extraction network can include a convolutional layer, a pooling layer and an upsampling layer, wherein the number of convolutional layers, pooling layers and upsampling layers can be set in combination with actual application requirements. In practical applications, data is often not linearly separable. In order to introduce nonlinear factors, an activation layer can be introduced into the neural network. In addition, in order to prevent the problem of gradient disappearance and gradient explosion caused by the depth of the network in the deep learning network, optionally, the problem of the gradient can be solved by using the initialization (normalized initialization) of the data and the normalization (Instance normalization, IN). However, as the network depth deepens, it will bring about other degradation problems of network performance, that is, the network depth deepens, but the error rate increases. Therefore, the IN layer can be used to solve the degradation problem, and the normalization layer also solves the gradient problem, so that the performance of the network is also improved. Correspondingly, in the embodiment of this specification, each convolutional layer of the neural convolutional network can be connected to the normalization layer and the activation layer in sequence.

[0120] In a specific embodiment, the sizes of the convolution layer, normalization layer, activation layer, pooling layer, and the upsampling multiple of the upsampling layer can be set according to actual application requirements. Figure 6 As shown, Figure 6 It is a block diagram of a first feature extraction network according to an exemplary embodiment. Figure 6 In the example, the input may be the low-frequency information of the reference image. Starting from the input, the first feature extraction network may include a 5*5 convolution layer (normalization layer, activation layer), a 2*2 pooling layer, a 5*5 convolution layer (normalization layer, activation layer), a 3*3 convolution layer (normalization layer, activation layer), a 2*2 pooling layer, a 3*3 convolution layer, and a 3*3 convolution layer. After the above feature extraction network, the first feature information of the reference image may be obtained. Optionally, the first feature information may be color feature information.

[0121] Optionally, the encoder can also be constructed based on a neural convolutional network in a deep learning network. The number of convolutional layers, pooling layers, and upsampling layers included therein can be set in combination with actual application requirements. The specific structure can refer to the feature extraction network structure, which will not be repeated here. Optionally, the second feature information can be texture feature information and / or illumination feature information.

[0122] In step S503, the decoder is used to perform image synthesis according to the first feature information and the second feature information, so that the style of the target object in the reference image is transferred to the target object in the original image, thereby obtaining a synthesized image.

[0123] In an optional embodiment, the decoder in step S503 can be a renderer, and the server uses the renderer to perform synthesis based on the first feature information and the second feature information (color feature information, texture feature information and / or lighting feature information) so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0124] In an optional embodiment, the first feature information is color feature information, and the second feature information is texture feature information and / or lighting feature information based on the target object being hair. If the target object is other things, the first feature information and the second feature information can also be determined based on other things.

[0125] In an optional embodiment, the above method further includes a training process of an image generation model, such as Figure 7 As shown, Figure 7 is a flowchart of a training process of an image generation model according to an exemplary embodiment, including:

[0126] In step S701, a sample original image and a sample reference image are obtained; the sample original image and the sample reference image contain sample target objects of different styles.

[0127] In an optional embodiment, the sample reference image and the sample original image are used to train the model. The specific acquisition process can refer to the acquisition process of the reference image and the original image. Since the above-mentioned different styles are exemplified by different hair colors, during the training process, the hair colors of the sample original image and the sample reference image are also the same. For example, one is red and the other is blue.

[0128] In step S703, low-frequency information of the sample reference image and high-frequency information of the sample original image are obtained.

[0129] In an optional embodiment, when the server actually obtains the low-frequency information of the sample reference image and the high-frequency information of the sample original image, the low-frequency information includes not only the color information of the object in the sample image, such as hair, but also the low-frequency illumination information of the hair. This is because, although the present disclosure uses the Gaussian blur function to perform Gaussian blur processing on the reference image and removes the high-frequency information to obtain the low-frequency information, the overall illumination information can include high-frequency illumination information and low-frequency illumination information. Therefore, the obtained low-frequency information includes not only the color information in the low-frequency space, but also the low-frequency illumination information.

[0130] Similarly, the high-frequency information of the sample original image may include high-frequency texture information and high-frequency illumination information in the high-frequency space. Since the above texture information is generally in the high-frequency space, the above high-frequency texture information may be referred to as texture information.

[0131] In step S705, the low-frequency information of the sample reference image and the high-frequency information of the sample original image are input into the neural network to obtain a sample composite image.

[0132] Figure 8 3 is a neural network structure diagram according to an exemplary embodiment, including an image generation model 300 and a discriminator 303, wherein the image generation model 300 includes a feature extraction network 301 and a generator 302. The image generation model 300 is used to generate a sample synthetic image, and the discriminator 303 is used to verify which of the two images, the sample synthetic image and the sample original image, is more fake, that is, to distinguish which one is a synthetic image.

[0133] Based on the information contained in the low-frequency information in step S703, the server uses a feature extraction network to extract features from the low-frequency information, and can extract color feature information. In addition, since the image generation model is in the early stage of training and has not yet met the training completion requirement, the feature extraction network therein can not only extract color feature information but also low-frequency illumination feature information when extracting features from the low-frequency information.

[0134] In an optional embodiment, referring to the generator structure mentioned above, the generator may include an encoder and a decoder, wherein the encoder may perform feature extraction on the high-frequency information of the sample original image to obtain texture feature information and / or illumination feature information. Subsequently, the server may use the decoder to perform image synthesis based on texture feature information, illumination feature information (high frequency), color feature information and low-frequency illumination feature information to obtain a sample synthesized image, wherein the object in the sample synthesized image, that is, the texture of the hair is the texture of the hair in the sample original image, and the color of the hair in the sample synthesized image is the color of the hair in the sample reference image. Since the neural network has not been trained at this time, the illumination of the hair in the sample synthesized image at this time may be the superposition of the high-frequency illumination in the sample original image and the low-frequency illumination in the sample reference image, which does not meet the requirements. After training, the feature extraction network can directly extract color feature information without extracting low-frequency illumination feature information.

[0135] In step S707, the sample original image and the sample composite image are input into the discriminator of the neural network to obtain verification result information.

[0136] Continuing to elaborate based on the above embodiment, in an optional embodiment, since the neural network is not trained, the lighting of the hair in the sample composite image can be the superposition of the high-frequency lighting in the sample original image and the low-frequency lighting in the sample reference image, which does not meet the requirements. At this time, the sample original image and the sample composite image are input into the discriminator of the neural network, and the sample original image and the sample composite image are still very different, or in other words, the synthesis traces on the sample composite image are still very obvious. Therefore, the server can easily distinguish which image is synthesized. In this case, the verification result information output by the server can be information such as the difference between the two is very large, and the synthesis signs in the sample re-added image are very obvious. In addition, a series of fields used to represent the above information can also be output.

[0137] Optionally, the discriminator can also be constructed based on a neural convolutional network in a deep learning network. The number of convolutional layers, pooling layers, and upsampling layers included can be set according to actual application requirements. The specific structure can refer to the feature extraction network structure, which will not be repeated here.

[0138] In step S709, the verification result information is used to train the neural network to obtain a trained image generation model.

[0139] In an optional embodiment, the above-mentioned verification result information can be used to train a neural network, and the feature extraction network and the generator in the finally trained neural network are combined into an image generation model.

[0140] In an optional embodiment, the disclosed embodiment further includes a process of specifically using the verification result information to train the neural network, such as Fig. 9 As shown, Fig. 9 is a flowchart of a training process of an image generation model according to an exemplary embodiment, including:

[0141] In step S901, true and false label information of a sample original image and a sample synthesized image is determined.

[0142] In the embodiment of the present disclosure, the specific manner in which the server determines the true or false label information of the sample original image and the sample composite image may be: marking the sample original image as true, and marking the sample composite image as false.

[0143] In step S903, a loss value is determined based on the verification result information and the true or false label information.

[0144] In an optional embodiment, the server may determine a first loss value between the verification result information and the true and false label information based on a first loss function, determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain, and determine the sum of the first loss value and the second loss value as the loss value.

[0145] The above-mentioned first loss function can be a regression loss function such as L2 loss function (mean square error), L1 loss function (mean absolute error), Huber loss function (smoothed mean absolute error), etc., or it can be a classification loss function such as logistic loss function, Hinge loss function, exponential loss function, etc.

[0146] Since the above-mentioned high-frequency information and low-frequency information are both determined based on the frequency domain space, the above-mentioned second loss function can be a regression loss function such as the L2 loss function (mean square error), L1 loss function (mean absolute error), Huber loss function (smoothed mean absolute error) in the frequency domain space.

[0147] In step S905, the neural network is trained according to the loss value to obtain a trained image generation model.

[0148] In the disclosed embodiment, when the loss value is greater than a preset threshold, back propagation is performed based on the loss value to update the network parameters of the neural network to obtain an updated neural network.

[0149] Repeatedly input the low-frequency information into the first feature extraction network of the neural network to obtain color feature information and low-frequency illumination feature information corresponding to the sample reference image; input the texture information and high-frequency illumination information corresponding to the sample original image, as well as the color feature information and low-frequency illumination feature information corresponding to the sample reference image into the generator of the neural network to obtain a sample composite image; input the sample original image and the sample composite image into the discriminator of the neural network to obtain verification result information.

[0150] As mentioned above, when the neural network is not well trained, the illumination of the hair in the sample synthetic image can be the superposition of the high-frequency illumination in the sample original image and the low-frequency illumination in the sample reference image, which does not meet the requirements. At this time, the sample original image and the sample synthetic image are input into the discriminator of the neural network, and the sample original image and the sample synthetic image are still very different, or in other words, the synthetic traces on the sample synthetic image are still very obvious, and the discriminator can easily identify which one is the sample synthetic image. However, as the number of training times increases, the loss function calculates the gradient of the weights of the first feature extraction network and the image generation network in the back propagation, and updates the network weights according to the gradient descent method. At the same time, the low-frequency illumination feature information of the sample reference image will gradually be ignored in the process of image synthesis, that is, the weight of the low-frequency illumination feature information of the sample reference image will become smaller and smaller, until it is close to nothing in the end. At this time, it is difficult for the server to identify which one of the sample original image and the sample synthetic image is true and which one is false.

[0151] In addition, although the low-frequency illumination feature information of the sample reference image is gradually ignored, the encoder also has the ability to generate illumination information. This ability is mainly due to the fact that when the discriminator discriminates the sample synthetic image and the sample original image, it forces the illumination information of the sample synthetic image to be as consistent as possible with the original image. In other words, the final decoder can generate the low-frequency illumination feature information of the sample original image under the pressure of the discriminator, and finally generate a synthetic image based on color feature information, illumination feature information, low-frequency illumination feature information and texture feature information. Until the loss value is less than or equal to the preset threshold, a trained neural network is obtained.

[0152] Since the discriminator is used to identify which of the sample original image and the sample synthetic image is real and which is fake, when the neural network is trained, the parameters of the entire neural network have been adjusted. When the original image and the reference image are directly applied later, the image generation model composed of the first feature extraction network and the generator can be used to generate a synthetic image without using the discriminator.

[0153] In summary, the embodiments of the present disclosure obtain frequency domain information of the original image and the reference image based on the frequency domain space in advance, so that color and texture illumination information can be smoothly decoupled in the frequency domain space, thereby completing the synthesis of objects in the image.

[0154] Fig.10 is a block diagram of an image generating device according to an exemplary embodiment. Fig.10 The device includes an image acquisition module 1001, a low-frequency information acquisition module 1002, a high-frequency information acquisition module 1003, and a synthesis module 1004.

[0155] The image acquisition module 1001 is configured to acquire an original image and a reference image; the original image and the reference image contain target objects of different styles;

[0156] The low frequency information acquisition module 1002 is configured to acquire the low frequency information of the reference image;

[0157] The high-frequency information acquisition module 1003 is configured to acquire the high-frequency information of the original image;

[0158] The synthesis module 1004 is configured to input the low-frequency information and the high-frequency information into an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0159] In an optional embodiment, the low-frequency information acquisition module is configured to execute:

[0160] Gaussian blur processing is performed on the reference image to obtain low-frequency information of the reference image.

[0161] In an optional embodiment, the high-frequency information acquisition module is configured to execute:

[0162] Performing Gaussian blur processing on the original image to obtain low-frequency information of the original image;

[0163] The high-frequency information of the original image is determined according to the difference between the original image and the low-frequency information of the original image.

[0164] In an optional embodiment, the synthesis module includes:

[0165] A feature extraction submodule is configured to perform feature extraction by inputting the low-frequency information into a feature extraction network in the image generation model to obtain first feature information;

[0166] The image synthesis submodule is configured to execute the input of the first feature information and the high-frequency information into the generator in the image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image.

[0167] In an optional embodiment, the generator includes an encoder and a decoder;

[0168] The above-mentioned image synthesis submodule is configured to perform:

[0169] Using the encoder to extract features from the high-frequency information to obtain second feature information;

[0170] The decoder is used to perform image synthesis according to the first feature information and the second feature information, so that the style of the target object in the reference image is transferred to the target object in the original image, thereby obtaining the synthesized image.

[0171] In an alternative embodiment,

[0172] The first characteristic information includes color characteristic information:

[0173] The second feature information includes texture feature information, illumination feature information, texture feature information and / or illumination feature information.

[0174] In an optional embodiment, the image generating device further includes:

[0175] A sample acquisition module is configured to acquire a sample original image and a sample reference image; the sample original image and the sample reference image contain sample target objects of different styles;

[0176] An information acquisition module, configured to acquire low-frequency information of the sample reference image and high-frequency information of the sample original image;

[0177] A sample synthesis module is configured to input the low-frequency information of the sample reference image and the high-frequency information of the sample original image into a neural network to obtain a sample synthesized image;

[0178] A result output module is configured to execute the discriminator of the neural network by inputting the sample original image and the sample composite image to obtain verification result information;

[0179] The training module is configured to train the neural network using the verification result information to obtain the trained image generation model.

[0180] In an optional embodiment, the training module includes:

[0181] A label determination module is configured to determine true and false label information of the sample original image and the sample composite image;

[0182] A loss determination module, configured to determine a loss value based on the verification result information and the true or false label information;

[0183] The model acquisition module is configured to execute training of the above-mentioned neural network according to the above-mentioned loss value to obtain a trained image generation model.

[0184] In an optional embodiment, the above-mentioned loss determination module includes:

[0185] A first loss determination module is configured to determine a first loss value between the verification result information and the true and false label information based on a first loss function;

[0186] A second loss determination module is configured to determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain;

[0187] The total loss determination module is configured to determine the sum of the first loss value and the second loss value as the loss value.

[0188] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0189] Fig.111 is a block diagram of an electronic device 1100 for image generation according to an exemplary embodiment. The electronic device may be a server, or other device having the same function as a server. The internal structure diagram thereof may be as shown in FIG. Fig.11 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image generation method is implemented.

[0190] Those skilled in the art will understand that Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0191] In an exemplary embodiment, a server is also provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image generation method in the embodiment of the present disclosure.

[0192] In an exemplary embodiment, a storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the image generating method in the embodiment of the present disclosure.

[0193] In an exemplary embodiment, a computer program product including instructions is also provided. When the computer program product is run on a computer, the computer is caused to execute the image generating method in the embodiment of the present disclosure.

[0194] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0195] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0196] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: include: Acquire an original image and a reference image; the original image and the reference image contain target objects of different styles; Acquiring low-frequency information of the reference image; Acquiring high-frequency information of the original image; Inputting the low-frequency information and the high-frequency information into an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image, thereby obtaining a composite image; The step of inputting the low-frequency information and the high-frequency information into an image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image includes: Inputting the low-frequency information into a feature extraction network in the image generation model to perform feature extraction to obtain first feature information; Inputting the first feature information and the high-frequency information into a generator in the image generation model to transfer the style of the target object in the reference image to the target object in the original image to obtain a composite image; the generator includes an encoder and a decoder; The step of inputting the first feature information and the high-frequency information into a generator in the image generation model so as to transfer the style of the target object in the reference image to the target object in the original image to obtain a synthesized image includes: Using the encoder to extract features from the high-frequency information to obtain second feature information; Using the decoder to perform image synthesis according to the first feature information and the second feature information, so as to transfer the style of the target object in the reference image to the target object in the original image, thereby obtaining the synthesized image; The training method of the image generation model includes: Acquire a sample original image and a sample reference image; the sample original image and the sample reference image contain sample target objects of different styles; Acquire low-frequency information of the sample reference image and high-frequency information of the sample original image; Inputting the low-frequency information of the sample reference image and the high-frequency information of the sample original image into a neural network to obtain a sample synthetic image; Inputting the sample original image and the sample composite image into the discriminator of the neural network to obtain verification result information; Determining true and false label information of the sample original image and the sample composite image; Determine a first loss value between the verification result information and the true and false label information based on a first loss function; Determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain; determining a sum of the first loss value and the second loss value as the loss value; The neural network is trained according to the loss value to obtain the trained image generation model.

2. The image generation method according to claim 1, characterized in that: The acquiring the low-frequency information of the reference image comprises: Gaussian blur processing is performed on the reference image to obtain low-frequency information of the reference image.

3. The image generation method according to claim 1, characterized in that: The obtaining of high-frequency information of the original image comprises: Performing Gaussian blur processing on the original image to obtain low-frequency information of the original image; The high-frequency information of the original image is determined according to a difference between the original image and the low-frequency information of the original image.

4. The image generation method according to claim 1, characterized in that: The first characteristic information includes color characteristic information: The second feature information includes texture feature information and / or illumination feature information.

5. An image generating device, characterized in that: include: An image acquisition module is configured to acquire an original image and a reference image; the original image and the reference image contain target objects of different styles; A low-frequency information acquisition module, configured to acquire the low-frequency information of the reference image; A high-frequency information acquisition module, configured to acquire high-frequency information of the original image; a synthesis module, configured to input the low-frequency information and the high-frequency information into an image generation model, so as to transfer the style of the target object in the reference image to the target object in the original image, thereby obtaining a synthesized image; The synthesis module comprises: A feature extraction submodule is configured to execute feature extraction by inputting the low-frequency information into a feature extraction network in the image generation model to obtain first feature information; an image synthesis submodule, configured to execute inputting the first feature information and the high-frequency information into a generator in an image generation model, so as to transfer the style of the target object in the reference image to the target object in the original image, and obtain a synthesized image; the generator includes an encoder and a decoder; The image synthesis submodule is configured to perform: Using the encoder to extract features from the high-frequency information to obtain second feature information; Using the decoder to perform image synthesis according to the first feature information and the second feature information, so as to transfer the style of the target object in the reference image to the target object in the original image, thereby obtaining the synthesized image; The image generating device further comprises: A sample acquisition module is configured to acquire a sample original image and a sample reference image; the sample original image and the sample reference image contain target objects of different styles; An information acquisition module, configured to acquire low-frequency information of the sample reference image and high-frequency information of the sample original image; A sample synthesis module is configured to input low-frequency information of the sample reference image and high-frequency information of the sample original image into an image generation model of a neural network to obtain a sample synthesized image; A result output module is configured to execute a discriminator that inputs the sample original image and the sample composite image into the neural network to obtain verification result information; A training module is configured to train the neural network using the verification result information to obtain the trained image generation model; The training module includes: A label determination module, configured to determine true and false label information of the sample original image and the sample composite image; A loss determination module, configured to determine a loss value based on the verification result information and the true or false label information; A model acquisition module is configured to train the neural network according to the loss value to obtain an image generation model; A first loss determination module is configured to determine a first loss value between the verification result information and the true and false label information based on a first loss function; A second loss determination module is configured to determine a second loss value between the verification result information and the true and false label information based on a second loss function in the frequency domain; The total loss determination module is configured to determine the sum of the first loss value and the second loss value as the loss value.

6. The image generating device according to claim 5, characterized in that: The low-frequency information acquisition module is configured to execute: Gaussian blur processing is performed on the reference image to obtain low-frequency information of the reference image.

7. The image generating device according to claim 5, characterized in that: The high-frequency information acquisition module is configured to execute: Performing Gaussian blur processing on the original image to obtain low-frequency information of the original image; The high-frequency information of the original image is determined according to a difference between the original image and the low-frequency information of the original image.

8. The image generating device according to claim 5, characterized in that: The first characteristic information includes color characteristic information: The second feature information includes texture feature information and / or illumination feature information.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image generating method according to any one of claims 1 to 4.

10. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image generating method according to any one of claims 1 to 4.

Images