Image conversion method, device, equipment and storage medium

Abstract

Provided are an image conversion method, device, equipment and storage medium, wherein an image conversion network is used to determine a first field corresponding to a first image; based on the first field and a second field corresponding to a preset image, a feature mapping relationship of the first image and the preset image at each layer of the image conversion network is determined; and based on the feature mapping relationship, a picture of the first image is converted from the first field to the second field to obtain a second image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of image processing, and relate to but are not limited to an image conversion method, device, equipment and storage medium. BACKGROUND

[0002] In the related art, image conversion is achieved based on cycle consistency, which requires that the image converted from the original field to the target field can be converted back to the original field, so as to reconstruct the image. However, for conversion between fields with large morphological differences, the converted image often loses too much information to effectively reconstruct the original image, so it is difficult to achieve ideal results. SUMMARY

[0003] Embodiments of the present application provide an image conversion technical solution.

[0004] The technical solution of the embodiments of the present application is implemented as follows:

[0005] The embodiments of the present application provide an image conversion method, which comprises:

[0006] determining a first field corresponding to a first image by using an image conversion network;

[0007] determining a feature mapping relationship of the first image and a preset image at each layer of the image conversion network based on the first field and a second field corresponding to the preset image;

[0008] converting a picture of the first image from the first field to the second field based on the feature mapping relationship to obtain a second image.

[0009] The embodiments of the present application provide an image conversion device, which comprises:

[0010] a first determining module configured to determine a first field corresponding to a first image by using an image conversion network;

[0011] a second determining module configured to determine a feature mapping relationship of the first image and a preset image at each layer of the image conversion network based on the first field and a second field corresponding to the preset image;

[0012] a first conversion module configured to convert a picture of the first image from the first field to the second field based on the feature mapping relationship to obtain a second image.

[0013] Correspondingly, the embodiments of the present application provide a computer storage medium, which stores computer executable instructions, and the computer executable instructions can implement the method steps described above after being executed.

[0014] The embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores computer executable instructions, and the processor can realize the steps of the above method when running the computer executable instructions on the memory.

[0015] The embodiment of the present application provides an image conversion method, device, equipment and storage medium. First, an image conversion network is used to analyze a domain to which a picture of a first image belongs. Then, a content mapping relationship between the first image and a second image at each layer is determined by analyzing the same feature category between a first domain and a second domain of a preset image. Thus, the picture of the first image can be converted from the first domain to the second domain based on the content mapping relationship at each layer, and a converted second image is obtained. In this way, the natural and accurate image conversion can be realized by establishing the content mapping relationship between the first image and the preset image at each layer. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 An implementation flowchart of the image conversion method provided by the embodiment of the present application is shown.

[0017] Figure 2 Another implementation flowchart of the image conversion method provided by the embodiment of the present application is shown.

[0018] Figure 3 An application scenario diagram of the image conversion method provided by the embodiment of the present application is shown.

[0019] Figure 4 A network architecture diagram for generating a priori distillation provided by the embodiment of the present application is shown.

[0020] Figure 5 A structure diagram of the image conversion network provided by the embodiment of the present application is shown.

[0021] Figure 6 A structure composition diagram of the image conversion device provided by the embodiment of the present application is shown.

[0022] Figure 7 A composition structure diagram of the electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the specific technical scheme of the application will be further described in detail below with reference to the drawings in the embodiment of the present application. The following embodiments are used to illustrate the present application, but not to limit the scope of the present application.

[0024] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or different subsets as each other and can be combined with each other as long as there is no conflict.

[0025] In the following description, the terms "first\second\third" are merely to distinguish similar objects, and do not represent a specific order of the objects. It is understood that the "first\second\third" can be interchanged in a specific order or sequence as long as it is allowed, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing embodiments of the application only and is not intended to be limiting of the application.

[0027] Before further detailing the embodiments of the application, the terms and phrases involved in the embodiments of the application are explained, and the terms and phrases involved in the embodiments of the application are applicable to the following explanations.

[0028] 1) Conditional Generative Adversarial Nets (cGAN), the generator of cGAN takes noise and labels as input, the discriminator takes generated objects, original objects and labels as input, and cGAN optimizes the judgment criteria of the discriminator to distinguish false data with arbitrary labels and true data with wrong labels as false, and distinguish true data with correct labels as true.

[0029] 2) Random noise, random noise is a random variable with Gaussian distribution. Adversarial networks are originally used to complete a distribution registration transformation. The generator of the adversarial network is to transform a Gaussian distribution (or other prior random distribution) into a target data distribution. Here, the input random variable is actually treated as a hidden space of data.

[0030] The following describes an exemplary application of the image conversion system provided by the embodiments of the application, wherein the terminal in the system provided by the embodiments of the application can be various types of electronic devices with image acquisition functions, such as notebook computers, tablet computers, desktop computers, mobile devices (e.g., personal digital assistants, dedicated messaging devices, portable game devices), etc.

[0031] The following will illustrate an exemplary application of the image conversion system when implemented as an electronic device.

[0032] Figure 1An implementation flowchart of the image conversion method provided by the embodiments of the present application is shown in FIG. 1, which is described in combination with the steps shown in FIG. 2. Figure 1 Figure 1

[0033] In step S101, a first image is input into an image conversion network to determine a first field corresponding to the first image.

[0034] In some embodiments, the first image can be an image captured in any scene, can be an image with complex picture content, or can be an image with simple picture content. The first image is input into the image conversion network to determine a field to which a picture of the first image belongs. The first field corresponding to the first image is a field to which picture content of the first image belongs; wherein the picture content can be at least one of the following: content indicated by an image region with an area ratio exceeding a preset threshold, foreground-indicated content, center region-indicated content, etc. By analyzing a scene in which the picture content is located, a field to which the picture content belongs is determined, i.e., the first field corresponding to the first image is obtained. The field to which the picture content belongs refers to a scene category of the picture content. For example, if the content indicated by the image region with the area ratio exceeding the preset threshold in the first image is a cat, it is determined that the first field is a cat; or if the foreground-indicated picture content in the first image is a building, it is determined that the first field is a building. The image conversion network is used to implement conversion of the picture of the first image from the first field to a second field of a preset image, which is obtained by determining prior information between a first sample image and a second sample image and establishing a mapping relationship between multi-layer content features based on the content features in each layer, so as to introduce the multi-layer content feature mapping relationship in the generator for training.

[0035] In step S102, based on the first field and a second field corresponding to the preset image, a feature mapping relationship of the first image and the preset image in each layer of the image conversion network is determined.

[0036] ​​In some embodiments, the preset image is an image whose picture content belongs to a second domain. The preset image can be an image captured in the second domain, or a random image generated by random noise. The picture content of the image indicated by the first domain is different from the picture content of the image indicated by the second domain; in some possible implementations, the domain of an image is determined by analyzing the category of the object in the picture content of the image. For example, if the picture content in the first image is a puppy, it is determined that the first domain of the first image is dog, and if the picture content in the preset image is a human face, it is determined that the second domain of the preset image is portrait. By analyzing the first domain of the first image and the second domain of the preset image, the same feature categories in the picture of the first image and the picture of the second image can be determined, and then the mapping relationship between these feature categories can be established. Each layer of the image conversion network represents a level of feature extraction; the feature mapping relationship of each layer is described by taking a single layer in the image conversion network as an example, wherein the feature mapping relationship of one layer is used to represent the mapping relationship between the content features of the same categories in the first image and the preset image at this layer; for example, if the first image is a dog face image and the preset image is a cat face image, the corresponding relationship between the feature categories of the dog face in the first image and the feature categories of the cat face in the preset image, and the corresponding relationship between the direction of the dog in the first image and the direction of the cat in the preset image, i.e., the mapping relationship of the content features of the first image and the preset image, are represented.

[0037] In step S103, the picture of the first image is converted from the first domain to the second domain based on the feature mapping relationship, to obtain a second image.

[0038] In some embodiments, the feature mapping relationship of each layer is input into the generator of the image conversion network, to realize the conversion of the first image from the first domain to the second domain, and to generate a second image.

[0039] In the embodiments of the present application, first, the domain to which the acquired picture of the first image belongs is analyzed; then, by analyzing the same feature categories between the first domain and the second domain of the preset image, the content mapping relationship of the first image and the second image at each layer is determined, so that the picture of the first image can be converted from the first domain to the second domain based on the content mapping relationship, to obtain a converted second image; in this way, by establishing the content mapping relationship of the first image and the preset image at each layer, natural and accurate image conversion can be realized.

[0040] In some embodiments, by establishing a cross-layer connection between the content encoder and the generator of the image conversion network, the content features and style features at each level are extracted, and the picture of the first image is converted from the first domain to the second domain in combination with the feature mapping relationship at each level, to obtain a second image; that is, the above step S103 can be realized by the following steps S131 and S132 (not shown in the figure).

[0041] Step S131: Use the style encoder in the image conversion network to determine the style features of the preset image in each layer of the image conversion network.

[0042] In some possible implementations, a preset image is input into the style encoder of an image transformation network to generate style features of the preset image at each layer.

[0043] Step S132: Take the style features of any layer in the image conversion network and the corresponding feature mapping relationship as a feature group.

[0044] In some possible implementations, for a layer in an image transformation network, the style features of that layer, as well as the feature mapping relationship between the first image and the preset image at that layer, are combined into a feature group, thus obtaining the feature group for each layer.

[0045] Step S133: Input the feature groups of each layer in the image conversion network into the generator of the image conversion network to obtain the second image.

[0046] In some possible implementations, a style encoder in the image transfer network is used to extract style features of a preset image in the second domain. These style features characterize the image style of the second domain, such as background, color, hue, texture, brightness, and foreground. The feature mapping relationship and the style features of the second domain are then input into the generator of the image transfer network to generate the second image. Thus, by combining the feature mapping relationships introduced from each layer with the style features of the preset image, more accurate domain transfer of the image can be achieved.

[0047] In some embodiments, to establish the mapping relationship between the content features of each layer, the feature mapping relationship between the first image and the preset image in each layer is determined by analyzing the feature categories shared between the two domains. That is, step S102 above can be achieved through... Figure 2 The steps shown are to be completed. Figure 2 This is a schematic diagram illustrating another implementation flow of the image conversion method provided in the embodiments of this application, combined with... Figure 2 The steps shown are explained below:

[0048] Step S201: Based on the first domain and the second domain, determine the feature categories shared between the first image and the preset image at each layer of the image conversion network.

[0049] In some embodiments, each layer represents a level of feature extraction of the image; for example, the level of the content encoder is determined as the level of feature extraction of the image by performing feature extraction on the first image and the preset image by the content encoder. By analyzing the internal relationship between the two fields, the same high-level feature categories and middle-level feature categories between the first image and the preset image are determined based on the internal relationship. For example, the first field is cat and the second field is dog, and the internal relationship between the two fields is the relationship between abstract content feature categories and the relationship between concrete content feature categories; wherein the abstract content feature categories include one or more of face orientation and position of the picture content in the image; the same concrete content feature categories of the first field and the second field include one or more of facial feature categories, limb feature categories and body feature categories.

[0050] In some possible implementations, an image conversion network is used to determine the content features shared by the first image and the preset image at each level. The image conversion network includes a content encoder, and the content encoder is used to perform feature extraction on the first field of the first image and the preset image at any layer of the content encoder, to obtain image features of the first image at the layer and image features of the preset image at the layer; based on this, the content feature categories commonly possessed by the first field of the first image and the preset image at any layer are determined.

[0051] In some embodiments, by analyzing the internal relationship between the two fields, the high-level feature categories and the middle-level feature categories between the first image and the preset image are determined, so as to obtain the feature categories shared by each layer, that is, the above step S201 can be realized by the following steps S211 to S213 (not shown in the figure):

[0052] Step S211, based on the first field and the second field, determining the abstract content feature categories shared by the first image and the preset image.

[0053] In some embodiments, by analyzing the internal relationship between the first field and the second field, the abstract content feature categories that are semantically the same between the first image and the preset image are determined, for example, the orientation and position of the picture in the first image and the orientation and position of the picture in the preset image.

[0054] Step S212, determining the concrete content feature categories shared by the first image and the preset image at each layer.

[0055] In some embodiments, the same concrete content feature category of the image content of the first image and the image content of the second image is determined by determining the number of layers of the content encoder for feature extraction of the first image, wherein the concrete content feature category includes a category of specific content features that the first image and the second image both have; for example, the first image is a human face image, which has facial features, and the preset image is a cat image, which also has facial features, and the facial features are the concrete content feature category shared by the first image and the preset image at the intermediate level of the image conversion network.

[0056] In step S213, the feature category shared by each layer is obtained based on the abstract content feature category and the concrete content feature category shared by each layer.

[0057] In some embodiments, the abstract content feature category and the concrete content feature category shared by each layer are used as the feature category shared by the first image and the preset image. In this way, by analyzing the internal relationship between the first field of the first image and the second field of the preset image, the abstract content feature category and the concrete content feature category of each layer shared by the first image and the preset image are determined, so that the feature category shared by the first image and the preset image at each layer can be accurately obtained.

[0058] After step S201, each layer of the image conversion network is taken as a target layer, and the following steps are performed:

[0059] In step S202, the first output feature of the first image at the target layer and the second output feature of the preset image at the target layer are respectively determined based on the feature category shared by the target layer.

[0060] In some embodiments, the first output feature of the first image at the target layer is determined based on the feature category shared by the target layer. For each layer of the image conversion network, the encoder feature of the feature category of the layer is obtained by performing feature extraction on the first image at the layer, and the encoder feature is fused with the generator feature output by the generator to obtain the feature finally output by the layer, i.e., the first output feature. Similarly, for each layer of the image conversion network, the encoder feature of the feature category of the layer is obtained by performing feature extraction on the preset image at the layer, and the encoder feature is fused with the generator feature output by the generator to obtain the feature finally output by the layer, i.e., the second output feature.

[0061] In some possible implementation manners, the content feature output by the encoder and the generator feature output by the generator are fused through the mask information to obtain the output feature of the final output of the layer, that is, the step S202 of determining the first output feature of the first image in the target layer based on the shared feature categories of the target layer can be implemented through the following steps S221-S223 (not shown in the figure):

[0062] In step S221, the content encoder of the image conversion network is used to determine the encoder feature of the shared feature categories in the first image.

[0063] In some embodiments, the first image is input into the content encoder of the image conversion network to obtain the content feature of the first image corresponding to the same feature categories of the first image and the preset image in the target layer of the content encoder, that is, the encoder feature. The content encoder in the image conversion network is used to extract the content feature of the first image to obtain the encoder feature belonging to the feature categories. For example, the first image is a cat face image, the preset image is a dog face image, and the feature categories of each layer include facial feature categories, facial orientation and position information, and the encoder feature includes facial feature, orientation feature and position feature.

[0064] In step S222, the encoder feature of the target layer is input into the generator of the image conversion network to determine the generator feature of the target layer.

[0065] In some embodiments, the encoder feature of the target layer is input into the generator of the image conversion network to obtain the generator feature of the layer. In other embodiments, the encoder feature of the target layer is input into the generator, and at the same time, the style feature of the preset image extracted by the style encoder of the image conversion network is also input into the generator, so that the generator generates the generator feature of the target layer.

[0066] In step S223, the generator feature of the target layer and the encoder feature of the target layer are fused to obtain the first output feature of the first image in the target layer.

[0067] In some embodiments, in any layer of the image conversion network, the generator feature output by the generator and the encoder feature output by the content encoder are fused to obtain the feature of the final output of the layer, i.e., the first output feature. In this way, before the high-level content features of the first image are extracted by the content encoder, the intermediate-level features of the first image are introduced into the generator through the cross-layer connection between the content encoder and the generator, so that the mapping relationship between the high-level features and the intermediate-level features can be established; in this way, multiple levels of features are introduced into the generator, so that the generator has more detailed features, and thus the second image generated by the generator has more detailed features, and the domain conversion of the first image to the preset image is more accurately realized.

[0068] In other embodiments, the implementation process of "determining the second output feature of the preset image in the target layer based on the feature categories shared by the target layer" in step S202 is similar to the implementation process of steps S221 to S223 described above, i.e., first, the second content feature of the preset image in the target layer is extracted by the content encoder of the image conversion network; in some possible implementation manners, if the preset image is a random image generated by random noise, then the second content feature is the content feature of the random image collected by the content encoder. If the preset image is an image with a preset picture collected, then the second content feature is the feature belonging to the feature category of each layer in the preset picture. Then, the second content feature is input into the generator to obtain the generator feature of the layer; finally, the generator feature output by the generator and the content feature output by the content encoder are fused to obtain the second output feature of the preset image output in the layer.

[0069] Step S203, determining the feature mapping relationship between the first image and the preset image in the target layer based on the first output feature and the second output feature.

[0070] In some embodiments, in the image conversion network, the feature mapping relationship between the content features of the first image and the preset image in each layer is determined. The feature mapping relationship can represent the corresponding relationship between the content features of the first image and the preset image in the target layer belonging to the same feature category. In this way, by analyzing the feature categories shared between the two domains, the mapping relationship between the content features of the first image and the preset image at each level can be accurately established, and the first image can be accurately converted into the second image of the second domain.

[0071] In some embodiments, based on the cross-layer connection between the content encoder and the generator, the intermediate-level concrete content feature is updated through the hidden state of the cross-layer connection to obtain the content feature of the layer, i.e., the above-mentioned step S221 can be implemented by the following steps:

[0072] In the first image, abstract content features belonging to an abstract content feature category in the feature categories shared by the target layer and concrete content features belonging to a concrete content feature category in the feature categories shared by the target layer are determined by using the content encoder.

[0073] In some embodiments, the abstract content feature category is determined in the feature categories shared by the layers, i.e., the feature categories shared by the first image and the preset image at a high layer of the image conversion network are determined; the content features corresponding to the feature categories of the high layer are the abstract content features. The concrete content features belonging to the concrete content feature category in the feature categories shared by the target layer are the content features of the first image corresponding to the feature categories shared by the first image and the preset image at an intermediate layer or a low layer of the image conversion network.

[0074] In the second step, the initial hidden state is determined based on the abstract content features.

[0075] In some possible implementations, the high-level content features of the first image, i.e., the abstract content features, extracted by the content encoder are used as the initial hidden state of the layer.

[0076] In the third step, the concrete content features of the target layer are updated based on the initial hidden state to obtain the encoder features of the target layer.

[0077] In some possible implementations, the hidden state of the second layer is obtained by fusing the initial hidden state and the content features of the second layer, i.e., the hidden state of the (i-1)th layer and the content features (i.e., the concrete content features) of the ith layer are input into the activation layer of the update network R, the hidden state of the (i-1)th layer is convolved with the concrete content features of the ith layer, and the convolution result is input into the activation function of the activation layer. The output of the activation function is multiplied element by element with the hidden state of the (i-1)th layer to obtain the hidden state of the ith layer. The hidden state of the ith layer and the concrete content features of the ith layer are concatenated and input into the feature update network U for convolution operation, and the convolution result is input into the activation function, so as to update the concrete content features of the ith layer to obtain the encoder features of the ith layer. Thus, the hidden state of the next layer is updated by the hidden state of the previous layer, and the hidden state and the concrete content features of the same layer are convolved to update the concrete content features, so as to obtain the encoder features of the encoder output of the layer; thus, the content features of the first image at the intermediate level of the network are introduced into the network.

[0078] In some embodiments, the encoder features of the multi-layers of the first image are introduced through the dynamic cross-layer connection established between the content encoder and the generator, so that the final output features of each layer can be obtained, i.e., the above step S222 can be implemented by the following steps:

[0079] In a first step, the mask information of the first image at the target layer is determined based on the hidden state of the target layer and the encoder features of the target layer.

[0080] In some embodiments, the abstract content features of the high layer are input into the mask network as the initial hidden state of the first image together with the concrete content features of any layer of the intermediate layers, to obtain the mask information of the layer. The mask information of the layer is used to select useful content features, i.e., target content features, from the content features of the layer, which generally refer to the content features of the subject region of the image, i.e., the content features other than the redundant information, to be input into the generator to generate the second image. The redundant information refers to the content features corresponding to the content in the image that does not directly point to the domain, for example, in the case of the domain being a building, the content features corresponding to the trees, blue sky, etc. in the image that are not buildings can be regarded as redundant information.

[0081] In some possible implementations, after the dynamic cross-layer connection is established between the content encoder and the generator of the image conversion network, the hidden state of the dynamic connection and the content features of any layer are obtained to determine the mask information of the layer, i.e., the above first step can be implemented by the following steps:

[0082] Step 1, in the case that the target layer is the first layer, the abstract content features are determined as the hidden state of the first layer; or in the case that the target layer is the i-th layer, the hidden state of the i-th layer is determined based on the hidden state of the (i-1)-th layer and the concrete content features of the i-th layer.

[0083] In some possible implementations, i is an integer greater than 1; and in the case that i is equal to 2, the hidden state of the first layer is the abstract content features. That is, the initial hidden state is the content features of the first image extracted by the image conversion network at the high layer. The initial hidden state and the concrete content features of the second layer are concatenated and then input into the activation layer of the image conversion network for upsampling and convolution to obtain the hidden state of the second layer; similarly, the hidden state of the second layer can be updated based on the hidden state of the second layer and the concrete content features of the third layer to obtain the hidden state of the third layer; and in this way, the hidden state of any layer can be obtained.

[0084] Step 2, the mask information of the first image at the (j+1)-th layer is determined based on the hidden state of the j-th layer and the concrete content features of the (j+1)-th layer.

[0085] In some possible implementation manners, j is an integer greater than or equal to 1. The hidden state of the jth layer and the embodied content feature of the (j+1)th layer are concatenated, and then input to an activation layer of the mask network for upsampling and convolution to obtain mask information of the (j+1)th layer. In some possible implementation manners, in order to enable the obtained mask information to more accurately select an effective part in the content feature, the mask information is subjected to a regularization process, so that the mask information is as sparse as possible. In this way, by means of the hidden state of any layer in the dynamic cross-layer connection and the content feature, the mask information of the first image can be determined, thereby facilitating selection of the target content feature from the content feature of the first image and improving the image conversion speed of the image conversion network.

[0086] Secondly, the embodied content feature of the target layer and the mask information of the target layer are input to the generator to generate generator feature of the target layer.

[0087] In some embodiments, the effective part in the intermediate layer content feature is selected by the mask information of the target layer and input to the generator. By means of the value in the mask information of any layer, the feature corresponding to the value 1 in the embodied content feature of the layer is selected and input to the generator to obtain the generator feature of the generator at the layer.

[0088] In some possible implementation manners, in order to reduce redundant information input to the generator and reduce the calculation amount of the generator, the above-mentioned second step can be implemented by the following steps:

[0089] Step a: subjecting the mask information of the target layer to a regularization process to obtain sparse mask information of the target layer.

[0090] In some possible implementation manners, L2 or L1 regularization is adopted to subject the mask information of any layer to a regularization process, so that the mask information is as sparse as possible to obtain sparse mask information of the layer. Since the sparse mask information changes the vector corresponding to the background feature in the mask information to 0, that is, discards the image picture redundant information in the mask information, the sparse mask information includes the target content feature of the target layer, thereby reducing the invalid information in the mask information. For example, the first image is a cat face image, and the information other than the cat face feature in the image is invalid information, for example, background information.

[0091] Step b: obtaining, from the embodied content feature of the target layer, a target content feature matched with the sparse mask information of the target layer.

[0092] In some possible implementation manners, in the concrete content features of the target layer, a target content feature matched with the sparse mask information of the target layer is selected; wherein the target content feature matched with the sparse mask information of the target layer can be understood as, in the content features of the layer, a content feature corresponding to a value 1 in the sparse mask information is selected, so that the selected features rarely include redundant information, and automatic screening of the multi-layer content features is implemented.

[0093] In step c, the target content feature of the target layer is input into the generator to generate the generator feature of the target layer.

[0094] In some possible implementation manners, the target content feature of the layer is input into the generator to enable the generator to output the generator feature of the layer. In this way, the effective part is selected from the concrete content features of the layer by using the sparse mask information as the input of the generator, so that the calculation amount of the generator is reduced.

[0095] In step c, the target content feature of the target layer is input into the generator to generate the generator feature of the target layer.

[0096] In some embodiments, in any layer, the encoder feature and the generator feature are fused by using the mask information of the layer to obtain the final output feature of the layer, that is, the first output feature. In this way, the effective features of the layers introduced between the content encoder and the generator are dynamically screened.

[0097] In some possible implementation manners, the encoder feature and the generator feature are fused by using the mask information and the adversarial information of the mask information, that is, the above-mentioned third step can be implemented by the following process:

[0098] First, the encoder feature of the target layer is adjusted based on the mask information of the target layer to obtain the first adjustment feature of the target layer.

[0099] In some possible implementation manners, in any layer, the mask information of the layer is multiplied with the encoder feature element by element to obtain the first adjustment feature of the layer.

[0100] Secondly, the generator feature of the target layer is adjusted based on the adversarial information of the mask information of the target layer to obtain the second adjustment feature of the target layer.

[0101] In some possible implementation manners, the adversarial information of the mask information of the target layer can be a sum of the opposite number of the mask information and a preset value (for example, a preset value 1); the second adjustment feature of the layer is obtained by multiplying the adversarial information with the generator feature element by element.

[0102] Finally, the first adjusted feature of the target layer and the second adjusted feature of the target layer are fused to obtain the first output feature of the target layer.

[0103] In some possible implementations, in any layer, the obtained first adjusted feature and the second adjusted feature are element-wise summed to obtain the final first output feature of the layer. In this way, the encoder feature generator features of the target layer are fused through the mask information, and the introduction of the content features of the intermediate layers in the network can be realized.

[0104] In some embodiments, the image conversion network is obtained by training a to-be-trained network based on a first sample image in which a picture belongs to a first sample field and a second sample image in which a picture belongs to a second sample field. The training process of the to-be-trained network can be implemented through the following steps S151 to S156 (not shown in the figure):

[0105] In step S151, the first sample image and the second sample image related to the picture are obtained.

[0106] In some embodiments, the first sample image and the second sample image have related prior information, and the picture of the first sample image and the picture of the second sample image have a certain correlation. For example, the first sample image and the second sample image are generated based on the same random noise, and the orientation and pose of the object in the image picture are the same.

[0107] In some possible implementations, the first sample image and the second sample image can be obtained through the following process:

[0108] First, a preset noise and a domain label set including at least a first sample field and a second sample field are obtained.

[0109] In some possible implementations, the preset noise can be Gaussian random noise, as shown in Figure 3 The preset noise can be Z1, Z2 or Z3; the domain label set includes multiple domain labels, as shown in Figure 3 including the domain bird, the domain cat and the domain dog, etc.

[0110] Then, the preset noise and the domain label set are input into a trained adversarial network to generate the first sample image and the second sample image with matching semantic information.

[0111] In some possible implementation manners, the trained adversarial network can be a bigGAN (big Generative Adversarial Network), preset noise and a domain label set are input into the adversarial network as conditional information, and a sample image corresponding to each random noise and different domain labels is output; and then the first sample image and the second sample image are obtained. In this way, the conditional adversarial network is used to generate sample images based on the same random noise and two random domains, so that the trained content encoder can extract content features shared by the sample images, and the shape and appearance of the sample images are beneficial to be reconstructed.

[0112] In step S152, a trained content encoder in the to-be-trained network is used to determine first sample features of the first sample image at each layer of the trained content encoder and second sample features of the second sample image at the each layer.

[0113] In some embodiments, the trained content encoder in the to-be-trained network is obtained by training based on the first sample image and the second sample image with prior information. The first sample image and the second sample image are input into two trained content encoders respectively, to obtain content features of the first sample image at each layer of the content encoder, that is, first sample features, and content features of the second sample image at each layer of the content encoder, that is, second sample features.

[0114] In some possible implementation manners, the to-be-trained content encoder is trained based on the first sample image and the second sample image, to obtain the trained content encoder; and the training can be implemented by the following steps:

[0115] In step one, the to-be-trained content encoder is used to determine first predicted content features of the first sample image and second predicted content features of the second sample image.

[0116] In some possible implementation manners, the first sample image and the second sample image are input into the to-be-trained content encoder; the to-be-trained content encoder is used to extract content features of the entire image of the first sample image, that is, first predicted content features; and the to-be-trained content encoder is also used to extract content features of the entire image of the second sample image, that is, second predicted content features.

[0117] In step two, the first predicted content features, sample style features of the first sample image and the first sample domain are input into a decoder, to reconstruct the first sample image, to obtain a second reconstructed sample image.

[0118] In some possible implementations, the first sample image is input to the style encoder to extract a sample style feature of the first sample image. The first predicted content feature of the first sample image, the sample style feature, and a domain label of the first sample domain are input to the decoder to cause the decoder to reconstruct the first sample image to obtain a second reconstructed image.

[0119] In a third step, a first loss between the second reconstructed sample image and the first sample image is determined.

[0120] In some possible implementations, the first loss is determined by analyzing a difference between a picture content of the second reconstructed sample image and a picture content of the first sample image. That is, the first loss is used to measure a similarity between the second reconstructed image and the first sample image.

[0121] In a fourth step, a second loss between the second predicted content feature and the first predicted content feature is determined.

[0122] In some possible implementations, the second loss is obtained by analyzing a difference between the second predicted content feature of the second sample image and the first predicted content feature of the first sample image. That is, the second loss represents a similarity between the second sample image and the first sample image.

[0123] In a fifth step, network parameters of the to-be-trained content encoder are adjusted based on the first loss and the second loss to obtain a trained content encoder.

[0124] In some possible implementations, the network parameters of the to-be-trained content encoder are adjusted by combining the first loss and the second loss, so that the trained content encoder can extract high-level content features shared by the first sample image and the second sample image. In this way, by training the to-be-trained content encoder using the sample image set with prior information, the accuracy of the trained content encoder in extracting the same high-level content features can be improved.

[0125] In step S153, a sample feature mapping relationship between the first sample image and the second sample image at each layer of the trained content encoder is determined based on the first sample feature of each layer and the second sample feature of each layer.

[0126] In some embodiments, the first sample feature of each layer of the first sample image and the second sample feature of each layer of the second sample image are extracted by the trained encoder, and a corresponding relationship between the first sample feature and the second sample feature is determined, so as to obtain the sample feature mapping relationship of multiple layers.

[0127] Step S154, inputting the sample feature mapping relationship of each layer and the sample style feature of the second sample image into the first generator of the to-be-trained network to obtain a converted sample image that converts the picture of the first sample image from the first sample field to the second sample field.

[0128] In some embodiments, in any layer, the sample feature mapping relationship of the layer and the sample style feature of the second sample image are both input into the first generator to predict a converted sample image that converts the picture of the first sample image from the first sample field to the second sample field.

[0129] Step S155, inputting the sample style feature of the second sample image and the second sample feature of each layer into the second generator of the to-be-trained network to obtain a first reconstructed sample image that reconstructs the picture of the second sample image.

[0130] In some embodiments, the sample style feature of the second sample image and the second sample feature of each layer are input into the second generator to predict a first reconstructed sample image that reconstructs the second sample image.

[0131] Step S156, adjusting the network parameters of the to-be-trained network based on the loss of the converted sample image and the loss of the first reconstructed sample image to obtain the image conversion network.

[0132] In some embodiments, the loss of the converted sample image is determined by the difference between the picture of the converted sample image and the second sample image; the loss of the first reconstructed sample image is determined based on the difference between the picture of the first reconstructed sample image and the second sample image; and based on this, the network parameters such as weights of the to-be-trained network are adjusted to obtain the image conversion network.

[0133] The above steps S151 to S156 provide an implementation manner of training the to-be-trained network to obtain the image conversion network, in which manner, by first obtaining the sample images with prior correlation, the trained content encoder is trained, then the cross-layer connection is established between the content encoder and the generator, thereby introducing the content features of the intermediate layers of the sample images, and further the accuracy of the image conversion network trained can be improved.

[0134] In some possible implementation manners, by respectively determining the difference between the converted sample image and the first sample image, the difference between the first reconstructed sample image and the second sample image, and the adversarial loss of the entire network, the adjustment of the network parameters of the to-be-trained network is realized, i.e., the above step 156 can be realized by the following steps:

[0135] In a first step, a loss of the converted sample image is determined based on a content feature of the converted sample image and a content feature of the first sample image.

[0136] In some embodiments, first, a difference between a content feature of an entire image of the converted sample image extracted by the trained content encoder and a content feature of an entire image of the first sample image is determined; then, the difference is taken as the loss of the converted sample image, and the network parameters of the first generator in the network to be trained are adjusted so that the loss of the converted sample image generated by the first generator meets a convergence condition, thereby obtaining the trained first generator.

[0137] In a second step, a loss of the first reconstructed sample image is determined based on a style feature of the first reconstructed sample image and a style feature of the second sample image.

[0138] In some embodiments, the style encoder to be trained is used to determine the style feature of the first reconstructed sample image and the style feature of the second sample image, respectively. The first reconstructed sample image is input into the style encoder to extract the style feature of the first reconstructed sample image, and the style encoder is used to extract the style feature of the second sample image in the same way. A difference between the two style features is taken as the loss of the first reconstructed sample image, and the network parameters of the second generator and the style encoder are adjusted using the loss of the first reconstructed sample image, so that the trained style encoder and the second generator can generate a reconstructed sample image that cannot be distinguished by the discriminator.

[0139] In a third step, an adversarial loss between the first reconstructed sample image and the second sample image is determined.

[0140] In some embodiments, the first reconstructed sample image and the second sample image are input into a discriminator of the network to be trained, respectively, to obtain an output of the first reconstructed sample image and an output of the second sample image; then, the output of the second sample image and the opposite of the output of the first reconstructed sample image are fused to obtain the adversarial loss, so that the discriminator cannot distinguish between the first reconstructed sample image and the second sample image.

[0141] In a fourth step, the network parameters of the network to be trained are adjusted based on the loss of the converted sample image, the loss of the first reconstructed sample image, and the adversarial loss, thereby obtaining the image conversion network.

[0142] In some possible implementation manners, the network parameters of the first generator are adjusted based on the loss of the converted sample image, the network parameters of the style encoder of the to-be-trained network are adjusted based on the loss of the first reconstructed sample image, and the network parameters of the discriminator of the to-be-trained network are adjusted based on the adversarial loss, to obtain the image conversion network. In this way, the discriminator, the generator, and the style encoder in the to-be-trained network are trained respectively, and the image conversion network with higher image conversion accuracy can be obtained.

[0143] In the following, an exemplary application of the embodiment of the present application in an actual application scenario will be described, for example, to implement image conversion between different fields based on generated prior.

[0144] The purpose of unsupervised image conversion is to convert images of one field to images of another field without paired image supervision, and has applications such as image editing, image artistic, virtual image reality rendering, etc. In related technologies, a scheme based on cycle consistency is adopted, which requires that the image converted from the original field to the target field can be converted back to the original field, so as to reconstruct the image. However, this scheme is only suitable for the case where the original field and the target field are highly similar, for example, the conversion between a horse and a zebra. For the conversion between fields with large morphological differences, for example, the conversion between any two of a human face, an animal face, and a cartoon face, the converted image often loses too much information to effectively reconstruct the original image.

[0145] Based on this, the embodiment of the present application provides an image conversion method, which proposes a new image conversion method based on generated prior to solve the problem that the image conversion method based on cycle consistency cannot handle the conversion between fields with large morphological differences, to establish robust mapping relationships between different fields and realize the conversion between fields with large morphological differences.

[0146] In the embodiment of the present application, the pre-trained conditional generative adversarial network provides prior information about the internal relationship between different fields, a content encoder is trained to extract high-level content features shared by fields, and further learns intermediate-level content features in specific tasks, so as to establish content mapping relationships at each level and realize natural and accurate image conversion.

[0147] The image conversion method provided by the embodiment of the present application can be implemented through the following two steps:

[0148] First, the pre-trained conditional generative adversarial network is distilled to generate prior information.

[0149] In some embodiments, the above first step can be implemented through the following steps:

[0150] Step 1.1, generate training data.

[0151] In some possible implementation manners, the training data includes the first sample image and the second sample image. First, a conditional generative adversarial network model pre-trained on a large-scale data set is selected, that is, a generative network that maps random noise z and a domain label l to a corresponding domain image. For the convenience of description, BigGAN pre-trained on the ImageNet data set is taken as an example, but the data set and the network model are not limited to this and belong to replaceable components. Then, different noises z1,..., zn are sampled, and the corresponding images are generated. n For the domain labels in the ImageNet data set, n images of the corresponding domain are generated, and k x n images are generated for k domains for subsequent training. As shown in Figure 3 As shown in the figure, for the domains of dogs, cats and birds, although they have different shapes, the images generated by the same noise (z1, z2 or z3) have a highly unified semantic correspondence; for example, the images 301 corresponding to the domain of dogs, the images 302 corresponding to the domain of cats and the images 303 corresponding to the domain of birds generated by the same noise z1 have unified orientation, posture and the like, and these rules provide effective generation priors that can be used to establish the relationship between different domains.

[0152] Step 1.2, training the content encoder based on the generated training data.

[0153] Training the content encoder E c Extracting high-level content features shared by the images generated by the same z. As shown in Figure 4 As shown in the figure, Figure 4 The network architecture schematic diagram provided by the embodiment of the application provides a distillation generation prior, and the content encoder E Figure 4 As can be seen, the network includes a content encoder E c 401 and a decoder F 403. In training, two training images are randomly sampled, including an image (x = cGAN(z, lx)) 41 and an image (y = cGAN(z, ly)) 42, that is, the image 41 and the image 42 are images generated according to the same noise. Wherein, cGAN is the conditional generative adversarial network in step 1.1, z is the shared noise, lx and ly are two domain labels respectively. The content encoder E c 401 extracts the content features of the image 41 to obtain the high-level content features 43 of the image 41; the content encoder E c 402 extracts the content features of the image 42 to obtain the high-level content features 44 of the image 42; the style encoder E s403 extract style feature sx of image 42; decoder F 404 reconstructs image 41 according to high-level content feature 43, style feature sx and domain label lx of image 41, to obtain image 45. In the embodiment of the present application, adaptive instance normalization is used to input the information of style feature sx and domain label lx into F, to obtain the reconstructed image F(cx, sx, lx), but the method is not limited to this. For example, the way of inputting the conditional information of style feature sx and domain label lx into F can also use the way of channel concatenation.

[0154] In the process of training, it is hoped that the reconstructed sample image restores the image x as much as possible, and the content feature extracted from the image y approximates the content feature of the training image x as much as possible, which correspond to the following L rec and L feat As shown in formulas (1) and (2):

[0155]

[0156] L feat = E (x,y) [||Ec(y)-Ec(x)||1] (2);

[0157] Wherein, E x represents the correlation function with respect to variable x, E (x,y) represents the correlation function with respect to variable (x, y); Ec(y), Ec(x) respectively represent the content features of image y and image x extracted by the content encoder. L rec The image similarity can also be determined by using the measurement methods such as perceptual loss and learned perceptual image patch similarity (LPIPS).

[0158] In some embodiments, L rec In addition to the reconstructed sample image itself, the foreground and background of the training image can also be segmented by using segmentation technology to obtain the foreground mask of image x, and the decoder F is used to reconstruct the foreground mask.

[0159] In the embodiment of the present application, L2 regularization constraint can also be added to the content feature to make it as sparse as possible.

[0160] Secondly, based on the generated prior information, the image is converted.

[0161] In some embodiments, the content encoder E c is fixed, and the image conversion network is trained to realize image conversion. As shown in Figure 5 Figure 5 ​The structural schematic diagram of the image conversion network provided by the embodiment of the present application, the image conversion network comprises a style encoder E s 505, the content encoder E trained in the first step, the generator 51, the generator 54, the discriminator D 53 c 503 and the content encoder E c 504; wherein the image 501 belongs to the field X, the image 502 belongs to the field Y, the content encoder E is used c 503 extracts the content features of the image 501, the content encoder E is used c 504 extracts the content features of the image 502, the style encoder E is used s 505 extracts the style features of the image 502. The content encoder E c 503 and the generator 51, the content encoder E is detected between the dynamic cross-layer connection c 503 extracts the multi-layer intermediate features of the image 501, and the style features of the image 502 are also input into the generator 51, and the image 506 converted from the field X to the field Y of the image 501 is generated; the content encoder E is used c 52 judges whether the image 506 is consistent with the high-level content features of the image 501; the discriminator D 53 is used to distinguish the true and false of the image 506 and the image 502. The content features of the image 502 and the style features of the image 502 are input into the generator 54, and the reconstruction of the image 502 is realized, and the reconstructed sample image 507 is obtained. In this way, the dynamic cross-layer connection is established between the content encoder E c and the generator G, so as to extract the content features of each layer and establish the content mapping relationship between each layer of images in different fields.

[0162] The above second step can be realized by the following steps:

[0163] Step 2.1, the dynamic cross-layer connection is established between the content encoder and the generator.

[0164] In some possible implementation manners, the effective part of the content features of the intermediate layer is extracted through the dynamic cross-layer connection, the extracted effective part, the content encoder E c The high-level content features extracted by the content encoder E c are input into the generator G, and the content mapping relationship of each layer is established. The dynamic cross-layer connection is established between the content encoder E c Each layer estimates a mask m, and the encoder features f E of the content encoder are combined by the mask m and the generator features f G of the generator to realize the introduction of the intermediate layer information.

[0165] The dynamic cross-layer connection comprises a hidden state h, an update network R, a mask network M, and a feature update network U. For convenience of description, superscript i is used to represent the generator feature and network corresponding to the i-th layer of the generator G. The operation of each layer first adjusts all features to a uniform spatial size using nearest neighbor interpolation. The i-th layer first uses the update network R i Update the hidden state h i As shown in equation (3):

[0166]

[0167] wherein, represents the i-th layer encoder feature, and the initial hidden state is set as the high-level content feature h 0 = Ec(x). Then, the mask information m i As shown in equation (4):

[0168]

[0169] Then, the encoder feature is updated As shown in equation (5):

[0170]

[0171] Finally, the encoder feature and the generator feature are fused to obtain the final output f of the generator G at the i-th layer i As shown in equation (6):

[0172]

[0173] Step 2.2, based on the intermediate layer information introduced through the dynamic cross-layer connection, training the image conversion network.

[0174] In some possible implementations, for a specific task from the image field X to the field Y, in the training process, the content encoder E c is fixed, and images x e X and y e Y are randomly sampled; the content encoder E c extracts content features cx and cy from the image x and the image y, respectively, and the E s extracts style features sx and sy from the image x and the image y, respectively; and the generator G generates a conversion image

[0175] The training requirement for the image conversion network is that the content feature of the image is consistent with the content feature cx of the image x, and the style feature is consistent with the style feature of the image y, so that the discriminant network cannot distinguish the conversion image The true and false correspond to L con , L sty , and L adv , as shown in formulas (7), (8), and (9):

[0176]

[0177]

[0178]

[0179] wherein F D is a style feature extracted by an intermediate layer of the discriminator D network, and is a vector composed of the mean value of each channel of the feature of the layer; that is, F D (y) respectively represent the style features of the converted image and the image y extracted by the intermediate layer of the discriminator D network.

[0180] In some embodiments, L sty may adopt other loss functions for measuring the consistency of the style, for example, a style loss based on a VGG network in a perceptual loss, a contextual loss, and the like.

[0181] In some embodiments, the generator G reconstructs in accordance with the content features cy and sy of the image y rec , and the consistency between the image y and the reconstructed sample image may be measured by L rec , or the image similarity is measured by using a perceptual loss, LPIPS, and the like. Specifically, L rec is:

[0182]

[0183] In some embodiments, L1 regularization constraint can be added to the mask information m to make it as sparse as possible, as shown in the following formula (11):

[0184]

[0185] In the embodiments of the present application, the distillation network for generating prior information can provide a highly abstract mapping relationship between images in different fields, thereby providing strong technical support for field conversion with large morphological differences; the content feature extraction of each layer can establish a robust mapping relationship between each layer in different fields; in this way, the overall framework can realize image conversion between fields with large morphological differences, while supporting generalization to fields other than the prior provided by the generation network. In this way, the image conversion method provided in the embodiments of the present application can realize artistic face swapping / head portrait generation, for example, by establishing a mapping relationship between a portrait and a cartoon / animal face, a user can generate various head portraits or replace a real face with an animal face; the image conversion method can also realize image intelligent editing: converting a specified object in an image into another object, for example, a user who is afraid of sharp-mouthed animals can replace a bird that he does not want to see in a photo with a pet dog / cat without affecting the layout of the photo; and data augmentation, for example, a large and complete data set of human faces with rich labels such as positioning points, angles, and light and shadow. The image conversion method provided in the embodiments of the present application can also convert a human face into an animal face and reuse common labels such as angles and light and shadow to realize data augmentation, provide labeled animal face data, and be used for deep learning and analysis.

[0186] The embodiments of the present application provide an image conversion device, Figure 6 The structure and composition of the image conversion device provided in the embodiments of the present application are shown in the schematic diagram as Figure 6 As shown in the schematic diagram, the image conversion device 600 comprises:

[0187] A first determination module 601 is configured to determine a first field corresponding to a first image by using an image conversion network.

[0188] A second determination module 602 is configured to determine a feature mapping relationship of the first image and a preset image at each layer of the image conversion network based on the first field and a second field corresponding to the preset image.

[0189] A first conversion module 603 is configured to convert a picture of the first image from the first field to the second field based on the feature mapping relationship to obtain a second image.

[0190] In some embodiments, the second determination module 602 comprises:

[0191] A first determination sub-module is configured to determine a feature category shared between the first image and the preset image at each layer of the image conversion network based on the first field and the second field.

[0192] a second determining submodule, configured to take each layer as a target layer, and determine first output features of the first image at the target layer and second output features of the preset image at the target layer respectively based on feature categories shared by the target layer;

[0193] a third determining submodule, configured to determine a feature mapping relationship of the first image and the preset image at the target layer based on the first output features and the second output features.

[0194] In some embodiments, the first determining submodule comprises:

[0195] a first determining unit, configured to determine abstract content feature categories shared by the first image and the preset image based on the first domain and the second domain;

[0196] a second determining unit, configured to determine concrete content feature categories shared by the first image and the preset image at each layer;

[0197] a third determining unit, configured to obtain feature categories shared by each layer based on the abstract content feature categories and the concrete content feature categories shared by each layer.

[0198] In some embodiments, the second determining submodule comprises:

[0199] a fourth determining unit, configured to determine, by using a content encoder of the image conversion network, encoder features of the feature categories shared by the target layer in the first image;

[0200] a fifth determining unit, configured to input the encoder features of the target layer into a generator of the image conversion network to determine generator features of the target layer;

[0201] a first fusing unit, configured to fuse the generator features of the target layer and the encoder features of the target layer to obtain the first output features of the first image at the target layer.

[0202] In some embodiments, the fourth determining unit comprises:

[0203] a first determining subunit, configured to determine, in the first image, abstract content features belonging to abstract content feature categories in the feature categories shared by the target layer and concrete content features belonging to concrete content feature categories in the feature categories shared by the target layer by using the content encoder;

[0204] a second determining subunit, configured to determine an initial hidden state based on the abstract content features;

[0205] The first updating sub-unit is configured to update the concrete content feature of the target layer based on the initial hidden state, to obtain an encoder feature of the target layer.

[0206] In some embodiments, the fifth determining unit comprises:

[0207] The third determining sub-unit is configured to determine the mask information of the first image in the target layer based on the hidden state of the target layer and the encoder feature of the target layer.

[0208] The first generating sub-unit is configured to input the concrete content feature of the target layer and the mask information of the target layer into the generator, to generate a generator feature of the target layer.

[0209] The first fusion unit comprises:

[0210] The first fusion sub-unit is configured to fuse the encoder feature of the target layer and the generator feature of the target layer based on the mask information of the target layer, to obtain the first output feature of the first image in the target layer.

[0211] In some embodiments, the third determining sub-unit is further configured to: in a case where the target layer is a first layer, determine the abstract content feature as the hidden state of the first layer; or in a case where the target layer is an i-th layer, determine the hidden state of the i-th layer based on the hidden state of an (i-1)-th layer and the concrete content feature of the i-th layer, where i is an integer greater than 1; and determine the mask information of the first image in a (j+1)-th layer based on the hidden state of the j-th layer and the concrete content feature of the (j+1)-th layer, where j is an integer greater than or equal to 1.

[0212] In some embodiments, the first generating sub-unit is further configured to: perform regularization processing on the mask information of the target layer, to obtain sparse mask information of the target layer; acquire a target content feature matching the sparse mask information of the target layer from the concrete content feature of the target layer; and input the target content feature of the target layer into the generator, to generate the generator feature of the target layer.

[0213] In some embodiments, the first fusion sub-unit is further configured to: adjust the encoder feature of the target layer based on the mask information of the target layer, to obtain a first adjusted feature of the target layer; adjust the generator feature of the target layer based on the adversarial information of the mask information of the target layer, to obtain a second adjusted feature of the target layer; and fuse the first adjusted feature of the target layer and the second adjusted feature of the target layer, to obtain the first output feature of the target layer.

[0214] In some embodiments, the first conversion module 603 further comprises:

[0215] A fourth determination sub-module is configured to determine, by using a style encoder in the image conversion network, style features of the preset image at each layer of the image conversion network;

[0216] A first merging sub-module is configured to take the style features of any layer of the image conversion network and the corresponding feature mapping relationship as a feature group.

[0217] A first input sub-module is configured to input the feature groups of the layers of the image conversion network into a generator of the image conversion network to obtain the second image.

[0218] In some embodiments, the image conversion network is obtained by training a to-be-trained network based on a first sample image in which a picture belongs to a first sample field and a second sample image in which a picture belongs to a second sample field, and the device further comprises a training module configured to train the to-be-trained network to obtain the image conversion network, wherein the training module comprises:

[0219] A first acquisition sub-module is configured to acquire the first sample image and the second sample image related to a picture.

[0220] A fifth determination sub-module is configured to determine, by using a trained content encoder in the to-be-trained network, first sample features of the first sample image at each layer of the trained content encoder and second sample features of the second sample image at each layer.

[0221] A sixth determination sub-module is configured to determine, based on the first sample features at each layer and the second sample features at each layer, a sample feature mapping relationship of the first sample image and the second sample image at each layer of the trained content encoder.

[0222] A second input sub-module is configured to input the sample feature mapping relationship at each layer and a sample style feature of the second sample image into a first generator of the to-be-trained network to obtain a conversion sample image in which a picture is converted from the first sample field to the second sample field.

[0223] A third input sub-module is configured to input the sample style feature of the second sample image and the second sample features at each layer into a second generator of the to-be-trained network to obtain a first reconstructed sample image in which a picture of the second sample image is reconstructed.

[0224] A first adjustment sub-module is configured to adjust network parameters of the to-be-trained network based on a loss of the conversion sample image and a loss of the first reconstructed sample image to obtain the image conversion network.

[0225] In some embodiments, the first obtaining sub-module comprises:

[0226] a first obtaining unit, configured to obtain a preset noise and a domain label set comprising at least a first sample domain and a second sample domain;

[0227] a first generating unit, configured to input the preset noise and the domain label set into a trained adversarial network to generate the first sample image and the second sample image with matched semantic information.

[0228] In some embodiments, the apparatus further comprises:

[0229] a second training module, configured to train a to-be-trained content encoder based on the first sample image and the second sample image to obtain the trained content encoder.

[0230] In some embodiments, the second training module comprises:

[0231] a seventh determining sub-module, configured to determine, by using the to-be-trained content encoder, a first predicted content feature of the first sample image and a second predicted content feature of the second sample image;

[0232] a first reconstructing sub-module, configured to input the first predicted content feature, a sample style feature of the first sample image and the first sample domain into a decoder to reconstruct the first sample image to obtain a second reconstructed sample image;

[0233] an eighth determining sub-module, configured to determine a first loss between the second reconstructed sample image and the first sample image;

[0234] a ninth determining sub-module, configured to determine a second loss between the second predicted content feature and the first predicted content feature;

[0235] a second adjusting sub-module, configured to adjust network parameters of the to-be-trained content encoder based on the first loss and the second loss to obtain the trained content encoder.

[0236] In some embodiments, the first adjusting sub-module comprises:

[0237] a sixth determining unit, configured to determine, based on a content feature of the converted sample image and a content feature of the first sample image, a loss of the converted sample image;

[0238] a seventh determining unit, configured to determine, based on a style feature of the first reconstructed sample image and a style feature of the second sample image, a loss of the first reconstructed sample image;

[0239] an eighth determining unit configured to determine an adversarial loss between the first reconstructed sample image and the second sample image;

[0240] a first adjusting unit configured to adjust the network parameter of the to-be-trained network based on the loss of the converted sample image, the loss of the first reconstructed sample image, and the adversarial loss, to obtain the image conversion network.

[0241] It should be noted that the above description of the device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0242] It should be noted that in the embodiments of the present application, if the image conversion method described above is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing an electronic device (which can be a terminal, a server, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a hard disk drive, a read-only memory (ROM), a magnetic disk or an optical disk, and various other media that can store program codes. Thus, the embodiments of the present application are not limited to any particular hardware and software combination.

[0243] Correspondingly, the embodiments of the present application further provide a computer program product, which includes computer executable instructions, and the computer executable instructions can implement the steps in the image conversion method provided by the embodiments of the present application after being executed.

[0244] Correspondingly, the embodiments of the present application further provide a computer storage medium, which stores computer executable instructions, and the computer executable instructions implement the steps of the image conversion method provided by the above embodiments when executed by a processor.

[0245] Correspondingly, the embodiments of the present application provide an electronic device, Figure 7 The composition structure of the electronic device provided by the embodiments of the present application is shown in the following Figure 7As shown, the electronic device 700 includes a processor 701, at least one communication bus, a communication interface 702, at least one external communication interface, and a memory 703. The communication interface 702 is configured to realize the connection communication between the components. The communication interface 702 can include a display screen, and the external communication interface can include a standard wired interface and a wireless interface. The processor 701 is configured to execute an image processing program in the memory to implement the steps of the image conversion method provided in the above embodiments.

[0246] The descriptions of the above image conversion device, electronic device, and storage medium embodiments are similar to the descriptions of the above method embodiments, have similar technical effects and advantages as the corresponding method embodiments, and are limited by the length of the article. The descriptions of the above method embodiments are not repeated here. For technical details not disclosed in the image conversion device, electronic device, and storage medium embodiments of the present application, please refer to the descriptions of the method embodiments of the present application.

[0247] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above sequence number of the embodiments of the present application is only for description, not representing the advantages and disadvantages of the embodiments. It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0248] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling, or direct coupling or communication connection between the components can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0249] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on multiple network units; and some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0250] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware, or in the form of hardware plus software functional units. Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and when the program is executed, the steps of the method embodiments are executed; and the foregoing storage medium includes mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and various media that can store program codes.

[0251] Alternatively, the above-mentioned integrated units of the present application, if realized in the form of software function modules and sold or used as independent products, can also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: mobile storage devices, ROM, magnetic disks or optical disks, and various media that can store program codes. The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image conversion method, characterized in that, The method includes: An image transformation network is used to determine the first region corresponding to the first image; Based on the same feature categories in the second domain corresponding to the first domain and the preset image, the feature mapping relationship between the first image and the preset image in each layer of the image conversion network is determined; the feature categories include: abstract content feature categories and concrete content feature categories; Based on the feature mapping relationship, the image of the first image is converted from the first region to the second region to obtain the second image.

2. The method according to claim 1, characterized in that, The step of determining the feature mapping relationship between the first image and the preset image at each layer of the image conversion network based on the same feature category in the second domain corresponding to the first domain and the preset image includes: Based on the first domain and the second domain, determine the feature categories shared between the first image and the preset image at each layer of the image conversion network; For each of these layers as the target layer, perform the following steps: Based on the feature categories shared by the target layer, the first output feature of the first image in the target layer and the second output feature of the preset image in the target layer are determined respectively. Based on the first output feature and the second output feature, the feature mapping relationship between the first image and the preset image in the target layer is determined.

3. The method according to claim 2, characterized in that, The step of determining the shared feature categories between the first image and the preset image at each layer of the image conversion network based on the first domain and the second domain includes: Based on the first domain and the second domain, determine the abstract content feature category shared by the first image and the preset image; Determine the concrete content feature category shared by the first image and the preset image in each layer; Based on the abstract content feature category and the concrete content feature category shared by each layer, the feature category shared by each layer is obtained.

4. The method according to claim 2 or 3, characterized in that, Determining the first output feature of the first image in the target layer based on the shared feature categories of the target layer includes: Using the content encoder of the image transformation network, the encoder features of the target layer shared feature categories in the first image are determined; The encoder features of the target layer are input into the generator of the image conversion network to determine the generator features of the target layer; The generator features and encoder features of the target layer are fused to obtain the first output features of the first image in the target layer.

5. The method according to claim 4, characterized in that, The content encoder using the image transformation network determines the encoder features of the target layer's shared feature categories in the first image, including: In the first image, the content encoder is used to determine the abstract content features belonging to the abstract content feature category in the shared feature category of the target layer and the concrete content features belonging to the concrete content feature category in the shared feature category of the target layer. Based on the aforementioned abstract content features, the initial hidden state is determined; Based on the initial hidden state, the concrete content features of the target layer are updated to obtain the encoder features of the target layer.

6. The method according to claim 4 or 5, characterized in that, The step of inputting the encoder features of the target layer into the generator of the image transformation network to determine the generator features of the target layer includes: Based on the hidden state of the target layer and the encoder features of the target layer, the mask information of the first image in the target layer is determined; The concrete content features of the target layer and the mask information of the target layer are input into the generator to generate the generator features of the target layer; The step of fusing the generator features and encoder features of the target layer to obtain the first output feature of the first image in the target layer includes: Based on the mask information of the target layer, the encoder features and generator features of the target layer are fused to obtain the first output feature of the first image in the target layer.

7. The method according to claim 6, characterized in that, Determining the mask information of the first image in the target layer based on the hidden state of the target layer and the encoder features of the target layer includes: When the target layer is the first layer, the abstract content feature is determined as the hidden state of the first layer; or, when the target layer is the i-th layer, the hidden state of the i-th layer is determined based on the hidden state of the (i-1)-th layer and the concrete content feature of the i-th layer, where i is an integer greater than 1. Based on the hidden state of layer j and the concrete content features of layer (j+1), the mask information of the first image in layer (j+1) is determined, where j is an integer greater than or equal to 1.

8. The method according to claim 6 or 7, characterized in that, The step of inputting the concrete content features of the target layer and the mask information of the target layer into the generator to generate the generator features of the target layer includes: The mask information of the target layer is regularized to obtain the sparse mask information of the target layer; From the concrete content features of the target layer, obtain the target content features that match the sparse mask information of the target layer; The target content features of the target layer are input into the generator to generate the generator features of the target layer.

9. The method according to any one of claims 6 to 8, characterized in that, The process of fusing the encoder features and generator features of the target layer based on the mask information of the target layer to obtain the first output feature of the first image in the target layer includes: The encoder features of the target layer are adjusted based on the mask information of the target layer to obtain the first adjusted features of the target layer; Based on the adversarial information of the mask information of the target layer, the generator features of the target layer are adjusted to obtain the second adjusted features of the target layer; The first adjustment feature and the second adjustment feature of the target layer are fused to obtain the first output feature of the target layer.

10. The method according to any one of claims 1 to 9, characterized in that, The step of converting the image of the first image from the first region to the second region based on the feature mapping relationship to obtain the second image further includes: The style encoder in the image conversion network is used to determine the style features of the preset image at each layer of the image conversion network. The style features of any layer in the image conversion network and the corresponding feature mapping relationship are used as a feature group; The feature sets of each layer in the image conversion network are input into the generator of the image conversion network to obtain the second image.

11. The method according to any one of claims 1 to 10, characterized in that, The image conversion network is obtained by training a network to be trained based on a first sample image belonging to a first sample region and a second sample image belonging to a second sample region. The training process of the network to be trained includes: Acquire the first sample image and the second sample image related to the scene; Using the trained content encoder in the network to be trained, determine the first sample feature of the first sample image in each layer of the trained content encoder and the second sample feature of the second sample image in each layer; Based on the first sample features and the second sample features of each layer, the sample feature mapping relationship between the first sample image and the second sample image in each layer of the trained content encoder is determined; The sample feature mapping relationship of each layer and the sample style features of the second sample image are input into the first generator of the network to be trained to obtain a transformed sample image from the first sample domain to the second sample domain; The sample style features of the second sample image and the second sample features of each layer are input into the second generator of the network to be trained to obtain the first reconstructed sample image for reconstructing the image of the second sample image. Based on the loss of the transformed sample image and the loss of the first reconstructed sample image, the network parameters of the network to be trained are adjusted to obtain the image transformation network.

12. The method according to claim 11, characterized in that, The acquisition of the first sample image and the second sample image related to the image includes: Obtain a preset noise and a set of domain labels that includes at least the first and second sample domains; The preset noise and the domain label set are input into the trained adversarial network to generate the first sample image and the second sample image with matching semantic information.

13. The method according to claim 11 or 12, characterized in that, The method further includes using a trained content encoder in the network to be trained to determine the first sample features of the first sample image at each layer of the network to be trained and the second sample image before the second sample features at each layer of the network to be trained. The trained content encoder is obtained by training the content encoder based on the first sample image and the second sample image.

14. The method according to claim 13, characterized in that, The step of training the content encoder to be trained based on the first sample image and the second sample image to obtain the trained content encoder includes: The first predicted content feature of the first sample image and the second predicted content feature of the second sample image are determined using the content encoder to be trained. The first predicted content feature, the sample style feature of the first sample image, and the first sample domain are input into the decoder to reconstruct the first sample image and obtain the second reconstructed sample image. Determine the first loss between the second reconstructed sample image and the first sample image; Determine a second loss between the second predicted content feature and the first predicted content feature; Based on the first loss and the second loss, the network parameters of the content encoder to be trained are adjusted to obtain the trained content encoder.

15. The method according to any one of claims 11 to 14, characterized in that, The process of adjusting the network parameters of the network to be trained based on the loss of the transformed sample image and the loss of the first reconstructed sample image to obtain the image transformation network includes: Based on the content features of the transformed sample image and the content features of the first sample image, the loss of the transformed sample image is determined; Based on the style features of the first reconstructed sample image and the style features of the second sample image, the loss of the first reconstructed sample image is determined. Determine the adversarial loss between the first reconstructed sample image and the second sample image; Based on the loss of the transformed sample image, the loss of the first reconstructed sample image, and the adversarial loss, the network parameters of the network to be trained are adjusted to obtain the image transformation network.

16. An image conversion apparatus, characterized in that, The device includes: The first determining module is used to determine the first region corresponding to the first image using an image conversion network; The second determining module is used to determine the feature mapping relationship between the first image and the preset image at each layer of the image conversion network based on the same feature categories in the second domain corresponding to the first domain and the preset image; the feature categories include: abstract content feature categories and concrete content feature categories; The first conversion module is used to convert the image of the first image from the first domain to the second domain based on the feature mapping relationship, so as to obtain the second image.

17. The apparatus according to claim 16, characterized in that, The second determining module includes: The first determining submodule is used to determine, based on the first domain and the second domain, the feature categories shared between the first image and the preset image at each layer of the image conversion network; The second determining submodule is used to take each layer as a target layer and, based on the feature categories shared by the target layers, determine the first output feature of the first image in the target layer and the second output feature of the preset image in the target layer, respectively. The third determining submodule is used to determine the feature mapping relationship between the first image and the preset image in the target layer based on the first output feature and the second output feature.

18. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which, when executed, enable an image conversion method, wherein the method includes: An image transformation network is used to determine the first region corresponding to the first image; Based on the same feature categories in the second domain corresponding to the first domain and the preset image, the feature mapping relationship between the first image and the preset image in each layer of the image conversion network is determined; the feature categories include: abstract content feature categories and concrete content feature categories; Based on the feature mapping relationship, the image of the first image is converted from the first region to the second region to obtain the second image.

19. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores computer-executable instructions, and when the processor executes the computer-executable instructions in the memory, it can implement an image conversion method, wherein the method includes: An image transformation network is used to determine the first region corresponding to the first image; Based on the same feature categories in the second domain corresponding to the first domain and the preset image, the feature mapping relationship between the first image and the preset image in each layer of the image conversion network is determined; the feature categories include: abstract content feature categories and concrete content feature categories; Based on the feature mapping relationship, the image of the first image is converted from the first region to the second region to obtain the second image.

20. A computer program product comprising computer-readable code, wherein when the computer-readable code is executed in an electronic device, a processor in the electronic device executes a method for implementing an image conversion method, wherein... The method includes: An image transformation network is used to determine the first region corresponding to the first image; Based on the same feature categories in the second domain corresponding to the first domain and the preset image, the feature mapping relationship between the first image and the preset image in each layer of the image conversion network is determined; the feature categories include: abstract content feature categories and concrete content feature categories; Based on the feature mapping relationship, the image of the first image is converted from the first region to the second region to obtain the second image.

Images