Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and storage medium, which belongs to the field of computer technology. The method includes: extracting features from a first image and a second image to obtain multiple first image features and multiple second image features; determining the similarity between the first image features and the second image features corresponding to the same channel position to obtain multiple feature similarities; determining a first target image and a second target image based on the multiple first image features, the multiple second image features and the multiple feature similarities. The above scheme can avoid the influence of the imbalance of information richness between images on style conversion, so that the result obtained by conversion is more reasonable and the quality of the converted image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] The conversion from graphics to images can be applied to many computer vision tasks, such as image segmentation, image modification, image style transfer, etc. For example, based on the generative adversarial network in deep learning, the style of the image in the target domain (composed of a group of images with similar styles) can be converted to the style of the source domain (composed of another group of images with similar styles), while keeping the image content of the source domain unchanged, such as converting the real face in the photographic image to a cartoon face, while keeping the content in the photographic image unchanged. Therefore, how to improve the quality of the image obtained by style transfer is an important research direction.

[0003] At present, the UGATIT (Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation) method uses class activation mapping to focus the network on areas where the gap between the source domain and the target domain is large, and then uses the instance normalization module in the adaptive layer to complete the style transfer task with large deformation and large span.

[0004] In the above technical solution, since the source domain and the target domain usually have different information richness, the conversion result will be unreasonable and the image quality will be poor. Summary of the invention

[0005] The present disclosure provides an image processing method, device, electronic device and storage medium, which can avoid the influence of the imbalance of information richness between images on style conversion, make the conversion result more reasonable, and improve the quality of the converted image. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0007] Extracting features from the first image and the second image to obtain a plurality of first image features and a plurality of second image features, wherein the plurality of first image features represent features of the first image, the plurality of second image features represent features of the second image, and the plurality of first image features correspond to the plurality of second image features in one-to-one correspondence;

[0008] Determine a similarity between a first image feature and a second image feature corresponding to a same channel position to obtain a plurality of feature similarities;

[0009] Based on the multiple first image features, the multiple second image features and the multiple feature similarities, a first target image and a second target image are determined, the first target image and the second image have the same style, and the second target image and the first image have the same style.

[0010] In some embodiments, determining the similarity between the first image feature and the second image feature corresponding to the same channel position to obtain multiple feature similarities includes:

[0011] Determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity between the first image feature and the second image feature; or,

[0012] The cross entropy between the first image feature and the second image feature corresponding to the same channel position is determined as the feature similarity between the first image feature and the second image feature.

[0013] In some embodiments, extracting features from the first image and the second image to obtain a plurality of first image features and a plurality of second image features includes:

[0014] Encoding the first image to obtain a first encoded image;

[0015] Performing feature extraction on the first encoded image to obtain the plurality of first image features;

[0016] encoding the second image to obtain a second encoded image;

[0017] Perform feature extraction on the second encoded image to obtain the plurality of second image features.

[0018] In some embodiments, encoding the first image to obtain a first encoded image includes:

[0019] The first image is downsampled, and the sampled image is input into a first residual network to obtain the first encoded image.

[0020] In some embodiments, extracting features from the first encoded image to obtain the plurality of first image features includes:

[0021] Based on the embedding function, extracting features from the first encoded image to obtain a plurality of first embedded features;

[0022] Average pooling is performed on the multiple first embedded features respectively to obtain the multiple first image features.

[0023] In some embodiments, the method further comprises:

[0024] Based on the projection function, the channel relationships between the plurality of first image features are rearranged.

[0025] In some embodiments, encoding the second image to obtain a second encoded image includes:

[0026] Down-sample the second image, and input the sampled image into a second residual network to obtain the second encoded image.

[0027] In some embodiments, extracting features from the second encoded image to obtain the plurality of second image features includes:

[0028] Based on the embedding function, extracting features from the second encoded image to obtain a plurality of second embedded features;

[0029] Average pooling is performed on the multiple second embedded features respectively to obtain the multiple second image features.

[0030] In some embodiments, the method further comprises:

[0031] Based on the projection function, the channel relationships between the plurality of second image features are rearranged.

[0032] In some embodiments, determining the first target image and the second target image based on the plurality of first image features, the plurality of second image features and the plurality of feature similarities comprises:

[0033] Determining a plurality of first target features based on the plurality of first image features and the plurality of feature similarities;

[0034] Decoding the plurality of first target features to obtain the first target image;

[0035] Determining a plurality of second target features based on the plurality of second image features and the plurality of feature similarities;

[0036] The plurality of second target features are decoded to obtain the second target image.

[0037] In some embodiments, determining a plurality of first target features based on the plurality of first image features and the plurality of feature similarities comprises:

[0038] For any first image feature, the product of the first image feature and the corresponding feature similarity is determined as the first target feature corresponding to the first image feature.

[0039] In some embodiments, decoding the plurality of first target features to obtain the first target image includes:

[0040] The multiple first target features are input into a third residual network, and a result output by the third residual network is upsampled to obtain the first target image.

[0041] In some embodiments, determining a plurality of second target features based on the plurality of second image features and the plurality of feature similarities comprises:

[0042] For any second image feature, the product of the second image feature and the corresponding feature similarity is determined as the second target feature corresponding to the second image feature.

[0043] In some embodiments, decoding the plurality of second target features to obtain the second target image includes:

[0044] The multiple second target features are input into a fourth residual network, and the result output by the fourth residual network is upsampled to obtain the second target image.

[0045] According to a second aspect of an embodiment of the present disclosure, a training method for an image processing model is provided, wherein the image processing model includes a generator and a discriminator, and the method includes:

[0046] Inputting the first sample image and the second sample image into the encoder and the aligned forgetting layer in the generator to obtain a plurality of sample image features, wherein the aligned forgetting layer is used to extract image features;

[0047] Inputting the plurality of sample image features into a decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image and the second sample image have the same style, and the second sample target image and the first sample image have the same style;

[0048] Inputting the first sample image, the second sample image, the first sample target image, and the second sample target image into the discriminator to obtain a training loss;

[0049] Model training is performed according to the training loss.

[0050] In some embodiments, inputting the first sample image, the second sample image, the first sample target image, and the second sample target image into the discriminator to obtain a training loss includes:

[0051] Input the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain a first loss, a second loss and a third loss, wherein the first loss includes a first adversarial loss and a second adversarial loss, the first adversarial loss represents the loss of converting the first sample image into the second sample target image, the second adversarial loss represents the loss of converting the second sample image into the first sample target image, the second loss includes a first image loss and a second image loss, the first image loss represents the difference between the first sample image and the first sample target image, the second image loss represents the difference between the second sample image and the second sample target image, the third loss includes a first consistency loss and a second consistency loss, the first consistency loss represents the difference between the first sample image and the second sample target image, and the second consistency loss represents the difference between the second sample image and the first sample target image;

[0052] The first loss, the second loss and the third loss are weightedly summed to obtain the training loss.

[0053] According to a third aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0054] a feature extraction module configured to perform feature extraction on the first image and the second image to obtain a plurality of first image features and a plurality of second image features, wherein the plurality of first image features represent features of the first image, the plurality of second image features represent features of the second image, and the plurality of first image features and the plurality of second image features correspond to each other one by one;

[0055] a similarity determination module configured to determine a similarity between a first image feature and a second image feature corresponding to a same channel position, and obtain a plurality of feature similarities;

[0056] The image determination module is configured to determine a first target image and a second target image based on the multiple first image features, the multiple second image features and the multiple feature similarities, wherein the first target image has the same style as the second image, and the second target image has the same style as the first image.

[0057] In some embodiments, the similarity determination module is configured to determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity of the first image feature and the second image feature; or, to determine the cross entropy of the first image feature and the second image feature corresponding to the same channel position as the feature similarity of the first image feature and the second image feature.

[0058] In some embodiments, the feature extraction module includes:

[0059] A first encoding unit, configured to encode the first image to obtain a first encoded image;

[0060] A first feature extraction unit is configured to perform feature extraction on the first encoded image to obtain the plurality of first image features;

[0061] A second encoding unit is configured to encode the second image to obtain a second encoded image;

[0062] The second feature extraction unit is configured to perform feature extraction on the second encoded image to obtain the plurality of second image features.

[0063] In some embodiments, the first encoding unit is configured to downsample the first image, input the sampled image into a first residual network, and obtain the first encoded image.

[0064] In some embodiments, the first feature extraction unit is configured to perform feature extraction on the first encoded image based on an embedding function to obtain a plurality of first embedded features; and perform average pooling on the plurality of first embedded features to obtain the plurality of first image features.

[0065] In some embodiments, the apparatus further comprises:

[0066] The first rearrangement module is configured to rearrange the channel relationship between the plurality of first image features based on the projection function.

[0067] In some embodiments, the second encoding unit is configured to downsample the second image, input the sampled image into a second residual network, and obtain the second encoded image.

[0068] In some embodiments, the second feature extraction unit is configured to perform feature extraction on the second encoded image based on an embedding function to obtain a plurality of second embedded features; and perform average pooling on the plurality of second embedded features to obtain the plurality of second image features.

[0069] In some embodiments, the apparatus further comprises:

[0070] The second rearrangement module is configured to rearrange the channel relationship between the plurality of second image features based on the projection function.

[0071] In some embodiments, the image determination module includes:

[0072] A first determining unit is configured to determine a plurality of first target features based on the plurality of first image features and the plurality of feature similarities;

[0073] A first decoding unit is configured to decode the plurality of first target features to obtain the first target image;

[0074] A second determining unit is configured to determine a plurality of second target features based on the plurality of second image features and the plurality of feature similarities;

[0075] The second decoding unit is configured to decode the plurality of second target features to obtain the second target image.

[0076] In some embodiments, the first determination unit is configured to determine, for any first image feature, a product of the first image feature and a corresponding feature similarity as a first target feature corresponding to the first image feature.

[0077] In some embodiments, the first decoding unit is configured to input the multiple first target features into a third residual network, upsample the result output by the third residual network, and obtain the first target image.

[0078] In some embodiments, the second determination unit is configured to determine, for any second image feature, a product of the second image feature and a corresponding feature similarity as the second target feature corresponding to the second image feature.

[0079] In some embodiments, the second decoding unit is configured to input the second target features into a fourth residual network, upsample the result output by the fourth residual network, and obtain the second target image.

[0080] According to a fourth aspect of an embodiment of the present disclosure, a training device for a graphics processing model is provided, wherein the graphics processing model includes a generator and a discriminator, and the device includes:

[0081] A first training module is configured to extract features of the first sample image and the second sample image based on the encoder and the aligned forgetting layer in the generator, wherein the aligned forgetting layer is used to extract image features;

[0082] A second training module is configured to decode the sample image features and similarities based on the decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image has the same style as the second sample image, and the second sample target image has the same style as the first sample image;

[0083] A third training module is configured to determine a training loss based on the discriminator, the first sample image, the second sample image, the first sample target image, and the second sample target image;

[0084] The third training module is further configured to perform model training based on the training loss.

[0085] In some embodiments, the fourth training module is configured to determine a first loss based on the discriminator, the first sample image, the second sample image, the first sample target image and the second sample target image, the first loss including a first adversarial loss and a second adversarial loss, the first adversarial loss representing the loss of converting the first sample image into the second sample target image, and the second adversarial loss representing the loss of converting the second sample image into the first sample target image; determine a second loss based on the first sample image, the second sample image, the first sample target image and the second sample target image, the second loss including a first image loss and a second image loss, the first image loss representing the difference between the first sample image and the first sample target image, and the second image loss representing the difference between the second sample image and the second sample target image; determine a third loss based on the first sample image, the second sample image, the first sample target image and the second sample target image, the third loss including a first consistency loss and a second consistency loss, the first consistency loss representing the difference between the first sample image and the second sample target image, and the second consistency loss representing the difference between the second sample image and the first sample target image.

[0086] According to a fifth aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0087] one or more processors;

[0088] A memory for storing program codes executable by the processor;

[0089] Among them, the processor is configured to execute the program code to implement the above-mentioned image processing method, or to implement the above-mentioned image processing model training method.

[0090] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned image processing method, or the electronic device is enabled to perform the above-mentioned image processing model training method.

[0091] According to the seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the above-mentioned image processing method when executed by a processor, or implements the above-mentioned image processing model training method when executed by the processor.

[0092] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0093] The disclosed embodiment provides an image processing method, which can reduce the influence of redundant and inconsistent features in the two images based on the multiple feature similarities by respectively determining multiple feature similarities between multiple first image features of a first image and multiple second image features of a second image, and retain the common features in the two images, thereby avoiding the influence of unbalanced information richness between images on style conversion, making the conversion result more reasonable and improving the quality of the converted image.

[0094] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0096] Figure 1 It is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment.

[0097] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0098] Figure 3 is a flowchart of another image processing method according to an exemplary embodiment.

[0099] Figure 4 The figure is a flowchart of another method for training an image processing model according to an exemplary embodiment.

[0100] Figure 5 It is a schematic diagram of the structure of a generator according to an exemplary embodiment.

[0101] Figure 6 It is a schematic diagram of the structure of an aligned forgetting layer according to an exemplary embodiment.

[0102] Figure 7 is a schematic diagram of a conversion result according to an exemplary embodiment.

[0103] Figure 8 It is a schematic diagram showing a style conversion result of a map and a satellite map according to an exemplary embodiment.

[0104] Fig. 9 It is a schematic diagram showing a style conversion result of a street scene and a segmented street scene according to an exemplary embodiment.

[0105] Fig.10 The figure is a schematic diagram showing a style conversion result of a building and a segmented building according to an exemplary embodiment.

[0106] Fig.11 is a comparison diagram of experimental results of a variant shown according to an exemplary embodiment.

[0107] Fig.12 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0108] Fig.13 is a block diagram of another image processing device according to an exemplary embodiment.

[0109] Fig.14 It is a block diagram of a training device for an image processing model according to an exemplary embodiment.

[0110] Fig.15 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0111] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0112] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims. The multiple in the embodiments of the present disclosure means two or more.

[0113] The terms "at least one", "a plurality of", "each", and "any" used in the present disclosure include one, two, or more than two, a plurality of includes two or more than two, each refers to each of the corresponding plurality, and any refers to any one of the plurality. For example, the plurality of first image features include three first image features, and each refers to each of the three first image features, and any refers to any one of the three first image features, which can be the first, the second, or the third.

[0114] It should be noted that the user information involved in this disclosure (including but not limited to user device information, user personal information, etc.) is all information authorized by the user or fully authorized by all parties.

[0115] The following is an explanation of terms related to the embodiments of the present disclosure.

[0116] ReLU (Rectified Linear Unit), also known as the rectified linear unit, is an activation function commonly used in artificial neural networks, usually referring to nonlinear functions represented by ramp functions and their variants.

[0117] Leaky ReLU (Leaky Linear Rectification Function) is a variation of the linear rectifier function based on the ramp function. The gradient of Leaky ReLU is a constant, not 0. When the input value is positive, the leaky linear rectifier function is consistent with the ordinary ramp function.

[0118] The image processing method provided in the embodiments of the present disclosure is executed by an electronic device. In some embodiments, the electronic device is a terminal, for example, the terminal is a mobile phone, a tablet computer, a computer, and other types of terminals. In some embodiments, the electronic device is a server, for example, the server is a server, or a server cluster composed of several servers, or a cloud computing service center.

[0119] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment. Taking the electronic device as a server as an example, see Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 and the server 102 can interact via a network connection.

[0120] The terminal 101 may be at least one of a smart phone, a desktop computer, a laptop computer, and the like. An application may be installed and run on the terminal 101, and a user may log in to the application through the terminal 101 to obtain services provided by the application. The terminal 101 may be connected to the server 102 through a wireless network or a wired network, and the first image and the second image to be processed may be sent to the server 102.

[0121] In some embodiments, terminal 101 may generally refer to one of multiple terminals. This embodiment is only illustrated by terminal 101. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be only a few, or the above terminals may be dozens or hundreds, or more. The embodiments of the present disclosure do not limit the number of terminals and device types.

[0122] The server 102 may be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 may be connected to the terminal 101 and other terminals via a wireless network or a wired network, and the server 102 may receive the first image and the second image sent by the terminal 101. In some embodiments, the number of the above servers may be more or less, and the embodiments of the present disclosure are not limited thereto. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0123] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the image processing method is executed by a server device and includes the following steps:

[0124] In step S201, feature extraction is performed on the first image and the second image to obtain a plurality of first image features and a plurality of second image features, wherein the plurality of first image features represent features of the first image, and the plurality of second image features represent features of the second image, and the plurality of first image features and the plurality of second image features correspond one to one.

[0125] The first image is an image to be converted into a target style, and the second image is an image with the target style; or the second image is an image to be converted into a target style, and the first image is an image with the target style.

[0126] In step S202, the similarity between the first image feature and the second image feature corresponding to the same channel position is determined to obtain a plurality of feature similarities.

[0127] Among them, for the first image feature and the second image feature corresponding to the same channel position, the feature similarity between the first image feature and the second image feature can be determined, and the feature similarity can be used to determine whether the corresponding image features are common features.

[0128] In step S203, based on the plurality of first image features, the plurality of second image features and the plurality of feature similarities, a first target image and a second target image are determined, the first target image having the same style as the second image, and the second target image having the same style as the first image.

[0129] The first target image is obtained by converting the first image, that is, the first target image is obtained after the first image is converted to have the same style as the second image. The second target image is obtained by converting the second image, that is, the second target image is obtained after the second image is converted to have the same style as the first image.

[0130] The solution provided by the embodiment of the present disclosure can reduce the influence of redundant and inconsistent features in the two images based on the multiple feature similarities by respectively determining multiple feature similarities between multiple first image features of the first image and multiple second image features of the second image, and retain the common features in the two images, thereby avoiding the influence of style conversion caused by the imbalance of information richness between images, making the conversion result more reasonable and improving the quality of the converted image.

[0131] In some embodiments, the determining of the similarity between the first image feature and the second image feature corresponding to the same channel position to obtain multiple feature similarities includes:

[0132] Determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity between the first image feature and the second image feature; or,

[0133] The cross entropy between the first image feature and the second image feature corresponding to the same channel position is determined as the feature similarity between the first image feature and the second image feature.

[0134] In the embodiment of the present disclosure, by calculating the dot product or cross entropy of the first image feature and the second image feature, the feature similarity between the two image features from different images but corresponding to the same channel position can be determined, and then based on the feature similarity, it can be determined whether the image features at the channel position are consistent.

[0135] In some embodiments, the feature extraction is performed on the first image and the second image to obtain a plurality of first image features and a plurality of second image features, including:

[0136] Encoding the first image to obtain a first encoded image;

[0137] Performing feature extraction on the first encoded image to obtain the plurality of first image features;

[0138] encoding the second image to obtain a second encoded image;

[0139] Feature extraction is performed on the second encoded image to obtain the plurality of second image features.

[0140] In the disclosed embodiment, by encoding the first image and the second image respectively, features can be extracted from the encoded images to obtain image features corresponding to each channel position, that is, channel-level features are extracted.

[0141] In some embodiments, encoding the first image to obtain a first encoded image includes:

[0142] The first image is downsampled, and the sampled image is input into a first residual network to obtain the first encoded image.

[0143] In the disclosed embodiment, by downsampling the first image and then performing residual processing, useful information can be extracted from the image to obtain a coded image, thereby facilitating subsequent feature extraction from the coded image.

[0144] In some embodiments, the performing feature extraction on the first encoded image to obtain the plurality of first image features includes:

[0145] Based on the embedding function, feature extraction is performed on the first encoded image to obtain a plurality of first embedded features;

[0146] Average pooling is performed on the multiple first embedded features respectively to obtain the multiple first image features.

[0147] In the disclosed embodiment, by performing feature extraction and average pooling on the encoded image based on the embedding function, the global information of each channel position in the encoded image can be extracted.

[0148] In some embodiments, the method further comprises:

[0149] Based on the projection function, the channel relationships between the plurality of first image features are rearranged.

[0150] In the disclosed embodiment, since the relationship between image features may be non-aligned and non-linear, by providing a projection function, the channel relationship between image features can be rearranged so that each image feature is aligned.

[0151] In some embodiments, encoding the second image separately to obtain a second encoded image includes:

[0152] The second image is downsampled, and the sampled image is input into a second residual network to obtain the second encoded image.

[0153] In the disclosed embodiment, by downsampling the second image and then performing residual processing, useful information can be extracted from the image to obtain a coded image, thereby facilitating subsequent feature extraction from the coded image.

[0154] In some embodiments, the extracting features from the plurality of channels of the second encoded image to obtain the plurality of second image features comprises:

[0155] Based on the embedding function, feature extraction is performed on the second encoded image to obtain a plurality of second embedded features;

[0156] Average pooling is performed on the multiple second embedded features to obtain the multiple second image features.

[0157] In the disclosed embodiment, by performing feature extraction and average pooling on the encoded image based on the embedding function, the global information of each channel position in the encoded image can be extracted.

[0158] In some embodiments, the method further comprises:

[0159] Based on the projection function, the channel relationships between the plurality of second image features are rearranged.

[0160] In the disclosed embodiment, since the relationship between image features may be non-aligned and non-linear, by providing a projection function, the channel relationship between image features can be rearranged so that each image feature is aligned.

[0161] In some embodiments, determining the first target image and the second target image based on the plurality of first image features, the plurality of second image features, and the plurality of feature similarities includes:

[0162] Determining a plurality of first target features based on the plurality of first image features and the plurality of feature similarities;

[0163] Decoding the plurality of first target features to obtain the first target image;

[0164] Determining a plurality of second target features based on the plurality of second image features and the plurality of feature similarities;

[0165] The plurality of second target features are decoded to obtain the second target image.

[0166] In the disclosed embodiment, the target features are determined based on image features and feature similarities, so that the target image after style conversion can be obtained by decoding the target features. Since the influence of image features with low similarity is reduced, the quality of the converted target image can be improved.

[0167] In some embodiments, the determining of a plurality of first target features based on the plurality of first image features and the plurality of feature similarities comprises:

[0168] For any first image feature, the product of the first image feature and the corresponding feature similarity is determined as the first target feature corresponding to the first image feature.

[0169] In the disclosed embodiment, the product of the image feature and the corresponding feature similarity is used as the corresponding target feature, so that based on the target feature, a target image with a corresponding style can be decoded.

[0170] In some embodiments, decoding the plurality of first target features to obtain the first target image includes:

[0171] The multiple first target features are input into a third residual network, and the result output by the third residual network is upsampled to obtain the first target image.

[0172] In the disclosed embodiment, residual processing is performed on the first target feature and then up-sampling is performed to decode the target feature, thereby obtaining the first target image.

[0173] In some embodiments, the determining of a plurality of second target features based on the plurality of second image features and the plurality of feature similarities comprises:

[0174] For any second image feature, the product of the second image feature and the corresponding feature similarity is determined as the second target feature corresponding to the second image feature.

[0175] In the disclosed embodiment, the product of the image feature and the corresponding feature similarity is used as the corresponding target feature, so that based on the target feature, a target image with a corresponding style can be decoded.

[0176] In some embodiments, decoding the plurality of second target features to obtain the second target image includes:

[0177] The multiple second target features are input into a fourth residual network, and the result output by the fourth residual network is upsampled to obtain the second target image.

[0178] In the disclosed embodiment, residual processing is performed on the second target feature and then up-sampling is performed to decode the target feature, thereby obtaining the second target image.

[0179] Above Figure 2 The following is a basic flow chart of the image processing method provided by the embodiment of the present disclosure. The solution provided by the present disclosure is further described below based on a specific implementation method. Figure 3 FIG. 1 is a flowchart of another image processing method according to an exemplary embodiment. Assume that the electronic device is provided as a server and the image processing method is executed by the server as an example. Figure 3 , the method comprising:

[0180] In step S301, a first image is encoded to obtain a first encoded image.

[0181] The first image is an image to be processed, that is, an image to be style-converted. For ease of description, the style of the first image is represented as a first style. The server can encode the first image based on a first encoder to obtain a first encoded image.

[0182] In some embodiments, the first encoder includes a first downsampling block and a first residual network, the first residual network includes at least one residual block, and the server can downsample the first image based on the first downsampling block, and then input the sampled image into the first residual network, and the first residual network processes the sampled image to obtain the first encoded image. The first downsampling block includes at least one convolutional layer for extracting useful information from the first image, that is, extracting features from the first image. It should be noted that each of the above convolutional layers and residual networks uses instance normalization for normalization, and the activation function uses the ReLU function.

[0183] For example, the first encoder includes three convolutional layers and three residual blocks in series. The convolution kernel of the first convolutional layer is 7×7 with a stride of 1; the convolution kernel of the second convolutional layer is 3×3 with a stride of 2; the convolution kernel of the third convolutional layer is 3×3 with a stride of 2. The series connection means that the output of the first convolutional layer is the input of the second convolutional layer, the output of the second convolutional layer is the input of the third convolutional layer, the output of the third convolutional layer is the input of the first residual block, the output of the first residual block is the input of the second residual block, and the output of the second residual block is the input of the third residual block.

[0184] In step S302, feature extraction is performed on the first encoded image to obtain a plurality of first image features, where the plurality of first image features represent features of the first image.

[0185] The encoded first encoded image includes multiple channels, and the server can extract features from each channel respectively to obtain the first image features corresponding to each channel.

[0186] In some embodiments, the server can map the information in the encoded image to a semantically consistent latent space based on an embedding function, where the embedding function is constructed by two residual blocks with shared weights. The server can then perform feature extraction on multiple channels of the first encoded image based on the embedding function to obtain multiple first embedded features, and then perform average pooling on the multiple first embedded features in the channel direction to extract global information of the first embedded features at the channel level to obtain the multiple first image features.

[0187] The embedding function is denoted as q(·), and the first embedding feature is denoted as z x =q(E X (x)), zx ∈R c×w×h , where x represents the first image, c represents the number of channels of the encoded image, c is a positive integer, and w and h represent the width and length of the first encoded image. The first image feature is represented by e x , the calculation method of the first image feature is shown in formula (1):

[0188]

[0189] in, represents the first image feature of the c-th channel, avg_pooling(·) represents average pooling, represents the first embedded feature of the cth channel, h represents the length of the first encoded image, and w represents the width of the first encoded image.

[0190] In some embodiments, the relationship between the plurality of first image features may be non-aligned and non-linear, and the server can rearrange the channel relationship between the plurality of first image features based on the projection function. Accordingly, the server rearranges the channel relationship between the plurality of first image features based on the projection function.

[0191] The projection function is represented by g(·), which is constructed by a convolution layer or a multi-layer perceptron with a 1×1 convolution kernel. The method of rearranging the first image feature is shown in formula (2):

[0192] f x =g(e x ), f x ∈R c (2);

[0193] Among them, f xrepresents the first image feature after alignment, g(·) represents the projection function, e x represents the first image feature, and c represents the number of channels.

[0194] In step S303, the second image is encoded to obtain a second encoded image.

[0195] The second image is a reference image, that is, a reference image for style conversion. For ease of description, the style of the second image is represented as a second style. The server can encode the second image based on a second encoder to obtain a second encoded image. The second encoder has the same structure as the first encoder.

[0196] In some embodiments, the second encoder includes a second downsampling block and a second residual network, the second residual network includes at least one residual block, and the server can downsample the second image based on the second downsampling block, and then input the sampled image into the second residual network, and the second residual network processes the sampled image to obtain the second encoded image. The second downsampling block includes at least one convolutional layer for extracting useful information from the second image, that is, extracting features from the second image. It should be noted that each of the above convolutional layers and residual networks uses instance normalization for normalization, and the activation function uses the ReLU function.

[0197] For example, the second encoder includes three convolutional layers and three residual blocks in series. The convolution kernel of the first convolutional layer is 7×7 with a stride of 1; the convolution kernel of the second convolutional layer is 3×3 with a stride of 2; the convolution kernel of the third convolutional layer is 3×3 with a stride of 2. The series connection means that the output of the first convolutional layer is the input of the second convolutional layer, the output of the second convolutional layer is the input of the third convolutional layer, the output of the third convolutional layer is the input of the first residual block, the output of the first residual block is the input of the second residual block, and the output of the second residual block is the input of the third residual block.

[0198] It should be noted that the embodiments of the present disclosure are described by taking the first image as the image to be processed and the second image as the reference image. In some embodiments, the second image may be the image to be processed and the first image may be the reference image.

[0199] In step S304, feature extraction is performed on the second encoded image to obtain a plurality of second image features, where the plurality of second image features represent features of the second image.

[0200] The encoded second encoded image includes multiple channels, and the server can extract features from each channel respectively to obtain second image features corresponding to each channel.

[0201] In some embodiments, the server can map the information in the encoded image to a semantically consistent latent space based on an embedding function, where the embedding function is constructed by two residual blocks with shared weights. The server can then perform feature extraction on multiple channels of the second encoded image based on the embedding function to obtain multiple second embedded features, and then perform average pooling on the multiple second embedded features in the channel direction to extract global information of the second embedded features at the channel level to obtain the multiple second image features.

[0202] The embedding function is denoted as q(·), and the second embedding feature is denoted as z y =q(E y (y)), z y ∈R c×w×h , where y represents the second image, c represents the channel of the encoded image, and w and h represent the width and length of the second encoded image. The second image feature is represented by e y , the calculation method of the second image feature is shown in formula (3):

[0203]

[0204] in, represents the second image feature of the c-th channel, avg_pooling(·) represents average pooling, represents the second embedded feature of the c-th channel, h represents the length of the second encoded image, and w represents the width of the second encoded image.

[0205] In some embodiments, the relationship between the plurality of second image features may be non-aligned and non-linear, and the server can rearrange the channel relationship between the plurality of second image features based on the projection function. Accordingly, the server rearranges the channel relationship between the plurality of second image features based on the projection function.

[0206] The projection function is represented by g(·), which is constructed by a convolution layer or a multi-layer perceptron with a 1×1 convolution kernel. The method of rearranging the second image features is shown in formula (4):

[0207] f y =g(e y ), f y ∈R c (4);

[0208] Among them, f y represents the second image feature after alignment, g(·) represents the projection function, e y represents the second image feature, and c represents the number of channels.

[0209] It should be noted that the server can process the first encoded image and the second encoded image based on the same embedding function and the same projection function, that is, the first encoded image and the second encoded image share the same embedding function and the same projection function.

[0210] It should be noted that the server can encode and extract features from the first image and the second image at the same time, that is, the above steps S301 and S303 have no sequence, and the step coding is for the convenience of description.

[0211] In step S305, the similarity between the first image feature and the second image feature corresponding to the same channel position is determined to obtain a plurality of feature similarities.

[0212] The server can establish a connection between multiple first image features and multiple second image features based on mutual learning, and associate the first image features and the second image features corresponding to the same channel position.

[0213] In some embodiments, the server can determine the feature similarity between the first image feature and the second image feature corresponding to the same channel position. The server can determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity of the first image feature and the second image feature; or the server can determine the cross entropy of the first image feature and the second image feature corresponding to the same channel as the feature similarity of the first image feature and the second image feature.

[0214] The calculation method of feature similarity is shown in formula (5):

[0215] s=sim(f x , f y ) (5);

[0216] Among them, s represents feature similarity, sim(·) represents the similarity function, and f x represents the first image feature after alignment, f y Represents the second image feature after alignment.

[0217] It should be noted that according to the above formula (5), Represents the first image feature corresponding to channel c and the second image feature The feature similarity between s c The higher the value of and the second image feature The higher the similarity, the higher the feature on this channel is shared by the first image and the second image; c The lower the value of and the second image feature The lower the similarity, the more inconsistent the features of the first image and the second image on this channel.

[0218] It should be noted that the server can also normalize the similarities of the above multiple features to obtain include c is a positive integer.

[0219] In step S306, a plurality of first target features are determined based on the plurality of first image features and the plurality of feature similarities.

[0220] For any first image feature, the server can determine the product of the first image feature and the corresponding feature similarity as the first target feature corresponding to the first image feature, thereby obtaining multiple first target features. The first target feature is represented as

[0221] In step S307, the plurality of first target features are decoded to obtain a first target image, where the first target image has the same style as the second image.

[0222] After obtaining the above-mentioned multiple first target features, the server can decode the multiple first target features based on the first decoder to obtain the first target image.

[0223] In some embodiments, the first decoder includes a first upsampling block and a third residual network, the third residual network includes at least one residual block, the server can input the multiple first target features into the third residual network, upsample the result output by the third residual network, and obtain the first target image, the first target image has a second style, that is, the same style as the second image. Among them, the first upsampling block includes at least one convolutional layer. It should be noted that each of the above convolutional layers and residual networks uses instance normalization when normalizing, and the activation function uses the ReLU function.

[0224] For example, the first decoder includes three residual blocks and convolution layers connected in series. The convolution kernel of the first convolution layer is 3×3 with a step size of 0.5; the convolution kernel of the second convolution layer is 3×3 with a step size of 0.5; the convolution kernel of the third convolution layer is 7×7 with a step size of 1. The series connection means that the output of the first residual block is the input of the second residual block, the output of the second residual block is the input of the third residual block, the output of the third residual block is the input of the first convolution layer, the output of the first convolution layer is the input of the second convolution layer, and the output of the second convolution layer is the input of the third convolution layer.

[0225] In step S308, a plurality of second target features are determined based on the plurality of second image features and the plurality of feature similarities.

[0226] For any second image feature, the server can determine the product of the second image feature and the corresponding feature similarity as the second target feature corresponding to the second image feature, thereby obtaining multiple second target features. The second target feature is represented as

[0227] In step S309, the plurality of second target features are decoded to obtain a second target image, where the second target image has the same style as the first image.

[0228] After obtaining the above-mentioned multiple second target features, the server can decode the multiple second target features based on the second decoder to obtain the second target image.

[0229] In some embodiments, the second decoder includes a second upsampling block and a fourth residual network, the fourth residual network includes at least one residual block, the server can input the multiple second target features into the fourth residual network, upsample the result output by the fourth residual network, and obtain the second target image, the second target image has a first style, that is, the same style as the first image. Among them, the second upsampling block includes at least one convolutional layer. It should be noted that each of the above convolutional layers and residual networks uses instance normalization when normalizing, and the activation function uses the ReLU function.

[0230] For example, the second decoder includes three residual blocks and convolution layers connected in series. The convolution kernel of the first convolution layer is 3×3 with a step size of 0.5; the convolution kernel of the second convolution layer is 3×3 with a step size of 0.5; the convolution kernel of the third convolution layer is 7×7 with a step size of 1. The series connection means that the output of the first residual block is the input of the second residual block, the output of the second residual block is the input of the third residual block, the output of the third residual block is the input of the first convolution layer, the output of the first convolution layer is the input of the second convolution layer, and the output of the second convolution layer is the input of the third convolution layer.

[0231] It should be noted that the server can decode and determine the first target image and the second target image at the same time, that is, the above steps S306 and S308 have no sequence, and the step numbers are for ease of description.

[0232] The solution provided by the embodiment of the present disclosure can reduce the influence of redundant and inconsistent features in the two images based on the multiple feature similarities by respectively determining multiple feature similarities between multiple first image features of the first image and multiple second image features of the second image, and retain the common features in the two images, thereby avoiding the influence of style conversion caused by the imbalance of information richness between images, making the conversion result more reasonable and improving the quality of the converted image.

[0233] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0234] Figure 4 FIG. 1 is a flowchart of another method for training an image processing model according to an exemplary embodiment. Assume that the electronic device is provided as a server and the server executes the method for training the image processing model as an example, see Figure 4 , the image processing model includes a generator and a discriminator, and the method includes:

[0235] In step S401, the first sample image and the second sample image are input into the encoder and the aligned forgetting layer in the generator to obtain a plurality of sample image features, and the aligned forgetting layer is used to extract image features.

[0236] In step S402, the multiple sample image features are input into the decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image has the same style as the second sample image, and the second sample target image has the same style as the first sample image.

[0237] In step S403, the first sample image, the second sample image, the first sample target image and the second sample target image are input into the discriminator to obtain a training loss.

[0238] In step S404, model training is performed according to the training loss.

[0239] In the implementation of the present disclosure, the image processing model is trained based on the training loss, so that the trained image processing model can generate reasonable conversion results, and the properties of the original image are not lost during the style conversion, and the output of the model is relatively stable and robust.

[0240] In some embodiments, the sample images used for training include a source domain dataset X and a target domain dataset Y, wherein the source domain dataset includes N first sample images, denoted as The target domain dataset includes M second sample images, denoted as N and M are both positive integers.

[0241] Figure 5 is a schematic diagram of a structure of a generator according to an exemplary embodiment. Figure 5 As shown, the generator includes a first encoder E X , the second encoder E Y , aligned forgetting layer U, first decoder G X and the second decoder G Y . Among them, the first encoder is used to encode the picture in the source domain data set, that is, the first sample image; the second encoder is used to encode the picture in the target domain data set, that is, the second sample image. The aligned forgetting layer is used to extract features from the encoded image to obtain sample image features. The aligned forgetting layer is also used to determine the similarity between the sample image features. The aligned forgetting layer is also used to determine the target sample features to be decoded based on the sample image features and the similarity. The first decoder is used to decode the sample target features determined based on the first sample image features to obtain the first sample target image. The second decoder is used to interface the sample target features determined based on the second sample image features to obtain the second sample target image.

[0242] In some embodiments, Figure 6 FIG. 1 is a schematic diagram showing a structure of an aligned forgetting layer according to an exemplary embodiment. Figure 6 As shown in Figure 1, the aligned forgetting layer consists of three parts: the embedding module, the projection module, and the mutual learning module. The embedding module is used to extract information from different domains into the same latent space based on the embedding function q(·), which is denoted as and Represents the embedded features of c channels of the source domain image; represents the embedded features of the c channels of the target domain image. The embedding module is also used to average pool the embedded features and extract the global information of the embedded features from the channel level, denoted as and Represents the first image features of c channels of the source domain image; The second image features of the c channels of the target domain image are calculated by referring to the above formula (1) and formula (3) and will not be repeated here. The projection module is used to rearrange the channel relationship of the image features based on the projection function g(·) to obtain f x and f y , f x represents the first image feature of the alignment, f y represents the aligned second image feature. The mutual learning module is used to determine the similarity between image features, denoted as sim(f x , f y ). The mutual learning module is also used to normalize the feature similarity to obtain represents the feature similarity of c channels, c is a positive integer, and k = c. The mutual learning module is also used to multiply the feature similarity and the image feature and output the multiplication result.

[0243] In some embodiments, the discriminator includes a first discriminator D X and the second discriminator D Y , the first discriminator and the second discriminator have the same structure. The discriminator is used to determine whether the image is a real image or an image generated by the generator. The discriminator is obtained by connecting four convolutional layers in series. Among them, the first three convolutional layers are three convolutional layers with a convolution kernel of 4×4 and a step size of 0.5, and the convolution kernel of the fourth convolutional layer is 4×4 and a step size of 1. Each convolutional layer uses instance normalization when normalizing, and the activation function uses the Leaky ReLU activation function.

[0244] It should be noted that the embodiment of the present disclosure provides an image processing model, through the first encoder E X , aligned forgetting layer U, and the second decoder G Y , which can transfer the style of image x in source domain dataset X to the style of target domain dataset Y, i.e. mapping Through the second encoder E Y , aligned forgetting layer U and first decoder G X , which can transfer the style of image y in the target domain dataset Y to the style corresponding to the source domain dataset X, that is, mapping It should be noted that the first encoder E x , aligned forgetting layer U and first decoder G X , which can realize the reconstruction of image x in the source domain dataset X; the second encoder E Y , Aligned forget layer UG X and the second decoder G Y , which can realize the reconstruction of image y in the target domain dataset Y. Reconstruction means that after processing the input image, the output image keeps the original style and content unchanged to improve the stability of the mapping. The reconstructed image is an intermediate image and is not used as an output result.

[0245] It should be noted that when training the model, the server can convert an image x with a first style into an image with a second style. Right now It means that the image x is converted into image after being processed by the first encoder, the aligned forgetting layer and the second decoder. This process is called forward translation. The server can also transfer images with a second style. Then convert back to the first style and get the image Right now Representing images After being processed by the second encoder, the aligned forgetting layer, and the first decoder, it is converted into an image This process is called backward conversion. Similarly, the server can also convert image y with the second style into an image with the first style. Right now It means that the image y is converted into image after being processed by the second encoder, the aligned forgetting layer and the first decoder. This process is called forward translation. The server can also transfer images with the first style Then convert back to the second style and get the image Right now Representing images After being processed by the first encoder, the aligned forgetting layer, and the second decoder, it is converted into an image This process is called backward conversion. Among them, the forward conversion is used to complete the style conversion task, and the backward conversion is used to ensure the stability of the network mapping, that is, to ensure that the input image can still be converted into the input image itself after two style conversions.

[0246] In some embodiments, since the output results of the forward conversion in the early stage of training are unstable, when performing backward conversion, the embedding module part in the alignment forgetting module is used instead of the projection module and the mutual learning module part, so as to ensure the stability of training.

[0247] In some embodiments, the training loss includes a first loss, a second loss, and a third loss. The first loss includes a first adversarial loss and a second adversarial loss, the first adversarial loss represents the loss of converting the first sample image to the second sample target image, and the second adversarial loss represents the loss of converting the second sample image to the first sample target image. The second loss includes a first image loss and a second image loss, the first image loss represents the difference between the first sample image and the first sample target image, and the second image loss represents the difference between the second sample image and the second sample target image. The third loss includes a first consistency loss and a second consistency loss, the first consistency loss represents the difference between the first sample image and the second sample target image, and the second consistency loss represents the difference between the second sample image and the first sample target image.

[0248] Correspondingly, the first sample image, the second sample image, the first sample target image and the second sample target image are input into the discriminator to obtain the training loss, including: inputting the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain the first loss, the second loss and the third loss, and performing weighted summation of the first loss, the second loss and the third loss to obtain the training loss.

[0249] In some embodiments, the first loss is an adversarial loss, which is used to enable the model to output a reasonable conversion result. X ,U,G Y} and {E Y ,U,G X The input image x and image y are processed respectively. The goal of the generator is to generate an image that is as realistic as possible. The goal of the discriminator is to correctly determine whether the image is an input image or an image generated by the generator. The first loss includes a first adversarial loss and a second adversarial loss calculated based on the following formulas (6) and (7).

[0250]

[0251] Among them, l adv (E X ,U,G Y , D Y ) represents the first adversarial loss, E X represents the first encoder, U represents the aligned forgetting layer, G Y Denotes the second decoder, D Y represents the second discriminator, S data (y) represents the target domain dataset, y represents the image in the target domain dataset, S data (x) represents the source domain dataset, and x represents an image in the source domain dataset.

[0252]

[0253] Among them, l adv (E Y ,U,G X , D X ) represents the second adversarial loss, E Y represents the second encoder, U represents the aligned forgetting layer, G X Denotes the first decoder, D X represents the first discriminator, S data (x) represents the source domain dataset, x represents the image in the source domain dataset, S data (y) represents the target domain dataset, and y represents the image in the target domain dataset.

[0254] In some embodiments, the second loss is an image loss, which can also be called an identity loss. The image loss is used to enable the model to retain important information of the image, thereby ensuring that the input attributes of the original image are not lost when the image is style-converted, that is, the content in the original image is not lost. The generator is based on {E X ,U,G X} and {E Y ,U,G Y}Realize the task of reconstructing the input image x and image y, where {E X ,U,G X} and {E Y ,U,G Y} can also be called an autoencoder. The first image loss and the second image loss included in the second loss are calculated based on the following formulas (8) and (9).

[0255]

[0256] Among them, l identity (E X ,U,G X ) represents the first image loss, S data (x) represents the source domain dataset, x represents the image in the source domain dataset, E X represents the first encoder, U represents the aligned forgetting layer, G x represents the first decoder, and ||·||1 represents the 1-norm.

[0257]

[0258] Among them, l identity (E Y ,U,G Y ) represents the second image loss, S data (y) represents the target domain dataset, y represents the image in the target domain dataset, E Y represents the second encoder, U represents the aligned forgetting layer, G Y represents the second decoder, and ||·||1 represents the 1-norm.

[0259] In some embodiments, the third loss is a cycle consistency loss, which is used to ensure the stability of the model mapping. The third loss is calculated based on the following formula (10).

[0260]

[0261] Among them, l cyc (G, F) indicates the third loss; S data (x) represents the source domain dataset; x represents an image in the source domain dataset; Represents the first encoder E X , aligned forgetting layer U and second decoder G Y Process the image; S data (y) represents the target domain dataset; y represents the image in the target domain dataset; Represents the second encoder E Y , aligned forgetting layer U and the first decoder G XProcess the image; ||·||1 represents the 1-norm.

[0262] In summary, the training loss in the embodiment of the present disclosure is calculated based on the following formula (11).

[0263]

[0264] Where l(E X , E Y ,U,G X , G Y , D X , D Y ) represents the training loss, l adv (E X ,U,G Y , D Y ) represents the first adversarial loss, l adv (E Y ,U,G X , D X ) represents the second adversarial loss, l identity (E X ,U,G X ) represents the first image loss, l identity (E Y ,U,G Y ) represents the second image loss, l cyc (E X , E Y ,U,G X , G Y )=l cyc (G, F) represents the third loss, and λ1 and λ2 represent the weights of the corresponding loss terms.

[0265] For example, the server performs model training with a learning rate of 0.0002, a batch training size of 1, an iteration cycle of 200, and loss function weights of λ1=5 and λ2=10 until the above image processing model is obtained.

[0266] It should be noted that in order to verify the processing effect of the image processing method provided by the embodiment of the present disclosure, the image processing method provided by the present disclosure is also qualitatively compared with the existing representative methods CycleGAN (Cycle Generative Adversarial Networks), UNIT (UNsupervised Image-to-image Translation Networks, an unsupervised image-to-image translation network), GcGAN (Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping, a geometry-consistent generative adversarial network for one-sided unsupervised domain mapping), DRIT++ (Diverse Image-to-Image Translation via Disentangled Representations, achieving different image-to-image conversions through decoupled representations) and UGATIT (Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation, an unsupervised generative attention network with adaptive layer instance normalization for image-to-image conversion). The data set used for the qualitative comparison is the cartoon face data set selfie2anime. The conversion results are shown in the figure. Figure 7 As shown, Figure 7 : is a schematic diagram of a conversion result according to an exemplary embodiment. The first column represents the input image, the second column represents the image output after style conversion based on CycleGAN, the third column represents the image output after style conversion based on UNIT, the fourth column represents the image output after style conversion based on GcGAN, the fifth column represents the image output after style conversion based on DRIT++, and the sixth column represents the image output after style conversion based on UGATIT. The seventh column represents the image output after style conversion based on the image processing method provided by the embodiment of the present disclosure.

[0267] It should be noted that the image processing method provided by the present disclosure is also quantitatively compared with the existing representative methods CycleGAN, UNIT, GcGAN, DRIT++ and UGATIT, and the comparison parameter is the FID score (Frechet Inception Distance score). The FID score is a measure of the distance between the feature vectors of the real image and the generated image. The lower the FID score, the closer the real image and the generated image are. The comparison results are shown in Table 1. The method disclosed in the present disclosure achieved the lowest score, indicating that the conversion result disclosed in the present disclosure is closer to the real cartoon face.

[0268] Table 1

[0269] method CycleGAN UNIT GcGAN DRIT++ UGATIT The Disclosure FID score 84.63 104.86 147.30 118.48 94.59 81.39

[0270] It should be noted that the image processing method provided by the present disclosure was also compared with the existing representative methods CycleGAN, UNIT, GcGAN, DRIT++ and UGATIT in a user evaluation. A total of 37 test users participated in the experiment. In each test question, the test users were required to select the one with the best conversion result among the representative algorithms, a total of 25 questions. The evaluation results are shown in Table 2. The present disclosure achieved the highest support rate, indicating that the conversion results of the present disclosure are more in line with the user's cognition and preference.

[0271] Table 2

[0272] method CycleGAN UNIT GcGAN DRIT++ UGATIT The Disclosure User support rate 8.43％ 2.48％ 7.56％ 2.91％ 32.64％ 46.16％

[0273] It should be noted that the image processing method provided by the embodiments of the present disclosure can be applied to style conversion tasks in a variety of scenarios. Figure 8 FIG. 1 is a schematic diagram showing a style conversion result of a map and a satellite map according to an exemplary embodiment. Figure 8 As shown, there are 6 pairs of images in total. In each pair of images, the image on the left is the input image, and the image on the right is the conversion result. Fig. 9 is a schematic diagram showing a style conversion result of a street scene and a segmented street scene according to an exemplary embodiment. Fig. 9 As shown, there are 6 pairs of images in total. In each pair of images, the image on the left is the input image, and the image on the right is the conversion result. Fig.10 FIG. 1 is a schematic diagram showing a style conversion result of a building and a segmented building according to an exemplary embodiment. Fig.10 As shown, there are 6 pairs of images in total. In each pair of images, the image on the left is the input image, and the image on the right is the conversion result.

[0274] It should be noted that the image processing method provided in the embodiment of the present application can further enhance the capabilities of the network to improve the quality of style conversion. In some embodiments, the above-mentioned alignment forgetting layer has multiple variations: for example, different numbers of residual blocks can be used in the embedding module to extract the common features of the two domains, a convolution layer with a 1×1 convolution kernel or a multi-layer perceptron can be used in the projection module, and cross entropy or dot product can be used in the mutual learning module to measure the similarity, so that the image processing method is scalable and flexible.

[0275] Fig.11 is a comparison chart of experimental results of a variant according to an exemplary embodiment. Fig.11 As shown in the figure, the first column is the input image; the second column is the conversion result after using two residual blocks in the aligned forget layer, using the convolution layer with a 1×1 convolution kernel in the projection module, and using cross entropy in the mutual learning module; the third column is the conversion result after using one residual block in the aligned forget layer, using the convolution layer with a 1×1 convolution kernel in the projection module, and using cross entropy in the mutual learning module; the fourth column is the conversion result after using three residual blocks in the aligned forget layer, using the convolution layer with a 1×1 convolution kernel in the projection module, and using cross entropy in the mutual learning module; the fifth column is the conversion result after using two residual blocks in the aligned forget layer, using the convolution layer with a 1×1 convolution kernel in the projection module, and using dot product in the mutual learning module; the sixth column is the conversion result after using only two residual blocks in the aligned forget layer, using two layers of multilayer perceptrons in the projection module, and using cross entropy in the mutual learning module; the seventh column is the conversion result of using two residual blocks in the aligned forget layer, using two layers of multilayer perceptrons in the projection module, and using dot product in the mutual learning module.

[0276] In addition, when training a graphics processing model, the present invention requires fewer training resources (GPU), less than 3000M of video memory, and a training time of 40 hours. The training efficiency is greatly improved, the training cost is significantly reduced, and the training efficiency is high.

[0277] Fig.12 is a block diagram of an image processing device according to an exemplary embodiment. Fig.12 The device includes: a feature extraction module 1201, a similarity determination module 1202 and an image determination module 1203.

[0278] The feature extraction module 1201 is configured to perform feature extraction on the first image and the second image to obtain a plurality of first image features and a plurality of second image features, wherein the plurality of first image features represent features of the first image, the plurality of second image features represent features of the second image, and the plurality of first image features and the plurality of second image features correspond to each other one by one;

[0279] A similarity determination module 1202 is configured to determine a similarity between a first image feature and a second image feature corresponding to a same channel position, and obtain a plurality of feature similarities;

[0280] The image determination module 1203 is configured to determine a first target image and a second target image based on the multiple first image features, the multiple second image features and the multiple feature similarities, the first target image having the same style as the second image, and the second target image having the same style as the first image.

[0281] The device provided by the embodiment of the present disclosure can reduce the influence of redundant and inconsistent features in the two images based on the multiple feature similarities by respectively determining multiple feature similarities between multiple first image features of the first image and multiple second image features of the second image, and retain the common features in the two images, thereby avoiding the influence of the unbalanced information richness between the images on the style conversion, making the conversion result more reasonable and improving the quality of the converted image.

[0282] In some embodiments, the similarity determination module 1202 is configured to determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity between the first image feature and the second image feature; or, to determine the cross entropy of the first image feature and the second image feature corresponding to the same channel position as the feature similarity between the first image feature and the second image feature.

[0283] In some embodiments, Fig.13 is a block diagram of another image processing device according to an exemplary embodiment. Fig.13 As shown, the feature extraction module 1201 includes:

[0284] A first encoding unit 12011 is configured to encode the first image to obtain a first encoded image;

[0285] A first feature extraction unit 12012 is configured to perform feature extraction on the first encoded image to obtain the plurality of first image features;

[0286] The second encoding unit 12013 is configured to encode the second image to obtain a second encoded image;

[0287] The second feature extraction unit 12014 is configured to perform feature extraction on the second encoded image to obtain the multiple second image features.

[0288] In some embodiments, see Fig.13As shown, the first encoding unit 12011 is configured to downsample the first image, input the sampled image into the first residual network, and obtain the first encoded image.

[0289] In some embodiments, see Fig.13 As shown, the first feature extraction unit 12012 is configured to perform feature extraction on the first encoded image based on the embedding function to obtain multiple first embedded features; and perform average pooling on the multiple first embedded features to obtain the multiple first image features.

[0290] In some embodiments, see Fig.13 As shown, the device also includes:

[0291] The first rearrangement module 1205 is configured to rearrange the channel relationship between the plurality of first image features based on the projection function.

[0292] In some embodiments, the second encoding unit 12013 is configured to downsample the second image, input the sampled image into a second residual network, and obtain the second encoded image.

[0293] In some embodiments, the second feature extraction unit 12014 is configured to perform feature extraction on the second encoded image based on an embedding function to obtain a plurality of second embedded features; and perform average pooling on the plurality of second embedded features to obtain the plurality of second image features.

[0294] In some embodiments, see Fig.13 As shown, the device also includes:

[0295] The second rearrangement module 1206 is configured to rearrange the channel relationship between the plurality of second image features based on the projection function.

[0296] In some embodiments, see Fig.13 As shown, the image determination module 1203 includes:

[0297] The first determining unit 12031 is configured to determine a plurality of first target features based on the plurality of first image features and the plurality of feature similarities;

[0298] The first decoding unit 12032 is configured to decode the plurality of first target features to obtain the first target image;

[0299] The second determining unit 12033 is configured to determine a plurality of second target features based on the plurality of second image features and the plurality of feature similarities;

[0300] The second decoding unit 12034 is configured to decode the multiple second target features to obtain the second target image.

[0301] In some embodiments, the first determination unit 12031 is configured to determine, for any first image feature, a product of the first image feature and the corresponding feature similarity as the first target feature corresponding to the first image feature.

[0302] In some embodiments, the first decoding unit 12032 is configured to upsample the multiple first target features, input the sampled features into a third residual network, and obtain the first target image.

[0303] In some embodiments, the second determination unit 12033 is configured to determine, for any second image feature, a product of the second image feature and the corresponding feature similarity as the second target feature corresponding to the second image feature.

[0304] In some embodiments, the second decoding unit 12034 is configured to upsample the multiple second target features, input the sampled features into a fourth residual network, and obtain the second target image.

[0305] It should be noted that the image processing device provided in the above embodiment only uses the division of the above functional units as an example when processing an image. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0306] Fig.14 is a block diagram of a training device for an image processing model according to an exemplary embodiment. Fig.14 The image processing model includes a generator and a discriminator, and the device includes: a first training module 1401, a second training module 1402 and a third training module 1403.

[0307] The first training module 1401 is configured to input the first sample image and the second sample image into the encoder and the aligned forgetting layer in the generator to obtain a plurality of sample image features, and the aligned forgetting layer is used to extract image features;

[0308] The second training module 1402 is configured to input the multiple sample image features into a decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image has the same style as the second sample image, and the second sample target image has the same style as the first sample image;

[0309] A third training module 1403 is configured to input the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain a training loss;

[0310] The third training module 1403 is further configured to perform model training according to the training loss.

[0311] In some embodiments, the third training module 1403 is configured to input the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain a first loss, a second loss and a third loss, the first loss includes a first adversarial loss and a second adversarial loss, the first adversarial loss represents the loss of converting the first sample image into the second sample target image, the second adversarial loss represents the loss of converting the second sample image into the first sample target image, the second loss includes a first image loss and a second image loss, the first image loss represents the difference between the first sample image and the first sample target image, the second image loss represents the difference between the second sample image and the second sample target image, the third loss includes a first consistency loss and a second consistency loss, the first consistency loss represents the difference between the first sample image and the second sample target image, and the second consistency loss represents the difference between the second sample image and the first sample target image; the first loss, the second loss and the third loss are weightedly summed to obtain the training loss.

[0312] It should be noted that the training device for the image processing model provided in the above embodiment only uses the division of the above functional units as an example when training the image processing model. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device for the image processing model provided in the above embodiment and the training method embodiment of the image processing model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0313] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0314] When the electronic device is provided as a server, Fig.15It is a block diagram of a server 1500 shown according to an exemplary embodiment. The server 1500 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1501 and one or more memories 1502, wherein the memory 1502 stores at least one program code, and the at least one program code is loaded and executed by the processor 1501 to implement the image processing method provided by the above-mentioned various method embodiments, or the at least one program code is loaded and executed by the processor 1501 to implement the training method of the image processing model provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1500 may also include other components for implementing device functions, which will not be repeated here.

[0315] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1502 including instructions, and the instructions can be executed by a processor 1501 of a server 1500 to perform the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0316] A computer program product includes a computer program, which implements the above-mentioned image processing method when executed by a processor, or implements the above-mentioned image processing model training method when executed by a processor.

[0317] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0318] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Encoding the first image to obtain a first encoded image; Based on the embedding function, extracting features from the first encoded image to obtain a plurality of first embedded features; Performing average pooling on the multiple first embedded features respectively to obtain multiple first image features, where the multiple first image features represent features of the first image; encoding the second image to obtain a second encoded image; Based on the embedding function, extracting features from the second encoded image to obtain a plurality of second embedded features; Performing average pooling on the plurality of second embedded features respectively to obtain a plurality of second image features, wherein the plurality of second image features represent features of the second image, and the plurality of first image features correspond to the plurality of second image features in one-to-one correspondence; Determine a similarity between a first image feature and a second image feature corresponding to a same channel position to obtain a plurality of feature similarities; Based on the multiple first image features, the multiple second image features and the multiple feature similarities, a first target image and a second target image are determined, the first target image and the second image have the same style, and the second target image and the first image have the same style.

2. The image processing method according to claim 1, characterized in that: The determining of the similarity between the first image feature and the second image feature corresponding to the same channel position to obtain multiple feature similarities includes: Determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity between the first image feature and the second image feature; or, The cross entropy between the first image feature and the second image feature corresponding to the same channel position is determined as the feature similarity between the first image feature and the second image feature.

3. The image processing method according to claim 1, characterized in that: The step of encoding the first image to obtain a first encoded image includes: The first image is downsampled, and the sampled image is input into a first residual network to obtain the first encoded image.

4. The image processing method according to claim 1, characterized in that: The method further comprises: Based on the projection function, the channel relationships between the plurality of first image features are rearranged.

5. The image processing method according to claim 1, characterized in that: The step of encoding the second image to obtain a second encoded image includes: Down-sample the second image, and input the sampled image into a second residual network to obtain the second encoded image.

6. The image processing method according to claim 1, characterized in that: The method further comprises: Based on the projection function, the channel relationships between the plurality of second image features are rearranged.

7. The image processing method according to claim 1, characterized in that: The determining of the first target image and the second target image based on the plurality of first image features, the plurality of second image features and the plurality of feature similarities comprises: Determining a plurality of first target features based on the plurality of first image features and the plurality of feature similarities; Decoding the plurality of first target features to obtain the first target image; Determining a plurality of second target features based on the plurality of second image features and the plurality of feature similarities; The plurality of second target features are decoded to obtain the second target image.

8. The image processing method according to claim 7, characterized in that: The determining of a plurality of first target features based on the plurality of first image features and the plurality of feature similarities comprises: For any first image feature, the product of the first image feature and the corresponding feature similarity is determined as the first target feature corresponding to the first image feature.

9. The image processing method according to claim 7, characterized in that: The decoding of the plurality of first target features to obtain the first target image includes: The multiple first target features are input into a third residual network, and a result output by the third residual network is upsampled to obtain the first target image.

10. The image processing method according to claim 7, characterized in that: The determining of a plurality of second target features based on the plurality of second image features and the plurality of feature similarities comprises: For any second image feature, the product of the second image feature and the corresponding feature similarity is determined as the second target feature corresponding to the second image feature.

11. The image processing method according to claim 7, characterized in that: The decoding of the plurality of second target features to obtain the second target image includes: The multiple second target features are input into a fourth residual network, and the result output by the fourth residual network is upsampled to obtain the second target image.

12. A training method for an image processing model, characterized in that: The image processing model includes a generator and a discriminator, and the method includes: Inputting the first sample image and the second sample image into the encoder and the aligned forgetting layer in the generator to obtain multiple sample image features, wherein the aligned forgetting layer is used to extract image features, and the aligned forgetting layer includes an embedding module, a projection module, and a mutual learning module, wherein the embedding module is used to perform feature extraction and average pooling through an embedding function to obtain image features, the projection module is used to arrange channel relationships of image features through a projection function, and the mutual learning module is used to determine the similarity between image features; Inputting the plurality of sample image features into a decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image and the second sample image have the same style, and the second sample target image and the first sample image have the same style; Inputting the first sample image, the second sample image, the first sample target image, and the second sample target image into the discriminator to obtain a training loss; Model training is performed according to the training loss.

13. The method for training an image processing model according to claim 12, characterized in that: The step of inputting the first sample image, the second sample image, the first sample target image, and the second sample target image into the discriminator to obtain a training loss includes: Input the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain a first loss, a second loss and a third loss, wherein the first loss includes a first adversarial loss and a second adversarial loss, the first adversarial loss represents the loss of converting the first sample image into the second sample target image, the second adversarial loss represents the loss of converting the second sample image into the first sample target image, the second loss includes a first image loss and a second image loss, the first image loss represents the difference between the first sample image and the first sample target image, the second image loss represents the difference between the second sample image and the second sample target image, the third loss includes a first consistency loss and a second consistency loss, the first consistency loss represents the difference between the first sample image and the second sample target image, and the second consistency loss represents the difference between the second sample image and the first sample target image; The first loss, the second loss and the third loss are weightedly summed to obtain the training loss.

14. An image processing device, characterized in that: The device comprises: A first encoding unit is configured to encode the first image to obtain a first encoded image; A first feature extraction unit is configured to perform feature extraction on the first encoded image based on an embedding function to obtain a plurality of first embedded features; perform average pooling on the plurality of first embedded features to obtain a plurality of first image features, wherein the plurality of first image features represent features of the first image; A second encoding unit is configured to encode the second image to obtain a second encoded image; A second feature extraction unit is configured to perform feature extraction on the second encoded image based on an embedding function to obtain a plurality of second embedded features; perform average pooling on the plurality of second embedded features to obtain a plurality of second image features, wherein the plurality of second image features represent features of the second image, and the plurality of first image features correspond to the plurality of second image features in one-to-one correspondence; a similarity determination module configured to determine a similarity between a first image feature and a second image feature corresponding to a same channel position, and obtain a plurality of feature similarities; The image determination module is configured to determine a first target image and a second target image based on the multiple first image features, the multiple second image features and the multiple feature similarities, wherein the first target image has the same style as the second image, and the second target image has the same style as the first image.

15. The device according to claim 14, characterized in that The similarity determination module is configured to determine the dot product of the first image feature and the second image feature corresponding to the same channel position as the feature similarity of the first image feature and the second image feature; or to determine the cross entropy of the first image feature and the second image feature corresponding to the same channel position as the feature similarity of the first image feature and the second image feature.

16. The image processing device according to claim 14, characterized in that: The first encoding unit is configured to downsample the first image, input the sampled image into a first residual network, and obtain the first encoded image.

17. The image processing device according to claim 14, characterized in that: The device also includes: The first rearrangement module is configured to rearrange the channel relationship between the plurality of first image features based on the projection function.

18. The image processing device according to claim 14, characterized in that: The second encoding unit is configured to downsample the second image, input the sampled image into a second residual network, and obtain the second encoded image.

19. The image processing device according to claim 14, characterized in that: The device also includes: The second rearrangement module is configured to rearrange the channel relationship between the plurality of second image features based on the projection function.

20. The image processing device according to claim 14, characterized in that: The image determination module comprises: A first determining unit is configured to determine a plurality of first target features based on the plurality of first image features and the plurality of feature similarities; A first decoding unit is configured to decode the plurality of first target features to obtain the first target image; A second determining unit is configured to determine a plurality of second target features based on the plurality of second image features and the plurality of feature similarities; The second decoding unit is configured to decode the plurality of second target features to obtain the second target image.

21. The image processing device according to claim 20, characterized in that: The first determination unit is configured to determine, for any first image feature, a product of the first image feature and a corresponding feature similarity as a first target feature corresponding to the first image feature.

22. The image processing device according to claim 20, characterized in that: The first decoding unit is configured to input the multiple first target features into a third residual network, upsample the result output by the third residual network, and obtain the first target image.

23. The image processing device according to claim 20, characterized in that: The second determination unit is configured to determine, for any second image feature, a product of the second image feature and a corresponding feature similarity as a second target feature corresponding to the second image feature.

24. The image processing device according to claim 20, characterized in that: The second decoding unit is configured to input the multiple second target features into a fourth residual network, upsample the result output by the fourth residual network, and obtain the second target image.

25. A training device for an image processing model, characterized in that: The image processing model includes a generator and a discriminator, and the device includes: A first training module is configured to input the first sample image and the second sample image into the encoder and the aligned forgetting layer in the generator to obtain multiple sample image features, the aligned forgetting layer is used to extract image features, the aligned forgetting layer includes an embedding module, a projection module and a mutual learning module, the embedding module is used to extract features and average pooling through an embedding function to obtain image features, the projection module is used to arrange channel relationships of image features through a projection function, and the mutual learning module is used to determine the similarity between image features; A second training module is configured to input the plurality of sample image features into a decoder in the generator to obtain a first sample target image and a second sample target image, wherein the first sample target image has the same style as the second sample image, and the second sample target image has the same style as the first sample image; A third training module is configured to input the first sample image, the second sample image, the first sample target image, and the second sample target image into the discriminator to obtain a training loss; The third training module is further configured to perform model training according to the training loss.

26. The image processing model training device according to claim 25, characterized in that: The third training module is configured to input the first sample image, the second sample image, the first sample target image and the second sample target image into the discriminator to obtain a first loss, a second loss and a third loss, wherein the first loss includes a first adversarial loss and a second adversarial loss, the first adversarial loss represents the loss of converting the first sample image to the second sample target image, the second adversarial loss represents the loss of converting the second sample image to the first sample target image, the second loss includes a first image loss and a second image loss, the first image loss represents the difference between the first sample image and the first sample target image, the second image loss represents the difference between the second sample image and the second sample target image, the third loss includes a first consistency loss and a second consistency loss, the first consistency loss represents the difference between the first sample image and the second sample target image, and the second consistency loss represents the difference between the second sample image and the first sample target image; The first loss, the second loss and the third loss are weightedly summed to obtain the training loss.

27. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the image processing method as described in any one of claims 1 to 11, or to implement the training method of the image processing model as described in any one of claims 12 to 13.

28. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image processing method as described in any one of claims 1 to 11, or to execute the training method of the image processing model as described in any one of claims 12 to 13.

29. A computer program product comprising a computer program / instructions, characterized in that When the computer program is executed by a processor, it implements the image processing method described in any one of claims 1 to 11, or when the computer program is executed by a processor, it implements the training method of the image processing model described in any one of claims 12 to 13.

Images