Image processing method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a method for image processing, electronic equipment and a storage medium, and relates to the field of artificial intelligence. A method for image processing includes: performing style migration on an input image by using a generator in an adversarial network to obtain a first migrated image; obtaining migration quality evaluation information of the first migration image by using a discriminator in the adversarial network, and obtaining gradient information associated with style migration loss according to the migration quality evaluation information; and obtaining update information about the first migrated image by using an artificial intelligence network based on the gradient information, and updating the first migrated image based on the update information to obtain a second migrated image. Optionally, the method may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to a method for image processing, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image style transfer (also known as "image style conversion") is an important direction in the field of image processing. Its task is to transfer the style of a target image while maintaining the content of the original target image. Image transfer technology has wide applications in artistic creation, image editing, etc., providing a powerful tool for image processing and creative design, enabling unique styles and effects to be achieved in images. However, in current image style transfer technologies, there are still problems such as low quality of image style transfer, slow running speed of image style transfer, and user experience during the style transfer process. In view of this, an image style transfer technology that can solve at least one of the above problems is needed. Summary of the Invention

[0003] According to a first aspect of an embodiment of the present disclosure, there is provided a method for image processing, including: performing style transfer on an input image by using a generator in an adversarial network to obtain a first transferred image; obtaining migration quality evaluation information of the first transferred image by using a discriminator in the adversarial network, and obtaining gradient information associated with a style transfer loss according to the migration quality evaluation information; obtaining update information about the first transferred image by using an artificial intelligence network based on the gradient information, and updating the first transferred image based on the update information to obtain a second transferred image.

[0004] Optionally, the method further includes: obtaining control information of a user about image style transfer, where obtaining the gradient information associated with the loss of the generator according to the migration quality evaluation information includes: obtaining the gradient information according to the migration quality evaluation information and the control information.

[0005] Optionally, performing style transfer on an input image by using a generator in the adversarial network to obtain a first transferred image includes: encoding the input image by using an encoder in the generator to obtain a first latent variable; decoding the first latent variable by using a decoder of the generator to obtain a first transferred image; where updating the first transferred image based on the update information to obtain a second transferred image includes: updating the first latent variable based on the update information to obtain a second latent variable; decoding the second latent variable by using the decoder to obtain a second transferred image.

[0006] Optionally, the migration quality evaluation information includes credibility information of the first migrated image compared with the real image, and obtaining gradient information associated with the style migration loss according to the migration quality evaluation information includes: calculating a style migration loss according to the migration quality evaluation information and the ground-truth information, and obtaining the gradient information based on the style migration loss, where the ground-truth information includes information related to the credibility expected for pixels in the first migrated image.

[0007] Optionally, the control information includes information for controlling at least one of the following: the direction of style migration, the degree of style migration, and the position of style migration.

[0008] According to a second aspect of the embodiments of the present disclosure, there is provided a method for image processing, including: performing style migration on an input image by using an image style migration model to obtain a first migrated image; obtaining control information of a user regarding image style migration; obtaining gradient information associated with a style migration loss according to the control information; and updating the first migrated image based on the gradient information to obtain a second migrated image.

[0009] Optionally, obtaining gradient information associated with a style migration loss according to the control information includes: adjusting the style migration loss according to the control information, and obtaining the gradient information based on the adjusted style migration loss.

[0010] Optionally, the image style migration model is a generator in an adversarial network, and obtaining gradient information associated with a style migration loss based on the control information includes: obtaining migration quality evaluation information of the first migrated image by using a discriminator in the adversarial network; and obtaining the gradient information according to the migration quality evaluation information and the control information.

[0011] Optionally, updating the first migrated image based on the gradient information to obtain a second migrated image includes: obtaining update information about the first migrated image by using an artificial intelligence network based on the gradient information; and updating the first migrated image based on the update information to obtain a second migrated image.

[0012] Optionally, the image style migration model is a generator of a generative adversarial network, and performing style migration on an input image by using the image style migration model to obtain a first migrated image includes: encoding the input image by using an encoder in the generator to obtain a first latent variable; and decoding the first latent variable by using a decoder of the generator to obtain a first migrated image; where updating the first migrated image based on the update information to obtain a second migrated image includes: updating the first latent variable based on the update information to obtain a second latent variable; and decoding the second latent variable by using the decoder to obtain a second migrated image.

[0013] Optionally, the control information includes information for controlling at least one of the following: the direction of style transfer, the degree of style transfer, and the location of style transfer.

[0014] In the method according to the first aspect of the embodiments of the present disclosure and the method according to the second aspect of the embodiments of the present disclosure above, optionally, obtaining the gradient information according to the migration quality evaluation information and the control information includes: obtaining a first gradient associated with the style transfer loss according to the migration quality evaluation information; obtaining a second gradient associated with the style transfer loss according to the control information; and fusing the first gradient and the second gradient to obtain the gradient information.

[0015] In the method provided according to the first aspect of the embodiments of the present disclosure or the method provided according to the second aspect of the embodiments of the present disclosure above, optionally, the migration quality evaluation information includes credibility information of the first migrated image compared with the real image, wherein obtaining the first gradient associated with the style transfer loss according to the migration quality evaluation information includes: calculating the style transfer loss according to the migration quality evaluation information and the ground truth information, and obtaining the first gradient based on the style transfer loss, wherein the ground truth information includes information related to the credibility expected of the pixels in the first migrated image.

[0016] In the method provided according to the first aspect of the embodiments of the present disclosure or the method provided according to the second aspect of the embodiments of the present disclosure above, optionally, obtaining the second gradient associated with the style transfer loss according to the control information includes: adjusting the style transfer loss according to the control information, and obtaining the second gradient based on the adjusted style transfer loss.

[0017] In the method provided according to the first aspect of the embodiments of the present disclosure or the method provided according to the second aspect of the embodiments of the present disclosure above, optionally, adjusting the style transfer loss according to the control information includes: adjusting the quality migration evaluation information and / or the ground truth information according to the control information; and obtaining the adjusted style transfer loss based on the adjusted quality migration evaluation information and / or the adjusted ground truth information.

[0018] In the method provided according to the first aspect of the embodiments of the present disclosure or the method provided according to the second aspect of the embodiments of the present disclosure above, optionally, adjusting the quality migration evaluation information and / or the ground truth information according to the control information includes: adjusting the credibility information corresponding to at least one pixel in the first migrated image according to the control information; and / or adjusting the ground truth information corresponding to at least one pixel in the first migrated image according to the control information.

[0019] In the method provided in the first aspect according to the embodiments of the present disclosure or the method provided in the second aspect according to the embodiments of the present disclosure, optionally, fusing the first gradient and the second gradient to obtain the gradient information includes: obtaining weight information of the first gradient and the second gradient according to the control information; and fusing the first gradient and the second gradient based on the weight information to obtain the gradient information.

[0020] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a memory; and a processor, coupled to the memory and configured to execute the method as described above.

[0021] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing instructions, which, when run by at least one processor, cause the at least one processor to execute the method as described above.

[0022] According to the embodiments of the present disclosure, after performing style transfer on an input image by using a generator in an adversarial network to obtain a first transferred image, since obtaining migration quality evaluation information of the first transferred image by using a discriminator in the adversarial network, obtaining gradient information associated with a style transfer loss according to the migration quality evaluation information, then obtaining update information about the first transferred image by using an artificial intelligence network based on the gradient information, and updating the first transferred image based on the update information to obtain a second transferred image, therefore, not only is the improvement of the image style transfer quality achieved by using the artificial intelligence network to mine the information contained in the gradient information to update the first transferred image, but also this update method runs faster than the method that needs to perform a large number of optimization operations to improve the image style transfer quality.

[0023] According to the embodiments of the present disclosure, after performing style transfer on an input image by using an image style transfer model to obtain a first transferred image, since obtaining control information of a user about image style transfer, obtaining gradient information associated with a style transfer loss based on the control information, and updating the first transferred image based on the gradient information to obtain a second transferred image, therefore, the control of the user over image style transfer can be realized by the user's control of the gradient information, so that a transferred image satisfactory to the user can be obtained and the user experience is better.

[0024] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing example embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation to the present disclosure.

[0026] Figure 1 It is a schematic diagram showing image style transfer according to an embodiment of the present disclosure.

[0027] Figure 2 It shows an application example of image style transfer according to an embodiment of the present disclosure.

[0028] Figure 3 It is a flowchart of a method for image processing according to an embodiment of the present disclosure.

[0029] Figure 4 It is a schematic diagram showing a method for image processing according to an embodiment of the present disclosure.

[0030] Figure 5 It is a schematic diagram showing an example of two-stage style transfer according to an embodiment of the present disclosure.

[0031] Figure 6 and Figure 7 It is a schematic diagram showing the effect of two-stage style transfer according to an embodiment of the present disclosure.

[0032] Figure 8 It is a flowchart of a method for image processing according to another embodiment of the present disclosure.

[0033] Figure 9 It is a schematic diagram showing a method for image processing according to another embodiment of the present disclosure.

[0034] Figure 10 and Figure 11 It is a schematic diagram of user-guided style enhancement according to an embodiment of the present disclosure.

[0035] Figure 12 and Figure 13 It is a schematic diagram of user-guided style weakening according to an embodiment of the present disclosure.

[0036] Figure 14 It is a schematic diagram of the effect of user-guided style transfer according to an embodiment of the present disclosure.

[0037] Figure 15 It is another schematic diagram showing two-stage style transfer according to an embodiment of the present disclosure.

[0038] Figure 16 It is a schematic diagram showing an example of the overall framework of a method for image processing according to an embodiment of the present disclosure.

[0039] Figure 17 It is a schematic diagram of gradient fusion according to an embodiment of the present disclosure.

[0040] Figure 18It is a schematic diagram of the application of the method for image processing according to an embodiment of the present disclosure on a mobile terminal.

[0041] Figure 19 It is a block diagram showing an electronic device according to an embodiment of the present disclosure.

[0042] Figure 20 It is a schematic structural diagram showing an electronic device according to an embodiment of the present disclosure. Detailed implementation manners

[0043] The following description with reference to the accompanying drawings is provided to facilitate a thorough understanding of the various embodiments of the present disclosure defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should only be considered exemplary. Therefore, those of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0044] The terms and phrases used in the following specification and claims are not limited to their dictionary meanings but are used solely by the inventors to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of the various embodiments of the present disclosure is provided only for illustrative purposes and not for the purpose of limiting the present disclosure as defined by the appended claims and their equivalents.

[0045] It should be understood that the singular forms "a", "an", and "the" may also include plural referents unless the context clearly indicates otherwise. Thus, for example, a reference to "the surface of a component" includes a reference to one or more such surfaces. When we say that one element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may mean that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here may include a wireless connection or wireless coupling.

[0046] The term "comprises" or "may comprise" refers to the presence of the corresponding disclosed functions, operations, or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations, or features. In addition, the term "comprises" or "has" may be interpreted as indicating certain characteristics, numbers, steps, operations, components, components, or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, components, components, or combinations thereof.

[0047] As used in various embodiments of the present disclosure, the term "or" includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, for the description of "parameter A includes A1, A2, and A3", it may be implemented such that parameter A includes A1 or A2 or A3, or it may also be implemented such that parameter A includes at least two of the three items A1, A2, and A3.

[0048] Unless otherwise defined, all terms (including technical or scientific terms) used in the present disclosure have the same meaning as understood by those skilled in the art described in the present disclosure. As used in a dictionary, ordinary terms are interpreted to have a meaning consistent with the context in the relevant technical field, and should not be interpreted idealistically or overly formally unless explicitly defined as such in the present disclosure.

[0049] At least some of the functions of the device or electronic device provided in the embodiments of the present disclosure can be implemented by an AI model. For example, at least one of the multiple modules of the device or electronic device can be implemented by an AI model. The functions associated with AI can be executed by a non-volatile memory, a volatile memory, and a processor.

[0050] The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), etc., or a pure graphics processing unit, such as a graphics processing unit (GPU), a vision processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0051] The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence (AI) models stored in the non-volatile memory and the volatile memory. The predefined operation rules or artificial intelligence models are provided through training or learning.

[0052] Here, providing through learning means obtaining a predefined operation rule or an AI model with desired characteristics by applying a learning algorithm to multiple learning data. The learning can be performed in the device or electronic device itself where the AI according to the embodiment is executed, and / or can be implemented through a separate server / system.

[0053] An AI model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations through the computation between the input data of the layer (such as the computation result of the previous layer and / or the input data of the AI model) and the multiple weight values of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0054] A learning algorithm is a method of using multiple learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0055] According to the present disclosure, at least one step in the method performed by an electronic device, such as steps of obtaining a first migrated image, obtaining update information about the first migrated image, etc., may be implemented using an artificial intelligence model. The processor of the electronic device may perform a preprocessing operation on the data to convert it into a form suitable for use as the input of the artificial intelligence model. The artificial intelligence model may be obtained through training. Here, "obtained through training" means obtaining a predefined operation rule or an artificial intelligence model configured to perform an expected feature (or purpose) by training a basic artificial intelligence model with multiple training data using a training algorithm.

[0056] The technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure will be described below through the description of several alternative embodiments. It should be noted that the following embodiments may refer to, draw on, or combine with each other. For the same terms, similar features, and similar implementation steps in different embodiments, they will not be described repeatedly.

[0057] The present disclosure relates to image style transfer. Therefore, the concept of image style transfer will be briefly introduced first. From the perspective of the style transfer task, an image is composed of two parts: content and style. The content mainly refers to the layout of the image, the relative positions of objects in the image, etc., while the style refers to the color, line texture, and shape of objects in the image. As Figure 1As shown, the task of image style transfer is to transform the style A of the target image while keeping the content A in the target image unchanged, so that the generated image has style B while having content A. That is to say, image style transfer decouples an image into content A and style A, and under the premise of keeping the image content A unchanged, transforms the style of the image from style A to style B. Image transfer technology has a wide range of applications in artistic creation, image editing, etc. For artistic creation, artists and designers can use image style transfer technology to endow their works with various different artistic styles. For image editing, users can use this technology to apply different artistic styles to their photos to achieve unique image effects. For film and game production, image style transfer can be used to change the scene style in a film or game to create different visual atmospheres. For virtual reality and augmented reality: in virtual and augmented reality applications, image style transfer can be used to improve the visual experience and make the virtual environment more realistic or fantastical. For example, as Figure 2 shown, an image can be transformed from a sunny-day style to a cloudy-day style (as shown in (a) of Figure 2 ), a person in the image can be transformed from not wearing makeup to wearing makeup (as shown in (b) of Figure 2 ), and a sheep in the image can be transformed into a giraffe (as shown in (c) of Figure 2 ). In short, image style transfer technology provides a powerful tool for image processing and creative design, enabling unique styles and effects to be achieved in images.

[0058] However, as mentioned in the background art, in current image style transfer technologies, there are still problems such as low quality of image style transfer, slow running speed of image style transfer, and poor user experience during the style transfer process. In response to this, the present disclosure proposes various methods for image processing to solve at least one of the above problems.

[0059] Figure 3 A flowchart showing a method for image processing according to an embodiment of the present disclosure is shown.

[0060] As Figure 3 shown, in step S310, a style transfer is performed on the input image using a generator in an adversarial network to obtain a first transferred image. As an example, the adversarial network may be a Generative Adversarial Network (GAN) or a variant of GAN, but is not limited thereto. According to an embodiment, step S310 may include: encoding the input image using an encoder in the generator to obtain a first latent variable; decoding the first latent variable using a decoder of the generator to obtain a first transferred image.

[0061] For most existing image style transfer methods based on adversarial networks, for example, image style transfer methods based on CycleGAN (Cycle Generative Adversarial Network), etc., the structure of the network consists of a discriminator and a generator, and the generator is continuously optimized by means of adversarial training. The discriminator is used to evaluate the authenticity of the generated images. The generator receives the feedback from the discriminator and tries to generate images that can fool the discriminator. At the same time, the discriminator is also constantly improving its ability to distinguish between real images and generated images. However, after most existing image style transfer methods based on adversarial networks obtain the generator through training, the discriminator is no longer used in the actual application stage, and only the generator is used to perform style transfer on the input image to obtain the transferred image. However, through the research of the applicant of the present disclosure, it is found that the discriminator can be used to evaluate the generation quality of each part of the generated image, which contains a lot of information such as style transfer and image generation. If the rich information contained in the output of the discriminator is fully exploited during the image generation process, it will be beneficial to generate higher-quality transferred images.

[0062] To this end, according to an embodiment of the present disclosure, in step S320, the discriminator in the adversarial network is used to obtain the transfer quality evaluation information of the first transferred image, and the gradient information associated with the style transfer loss is obtained according to the transfer quality evaluation information. In addition, in step S330, based on the gradient information, an artificial intelligence network is used to obtain the update information about the first transferred image, and the first transferred image is updated based on the update information to obtain the second transferred image. According to an embodiment of the present disclosure, since the artificial intelligence network is used to mine the information contained in the gradient information to update the first transferred image, the improvement of the image style transfer quality can be achieved, and the running speed of this way of updating the transferred image is faster compared with the way that requires a large number of optimization operations and consumes too much computing resources to improve the image style transfer quality.

[0063] Figure 4 It is a schematic diagram showing a method for image processing according to an embodiment of the present disclosure. As Figure 4 shown, after obtaining the input image, the input image can be input into the generator in the adversarial network, and the generator is used to perform style transfer on the input image to obtain the first transferred image. The first transferred image is a rough transferred image. The present disclosure does not discard the discriminator after the adversarial network training is completed and only uses the generator to obtain the transferred image. Instead, when performing the image style transfer task, the discriminator is used to update the first transferred image to obtain a higher-quality transferred image. Specifically, as Figure 4In the example shown, the first transferred image obtained by the generator can be input into the discriminator. The discriminator is used to obtain the transferred quality evaluation information of the first transferred image (such as discrimination information regarding true / false), and based on the transferred quality evaluation information, gradient information associated with the style transfer loss is obtained. The gradient information contains information about where in the first transferred image needs to be modified or updated. Finally, the gradient information is input into an artificial intelligence network (in the following text and in other figures including Figure 4 the artificial intelligence network may also be referred to as a "gradient-guided network"), to obtain update information regarding the first transferred image. Then, the generator can update the first transferred image based on this update information to obtain the final transferred image, that is, the second transferred image. That is to say, the method for image style transfer according to the embodiments of the present disclosure is a two-stage style transfer. The first transferred image is generated through a one-stage style transfer, and then, the first transferred image is updated through a two-stage style transfer to obtain a second transferred image of higher quality.

[0064] Figure 5 FIG. is a schematic diagram showing an example of a two-stage style transfer according to the embodiments of the present disclosure.

[0065] As Figure 5 shown in the example, in the one-stage style transfer, first, the encoder in the generator is used to encode the input image to obtain a first latent variable, and the decoder of the generator is used to decode the first latent variable to obtain the first transferred image. The first latent variable can be the encoded image features of the input image. Subsequently, the discriminator is used to obtain the transferred quality evaluation information of the first transferred image. According to the embodiment, the transferred quality evaluation information may include the credibility information of the first transferred image compared with the real image, such as discrimination information regarding whether the first transferred image is true or false, and this discrimination information includes the credibility information corresponding to the pixels (for example, each pixel) in the first transferred image. For example, the transferred quality evaluation information can be a discrimination score map DM A output by the discriminator, where the value at each position in the discrimination score map DM A represents the credibility corresponding to the pixel at that position. A position with a lower value indicates a lower credibility of the pixel at the corresponding position, and a position with a higher value indicates a higher credibility of the pixel at the corresponding position. As an example, the discriminator here can be a discriminator based on a fully convolutional network (FCN), but is not limited thereto. In fact, the present disclosure places no restrictions on the specific implementation manners of the generator and discriminator in the adversarial network mentioned above.

[0066] In the two-stage style transfer, according to an embodiment, a style transfer loss can be calculated based on transfer quality assessment information and ground truth information, and gradient information can be obtained based on the style transfer loss, where the ground truth information includes information related to the credibility expected for each pixel in the first transferred image. For example, the ground truth information (GT) can be a map with the same size as the discriminant score map DM A in which the value at the position corresponding to each pixel in the first transferred image is 1, indicating that the credibility expected for each pixel is 1. For example, the style transfer loss loss A can be calculated based on the discriminant score map DM A and the ground truth information GT

[0067] For example, the mean square error (MSE) loss function can be used to calculate the style transfer loss. For the MSE loss function, the higher the loss, the greater the gradient; on the contrary, the lower the loss, the smaller the gradient. The MSE loss can be calculated based on the discriminant score map DM A and the ground truth information GT as the style transfer loss loss A . According to an embodiment, gradient information can be obtained by performing one backpropagation based on the style transfer loss. For example, one backpropagation can be performed to obtain the gradient G A as shown in Equation 1 below:

[0068]

[0069] where DM A represents the discriminant score map output by the discriminator, GT is the ground truth information, |DM A - represents the style transfer loss, and L 0 represents the first latent variable mentioned above.

[0070] After obtaining the gradient information, update information about the first transferred image can be obtained based on the gradient information using an artificial intelligence network, and the first transferred image can be updated based on the update information to obtain a second transferred image. For example, the first latent variable can be updated based on the update information to obtain a second latent variable, and then the decoder of the generator can be used to decode the second latent variable to obtain the second transferred image. Optionally, the artificial intelligence network can be a pre-trained artificial intelligence network, and the performance of the artificial intelligence network can be improved through pre-training. Optionally, the artificial intelligence network may not be pre-trained.

[0071] For example, as Figure 5 shown, after obtaining the gradient G AAfter that, as shown in Equation 2 below, the gradient guidance network U can be used to mine the gradient G A to obtain the information contained therein and obtain the update information U(G 0 ) of the first latent variable L A . Based on U(G A ), the first latent variable L 0 is updated to obtain the second latent variable L 1 .

[0072] L 1 = L 0 + U(G A ) Equation 2

[0073] Finally, the decoder in the generator is used to decode the second latent variable L 1 to obtain the second transferred image.

[0074] Figure 6 and Figure 7 are schematic diagrams showing the two-stage style transfer effect according to an embodiment of the present disclosure.

[0075] For example, as Figure 6 shown, there are some unrealistic regions in the first transferred image output after performing one-stage style transfer on the input image, such as regions with artificial generation traces. The discriminant score map output by the discriminator is used to evaluate the credibility of the first transferred image. The discriminant score map contains information about the style transfer quality of each pixel in the first transferred image, where lower values indicate lower credibility of the corresponding pixels. For example, according to the MSE loss function defined above, it can be known that the lower the credibility, the greater the loss, and the relatively higher the gradient at the corresponding position. As can be seen from the Figure 6 visualization result shown, the positions with higher values in the gradient visualization map exactly correspond to the positions with artificial generation traces in the first transferred image, which also proves that there is indeed information in the gradient information that can be used to improve the style transfer quality, which implies where in the first transferred image needs to be modified or updated, and thus can be used for guiding image style transfer. As Figure 6 shown, after performing two-stage style transfer based on the gradient information, compared with the first transferred image obtained by one-stage style transfer, the artificial generation traces at the corresponding positions in the figure are significantly eliminated. From the overall display effect, the image quality of the obtained second transferred image has been greatly improved.

[0076] For another example, as Figure 7 shown, compared with the first transferred image obtained by one-stage style transfer, the second transferred image obtained after two-stage style transfer significantly eliminates the artificial generation traces, making the image look more realistic and natural.

[0077] According to the method for image processing described above, since the gradient guidance network is used to mine the information contained in the gradient information obtained by the discriminator to update the first transferred image, the quality of image style transfer can be improved. Moreover, since this way of updating the transferred image uses the gradient guidance network to obtain the update information instead of directly using the gradient to perform a large number of optimization operations to update the first transferred image, the running speed is faster.

[0078] As mentioned in the background art, in the current image style transfer technology, there is still a problem of poor user experience during the style transfer process. In view of this, according to another embodiment of the present disclosure, a method for image processing is further provided. In this method, the gradient associated with the style transfer loss is controlled by the user to realize the control of the image style transfer process, so that a transferred image satisfactory to the user can be obtained and the user experience is better.

[0079] Figure 8 It is a flowchart of a method for image processing according to another embodiment of the present disclosure. Figure 9 It is a schematic diagram showing a method for image processing according to another embodiment of the present disclosure.

[0080] Referring to Figure 8 , in step S810, an image style transfer model is used to perform style transfer on the input image to obtain a first transferred image. As an example, the image style transfer model may be a generator in an adversarial network. For example, the adversarial network may be a generative adversarial network, but is not limited thereto. For example, as Figure 9 shown, the input image can be input into the generator to obtain the first transferred image. However, the image style transfer model is not limited to the generator, but can be any pre-trained model for image style transfer. When the image style transfer model is the generator of the adversarial network, step S810 may include: encoding the input image using an encoder in the generator to obtain a first latent variable; decoding the first latent variable using a decoder of the generator to obtain the first transferred image.

[0081] In step S820, control information of the user about the image style transfer can be obtained. According to an embodiment, the control information may include information for controlling at least one of the following: the direction of style transfer, the degree of style transfer, the position of style transfer, but is not limited thereto. The present disclosure places no limitation on the way of obtaining the control information. For example, the user can input the control information through a touch operation on the user interface provided by the electronic device, and so on.

[0082] Next, in step S830, gradient information associated with the style transfer loss is obtained according to the control information. According to an embodiment, optionally, step S830 may include adjusting the style transfer loss according to the control information and obtaining gradient information based on the adjusted style transfer loss. For example, the image style transfer model may be a generator in an adversarial network. In this case, the style transfer loss may be obtained by: using a discriminator in the adversarial network to obtain migration quality assessment information, and calculating the style transfer loss according to the migration quality assessment information and the ground truth information. For example, the style transfer loss here may be the style transfer loss loss mentioned in the description regarding Figure 5 above A . The calculation of loss A has been mentioned above and will not be elaborated here.

[0083] Since the style transfer loss is calculated according to the migration quality assessment information and the ground truth information, according to an embodiment, adjusting the style transfer loss according to the control information may include: adjusting the quality migration assessment information and / or the ground truth information according to the control information; obtaining the adjusted style transfer loss based on the adjusted quality migration assessment information and / or the adjusted ground truth information. For example, only the quality migration assessment information may be adjusted. In this case, the adjusted style transfer loss may be obtained based on the adjusted quality migration information and the unadjusted ground truth information. For another example, only the ground truth information may be adjusted. In this case, the adjusted style transfer loss may be obtained based on the unadjusted quality migration information and the adjusted ground truth information. For yet another example, both the quality migration assessment information and the ground truth information may be adjusted. In this case, the adjusted style transfer loss may be obtained based on the adjusted quality migration information and the adjusted ground truth information. Next, gradient information may be obtained based on the adjusted style transfer loss.

[0084] For example, the style transfer loss may be the style transfer loss loss calculated according to the discriminant score map DM A and the ground truth GT mentioned above. A . Accordingly, the style transfer loss loss may be adjusted by adjusting the discriminant score map DM A and / or the ground truth GT A , and then the adjusted style transfer loss may be obtained. The adjusted style transfer loss may be referred to as the style transfer loss loss B .

[0085] Figure 10 and Figure 11 are schematic diagrams of user-guided style enhancement according to embodiments of the present disclosure.

[0086] As Figure 10 shown, by adjusting the discriminant score map DM according to the control informationA , which can enhance the degree of style transfer of specified pixels. For example, the discriminant score map DM can be modified according to the control information A the value at a specific position on. When modifying the discriminant score map DM A the value at a specific position on to obtain the adjusted discriminant score map DM B After that, according to the discriminant score map DM B and the ground truth GT to obtain the adjusted style transfer loss loss B . Then, a backpropagation can be performed once according to the following equation 3 to obtain the gradient G B , as the gradient information mentioned in S830.

[0087]

[0088] Among them, DM B represents the discriminant score map obtained by modifying the discriminant score map DM according to the control information A GT represents the ground truth information, represents the adjusted style transfer loss loss B , L 0 is the first latent variable.

[0089] For example, as Figure 10 shown, the credibility of a specific position on the discriminant score map DM A can be reduced from 1.0 to 0.1. Then, the corresponding style transfer loss loss B at that position will increase. If the loss is positively correlated with the gradient, the gradient at the corresponding position will be enhanced, so that the degree of style transfer of the pixels at the specified position can be enhanced.

[0090] For example, as Figure 11 shown, if the user finds that the degree of style transfer of the pixels at a specific position (e.g., the pixels at the position of the zebra's head) in the first transfer image obtained by performing style transfer on the input image in step S810 should be enhanced, the discriminant score map DM A the credibility at that position can be adjusted, for example, from 1.0 to 0.1, so as to increase the style transfer loss at that position, and then enhance the gradient at that position, so that the degree of style transfer of the pixels at that position can be enhanced.

[0091] Optionally, in addition to adjusting the discriminant score map DM according to the control information AIn addition to controlling the degree of style transfer enhancement based on the credibility of the pixel at this position, the true value of the pixel at this position in the ground truth GT can also be adjusted according to the control information to control the enhancement of the style transfer degree at this position. For example, the true value of the pixel at this position in the ground truth information GT can be increased, making the style transfer loss larger, and then increasing the gradient, so as to enhance the degree of style transfer of the pixel at this position. Or, optionally, both the ground truth information and the discriminant score map can be adjusted to jointly increase the style transfer loss, and then increase the gradient, so as to enhance the degree of style transfer of the pixel at this position.

[0092] Figure 12 and Figure 13 is a schematic diagram of user-guided style weakening according to an embodiment of the present disclosure.

[0093] As Figure 12 shown, by adjusting the ground truth GT according to the control information, the degree of style transfer at the specified position can be weakened. For example, the value at a specific position on the ground truth map with the same size as the discriminant score map DM A can be modified according to the control information to obtain the modified ground truth GT B . After obtaining the adjusted ground truth GT B , the adjusted style transfer loss loss A can be obtained according to the discriminant score map DM B and the ground truth GT B . For example, a backpropagation can be performed once according to the following equation 4 to obtain the gradient G B , as the gradient information mentioned in S830.

[0094]

[0095] wherein, DM A represents the discriminant score map obtained by using the discriminator, GT B represents the ground truth information obtained by modifying the ground truth GT according to the control information,

[0096] represents the adjusted style transfer loss loss B , L 0 represents the first latent variable.

[0097] For example, as Figure 12 shown, the true value of the pixel at a specific position in the ground truth information can be reduced from 1 to 0, then the corresponding style transfer loss loss B at this position will become smaller, and a reduced gradient in the opposite direction to the original gradient can be obtained, so that the degree of style transfer of the pixel at the specific position can be weakened.

[0098] For example, asFigure 13 As shown, if the user finds that the degree of style transfer of the pixels at a specific position in the first transferred image obtained by performing style transfer on the input image in step S810 (e.g., the pixels at the position of the zebra's head) should be weakened, the true value of the pixels at this position in the ground truth GT can be adjusted. For example, it can be reduced from 1 to 0, so as to reduce the style transfer loss at this position, and further reduce the gradient at this position, so that the degree of style transfer of the pixels at this position is weakened.

[0099] Optionally, in addition to adjusting the true value of the pixels at this position in the ground truth GT according to the control information to control the weakening of the style transfer degree, the discriminant score map DM A can also be adjusted according to the control information to control the weakening of the style transfer degree by adjusting the credibility corresponding to the pixels at this position in it. For example, the credibility corresponding to the pixels at this position in the discriminant score map DM A can be increased, so that the style transfer loss is reduced, and further the gradient is reduced, so that the degree of style transfer of the pixels at this position is weakened. Or, optionally, both the ground truth and the discriminant score map can be adjusted to jointly achieve the reduction of the style transfer loss, and further reduce the gradient, so that the degree of style transfer of the pixels at this position is weakened.

[0100] After obtaining the gradient information, in step S840, the first transferred image can be updated based on the gradient information to obtain a second transferred image. According to an embodiment, optionally, step S840 may include: obtaining update information about the first transferred image using an artificial intelligence network based on the gradient information; updating the first transferred image based on the update information to obtain a second transferred image. For example, as described above, the image style transfer model can be the generator of an adversarial network. In step S810, the encoder in the generator can be used to encode the input image to obtain a first latent variable, and the decoder of the generator can be used to decode the first latent variable to obtain a first transferred image. In this case, updating the first transferred image based on the update information to obtain a second transferred image may include: updating the first latent variable based on the update information to obtain a second latent variable; using the decoder to decode the second latent variable to obtain a second transferred image. The artificial intelligence network can be the "gradient guidance network" mentioned above. As Figure 9 and Figure 10 shown, the gradient information can be input into the gradient guidance network, and the gradient guidance network can be used to mine the information contained in the gradient information to obtain the update information of the first transferred image, so that the generator can update the first transferred image according to the update information to obtain a second transferred image that more meets the user's expectations.

[0101] Figure 14 is a schematic diagram of the effect of user-guided style transfer according to an embodiment of the present disclosure. As Figure 14As shown, after obtaining the input image and performing style transfer on the input image using the image style transfer model to obtain the first transferred image, the user can select a specified area in the first transferred image and adjust the degree of style transfer of the specified area. For example, as Figure 14 shown in (a) and (b), the area to be adjusted can be directly circled with a brush in the first transferred image, and the degree of style transfer of this area can be increased by controlling the slider displayed on the first transferred image. For another example, as Figure 14 shown in (c) and (d), the area to be adjusted can be directly circled with a brush in the first transferred image, and the degree of style transfer of this area can be decreased by controlling the slider displayed on the first transferred image. Different positions on the slider can correspond to different degrees of style transfer. According to the input of the user's selected area and adjustment degree, corresponding control information can be generated. As Figure 14 shown, compared with the first transferred image, the artificial generation traces at the corresponding positions in the second transferred image generated by the user's control of style transfer are significantly reduced.

[0102] According to Figure 8 the method for image style transfer shown, after performing style transfer on the input image using the image style transfer model to obtain the first transferred image, due to obtaining the user's control information regarding image style transfer, obtaining gradient information associated with the style transfer loss based on the control information, and updating the first transferred image based on the gradient information to obtain the second transferred image, therefore, the user's control of image style transfer can be achieved by the user's control of the gradient information, enabling the obtained transferred image to satisfy the user and providing a better user experience.

[0103] Although in the above text, the scheme for performing image style transfer based on discriminator guidance ( Figure 3 the method shown) and the scheme for performing image style transfer based on user guidance ( Figure 8 the method shown) are separately described, however, optionally, the two can be combined to improve the quality and running speed of image style transfer while meeting the user's expectations and enhancing the user experience.

[0104] Therefore, user control can be further combined in the Figure 3 method to perform image style transfer. In this case, optionally, Figure 3The method for image style transfer shown may further include: obtaining control information of the user regarding image style transfer. According to an embodiment, the control information may include information for controlling at least one of the following: the direction of style transfer, the degree of style transfer, the position of style transfer, but is not limited thereto. In the case where the control information is obtained, obtaining gradient information associated with the style transfer loss according to the transfer quality evaluation information in step S320 may include: obtaining gradient information according to the transfer quality evaluation information and the control information. Finally, based on the gradient information, update information regarding the first transferred image is obtained by using an artificial intelligence network, and the first transferred image is updated based on the update information to obtain a second transferred image.

[0105] Similarly, in Figure 8 's method, discriminator guidance can be further combined to perform image style transfer. In this case, optionally, the image style transfer model is the generator in the adversarial network, and step S830 may include: obtaining transfer quality evaluation information of the first transferred image by using the discriminator in the adversarial network; obtaining gradient information according to the transfer quality evaluation information and the control information.

[0106] According to an embodiment, in Figure 3 and Figure 8 's alternative of the method, "obtaining gradient information according to the transfer quality evaluation information and the control information" is mentioned. The following introduces specific implementation examples thereof.

[0107] According to an embodiment, obtaining gradient information according to the transfer quality evaluation information and the control information may include: obtaining a first gradient associated with the style transfer loss according to the transfer quality evaluation information; obtaining a second gradient associated with the style transfer loss according to the control information; fusing the first gradient and the second gradient to obtain gradient information. The first gradient is obtained according to the transfer quality evaluation information obtained by using the discriminator, and thus may also be referred to as the "discriminator-guided gradient" hereinafter. The second gradient is obtained according to the user's control information, and thus may also be referred to as the "user-guided gradient" hereinafter. That is to say, in two-stage style transfer, the present disclosure can combine the discriminator-guided gradient and the user-guided gradient to jointly guide image style transfer.

[0108] Figure 15 is another schematic diagram showing two-stage style transfer according to an embodiment of the present disclosure. Figure 16 is a schematic diagram showing an example of the overall framework of the method for image processing according to an embodiment of the present disclosure. Next, with reference to Figure 15 and Figure 16 a further improved two-stage style transfer according to an embodiment of the present disclosure will be described.

[0109] As Figure 15As shown, first, after obtaining the input image A, one-stage style transfer is performed. In the one-stage style transfer, the generator is used to perform style transfer on the input image A to obtain the first transferred image B. 0 For example, the encoder in the generator is used to encode the input image A to obtain the first latent variable L. 0 Then, the decoder of the generator is used to decode the first latent variable L 0 to obtain the first transferred image B. 0 Next, two-stage style transfer is performed. In the two-stage style transfer, as Figure 15 shown, the discriminator-guided gradient G A and the user-guided gradient G B can be calculated respectively. Then, the gradients G A and G B are fused to obtain the fused gradient G C , which serves as the gradient information. After obtaining the fused gradient G C , the generator can obtain the second transferred image B C under the guidance of the fused gradient G 1 . For example, as Figure 16 shown, the gradient-guided network can be used to obtain the update information about the first transferred image B C based on the fused gradient G 0 . Then, the generator uses this update information to update the first transferred image. For example, the first latent variable L 0 is updated based on this update information to obtain the second latent variable L 1 . For example, L 1 = L 0 + U(G c ). Then, the decoder in the generator is used to decode the second latent variable to obtain the second transferred image B 1 .

[0110] As Figure 16 shown, the first transferred image generated by the generator can be input into the discriminator to obtain the transfer quality evaluation information (such as the discrimination information about the image being real / fake). Then, the first gradient associated with the style transfer loss can be obtained according to the transfer quality evaluation information, that is, the gradient G AThe migration quality assessment information may include the credibility information of the first migrated image compared with the real image. According to an embodiment, obtaining the first gradient associated with the style migration loss based on the migration quality assessment information may include: calculating the style migration loss according to the migration quality assessment information and the ground truth information, and obtaining the first gradient based on the style migration loss, where the ground truth information includes information related to the expected credibility of each pixel in the first migrated image. The information related to the expected credibility of each pixel in the first migrated image can be understood as the ground truth information of each pixel. For example, as mentioned above, the discrimination score map DM can be obtained according to the migration quality assessment information A , and calculate the style migration loss loss A according to the discrimination score map DM A and the ground truth GT, and then obtain the gradient G A by performing backpropagation. A As an example, the style migration loss loss A can be the MSE loss, but is not limited thereto. The method of calculating the gradient G Figure 5 can be the same as the method of calculating the gradient G A mentioned in the description above in

[0111] Therefore, it will not be elaborated here. B Next, an introduction will be given on how to obtain the second gradient (i.e., the user-guided gradient G

[0112] According to an embodiment, obtaining the second gradient associated with the style migration loss based on the control information as described above may include: adjusting the style migration loss according to the control information, and obtaining the second gradient based on the adjusted style migration loss. According to an embodiment, adjusting the style migration loss according to the control information may include: adjusting the quality migration assessment information and / or the ground truth information according to the control information; obtaining the adjusted style migration loss based on the adjusted quality migration assessment information and / or the adjusted ground truth information. For example, only the quality migration assessment information can be adjusted. In this case, the adjusted style migration loss can be obtained based on the adjusted quality migration information and the unadjusted ground truth information. For another example, only the ground truth information can be adjusted. In this case, the adjusted style migration loss can be obtained based on the unadjusted quality migration information and the adjusted ground truth information. For yet another example, both the quality migration assessment information and the ground truth information can be adjusted. In this case, the adjusted style migration loss can be obtained based on the adjusted quality migration information and the adjusted ground truth information.

[0113] As mentioned above, the migration quality assessment information may include the credibility information of the first migration image compared with the real image, and the credibility information may include the credibility information corresponding to each pixel in the first migration image. The ground truth information may include information related to the credibility expected for each pixel in the first migration image, such as the ground truth information of each pixel in the first migration image. In this case, by way of example, adjusting the quality migration assessment information and / or the ground truth information according to the control information may include: adjusting the credibility information corresponding to at least one pixel in the first migration image according to the control information; and / or adjusting the ground truth information corresponding to at least one pixel in the first migration image according to the control information.

[0114] For example, the style transfer loss may be the style transfer loss loss calculated according to the discriminant score map DM mentioned above A and the ground truth GT A . Accordingly, the style transfer loss loss can be adjusted by adjusting the discriminant score map DM A and / or the ground truth GT A , and then the adjusted style transfer loss can be obtained. The adjusted style transfer loss can be referred to as the style transfer loss loss B . For example, the discriminant score map DM can be adjusted A to obtain the discriminant score map DM B , and then the gradient G can be obtained according to the discriminant score map DM B and the ground truth GT B . Above, how to adjust the style transfer loss loss by adjusting the discriminant score map DM A and / or the ground truth GT B , and then obtain the gradient G B has been described, and will not be elaborated here. For relevant details, reference can be made to the description of the corresponding content above.

[0115] Next, the fusion of the gradient G A and the gradient G B will be described. According to an embodiment, fusing the first gradient (gradient G A ) and the second gradient (gradient G B ) to obtain gradient information may include: obtaining the weight information of the first gradient and the second gradient according to the control information; and based on the weight information, fusing the first gradient and the second gradient to obtain gradient information.

[0116] Figure 17 is a schematic diagram of gradient fusion according to an embodiment of the present disclosure. As Figure 17 shown, when obtaining the gradient G A according to the discriminant score map DM A and the ground truth GT, and according to the discriminant score map DM BObtain gradient G with the true value GT B After that, the gradient G can be obtained according to the control information A and the gradient G B The corresponding weight W for each A and W B According to the embodiment, the control information may include information for controlling the weights of user guidance and discriminator guidance in the style transfer process. The weight of user guidance may correspond to the gradient G A The weight W of A The weight of discriminator guidance may correspond to the gradient G B The weight W of B Next, gradient fusion can be performed according to the following equation to obtain the fused gradient G C :

[0117] G C =G A *W A +G B *W B Equation 5

[0118] Finally, the information in the fused gradient is mined through the gradient guidance network to update the first transferred image to obtain the second transferred image, that is, the final transferred image.

[0119] By controlling the weights W A and W B The following three image style transfer modes can be achieved:

[0120] - Mode 1: When W A ≠0, W B =0, only use discriminator guidance to refine the style transfer

[0121] (That is, update the first transferred image) to obtain the second transferred image;

[0122] - Mode 2: When W A =0, W B ≠0, only use user guidance to refine the style transfer to obtain the second transferred image;

[0123] - Mode 3: When W A ≠0, W B ≠0, refine the image style transfer by combining discriminator guidance and user guidance to obtain the second transferred image.

[0124] The method for image processing according to the embodiments of the present disclosure can be applied to various scenarios and devices that require image style transfer. Figure 18 is a schematic diagram of the application of the method for image processing according to the embodiments of the present disclosure on a mobile terminal. As Figure 18As shown, the user can first select the target style to which they expect to migrate. After selecting the target style, a one-stage style migration is performed to obtain a rough first migrated image. For example, after the user selects the zebra style, a style migration can be performed on the input horse image to obtain a zebra image. Next, for the obtained rough first migrated image, the area that the user expects to adjust can be further selected. For example, the user can click the selection button below the generated zebra image to select the area to be adjusted. Then, the degree of style migration can be further adjusted through the slider bar displayed on the user interface. For example, the degree of style migration can be increased by swiping to the right. In response to the user releasing the finger touching the slider bar or clicking the corresponding confirmation button, a two-stage style migration can be performed. For example, according to the touch operation of the user on the user interface, the corresponding control information is obtained. Then, as described above, according to the migration quality evaluation information and the control information, the gradient information is obtained. Finally, based on the gradient information, the update information about the first migrated image is obtained by using an artificial intelligence network, and the first migrated image is updated based on the update information to obtain a second migrated image. As Figure 18 shown, compared with the original migrated image, the updated migrated image has a stronger style migration in the area selected by the user, and there are fewer artificial generation traces in the obtained migrated image, making it more realistic and natural.

[0125] Above, a method for image processing according to an embodiment of the present disclosure has been described. Next, an electronic device according to an embodiment of the present disclosure will be briefly described.

[0126] Figure 19 is a block diagram showing an electronic device according to an embodiment of the present disclosure. Referring to Figure 19 , the electronic device 1900 may include a memory 1901 and a processor 1902, where the processor 1902 is coupled to the memory 1901 and is configured to execute any of the methods described above.

[0127] In an embodiment of the present disclosure, an electronic device is also provided. The electronic device includes at least one processor. Optionally, it may further include at least one transceiver and / or at least one memory coupled to the at least one processor. The at least one processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0128] Figure 20 shows a schematic structural diagram of an electronic device applicable to an embodiment of the present invention. As Figure 20 shown, Figure 20The electronic device 4000 shown includes a processor 4001 and a memory 4003. Among them, the processor 4001 and the memory 4003 are connected, such as being connected through a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, and the transceiver 4004 can be used for data interaction between this electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in practical applications, each of the processor 4001, the memory 4003, and the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation to the embodiments of the present disclosure. Optionally, this electronic device may be a first network node, a second network node, or a third network node.

[0129] The processor 4001 may be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in combination with the content disclosed in the present disclosure. The processor 4001 may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0130] The bus 4002 may include a path for transmitting information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture, extended industry standard structure) bus, etc. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 20 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0131] The memory 4003 can be a ROM (Read Only Memory), or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory), or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0132] The memory 4003 is used to store the computer programs or executable instructions for implementing the embodiments of the present disclosure, and is controlled by the processor 4001 for execution. The processor 4001 is used to execute the computer programs or executable instructions stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0133] The embodiments of the present disclosure provide a computer-readable storage medium, on which computer programs or instructions are stored. When the computer programs or instructions are executed by at least one processor, the steps and corresponding contents of the foregoing method embodiments can be executed or implemented.

[0134] The embodiments of the present disclosure further provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0135] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification, claims and the above drawings of the present disclosure are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in words.

[0136] It should be understood that although the flowchart of the embodiments of the present disclosure indicates each operation step by an arrow, the execution order of these steps is not limited to the order indicated by the arrow. Unless there is a clear description in this article, in some implementation scenarios of the embodiments of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage among these sub-steps or stages can also be executed at different times respectively. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present disclosure do not limit this.

[0137] The above text and drawings are provided only as examples to assist the reader in understanding the present disclosure. They are not intended and should not be construed as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art based on the content disclosed herein that changes can be made to the illustrated embodiments and examples without departing from the scope of the present disclosure, and other similar implementation means based on the technical idea of the present disclosure can be adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method for image processing, comprising: performing style transfer on an input image using a generator in an adversarial network to obtain a first transferred image; obtaining transfer quality evaluation information of the first transferred image using a discriminator in the adversarial network, and obtaining gradient information associated with a style transfer loss according to the transfer quality evaluation information; obtaining update information about the first transferred image using an artificial intelligence network based on the gradient information, and updating the first transferred image based on the update information to obtain a second transferred image.

2. The method according to claim 1, further comprising: obtaining control information of a user regarding image style transfer, wherein the obtaining the gradient information associated with the loss of the generator according to the transfer quality evaluation information includes: obtaining the gradient information according to the transfer quality evaluation information and the control information.

3. The method according to claim 1, wherein, the performing style transfer on an input image using a generator in an adversarial network to obtain a first transferred image includes: encoding the input image using an encoder in the generator to obtain a first latent variable; decoding the first latent variable using a decoder of the generator to obtain a first transferred image; wherein the updating the first transferred image based on the update information to obtain a second transferred image includes: updating the first latent variable based on the update information to obtain a second latent variable; decoding the second latent variable using the decoder to obtain a second transferred image.

4. The method according to claim 1, wherein, the transfer quality evaluation information includes credibility information of the first transferred image compared with a real image, the obtaining the gradient information associated with a style transfer loss according to the transfer quality evaluation information includes: calculating a style transfer loss according to the transfer quality evaluation information and ground truth information, and obtaining the gradient information based on the style transfer loss, wherein the ground truth information includes information related to the credibility expected of pixels in the first transferred image.

5. The method according to claim 2, wherein, the control information includes information for controlling at least one of the following: the direction of style transfer, the degree of style transfer, the position of style transfer.

6. A method for image processing, comprising: performing style transfer on an input image using an image style transfer model to obtain a first transferred image; obtaining control information of a user regarding image style transfer; obtaining gradient information associated with a style transfer loss according to the control information; updating the first transferred image based on the gradient information to obtain a second transferred image.

7. The method according to claim 6, wherein, the obtaining gradient information associated with a style transfer loss according to the control information includes: adjusting the style transfer loss according to the control information, and obtaining the gradient information based on the adjusted style transfer loss.

8. The method according to claim 6, wherein, the image style transfer model is a generator in an adversarial network, wherein the obtaining gradient information associated with a style transfer loss based on the control information includes: Obtain the migration quality evaluation information of the first migration image by using the discriminator in the adversarial network; Obtain the gradient information according to the migration quality evaluation information and the control information.

9. The method according to claim 2 or 8, wherein, The obtaining the gradient information according to the migration quality evaluation information and the control information includes: Obtain a first gradient associated with the style transfer loss according to the migration quality evaluation information; Obtain a second gradient associated with the style transfer loss according to the control information; Fuse the first gradient and the second gradient to obtain the gradient information.

10. The method according to claim 9, wherein, The migration quality evaluation information includes the credibility information of the first migration image compared with the real image, wherein, the obtaining the first gradient associated with the style transfer loss according to the migration quality evaluation information includes: Calculate the style transfer loss according to the migration quality evaluation information and the ground truth information, and obtain the first gradient based on the style transfer loss, wherein the ground truth information includes information related to the expected credibility of the pixels in the first migration image.

11. The method according to claim 10, wherein, The obtaining the second gradient associated with the style transfer loss according to the control information includes: Adjust the style transfer loss according to the control information, and obtain the second gradient based on the adjusted style transfer loss.

12. The method according to claim 11, wherein, The adjusting the style transfer loss according to the control information includes: Adjust the quality migration evaluation information and / or the ground truth information according to the control information; Obtain the adjusted style transfer loss based on the adjusted quality migration evaluation information and / or the adjusted ground truth information.

13. The method according to claim 12, wherein, The adjusting the quality migration evaluation information and / or the ground truth information according to the control information includes: Adjust the credibility information corresponding to at least one pixel in the first migration image according to the control information; and / or Adjust the ground truth information corresponding to at least one pixel in the first migration image according to the control information.

14. The method according to claim 9, wherein, The fusing the first gradient and the second gradient to obtain the gradient information includes: Obtain the weight information of the first gradient and the second gradient according to the control information; Based on the weight information, fuse the first gradient and the second gradient to obtain the gradient information.

15. The method according to claim 6, wherein, The updating the first migration image based on the gradient information to obtain a second migration image includes: Obtain the update information about the first migration image by using an artificial intelligence network based on the gradient information; Update the first migration image based on the update information to obtain a second migration image.

16. The method according to claim 15, wherein, The image style transfer model is the generator of a generative adversarial network. The performing of style transfer on an input image by using the image style transfer model to obtain a first transferred image includes: encoding the input image by using an encoder in the generator to obtain a first latent variable; decoding the first latent variable by using a decoder of the generator to obtain a first transferred image; wherein, the updating of the first transferred image based on the update information to obtain a second transferred image includes: updating the first latent variable based on the update information to obtain a second latent variable; decoding the second latent variable by using the decoder to obtain a second transferred image.

17. The method according to claim 6, wherein, the control information includes information for controlling at least one of the following: the direction of style transfer, the degree of style transfer, the position of style transfer.

18. An electronic device, comprising: a memory; a processor, coupled to the memory and configured to execute the method according to any one of claims 1 to 17.

19. A computer-readable storage medium storing instructions, which when run by at least one processor, cause the at least one processor to execute the method according to any one of claims 1 to 17.

Images