Image style migration method and device, equipment, medium and program product

Abstract

The invention discloses an image style migration method, device and equipment, a medium and a program product, and relates to the technical field of financial science and technology. The method comprises the following steps: acquiring a to-be-migrated sample image and a style sample image; inputting the to-be-migrated sample image and the style sample image into an original image generation network to obtain an initial migrated sample image; calculating image content loss, image style loss and image edge loss according to the to-be-migrated sample image, the style sample image and the initial migration sample image; and according to the image content loss, the image style loss and the image edge loss, adjusting the original image generation network to obtain a target image generation network, and carrying out image style migration through the target image generation network. According to the technical scheme of the embodiment of the invention, the image quality of image style migration can be improved, and the requirements of customers on high-quality migration images are met.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to an image style migration method, device, equipment, medium and program product. Background Art

[0002] Image style transfer refers to applying the style of one image to another image to achieve image processing. Through image style transfer, the image to be transferred can retain the original image content while having a unique style, thus achieving a style transformation effect.

[0003] Traditional style transfer algorithms manually analyze image styles and design specific mathematical models to achieve image style transfer. The disadvantage is that they can only extract the underlying features of the image and can only simulate specific styles. With the rapid development of artificial intelligence, image style transfer technology uses deep learning to perform real-time, arbitrary style image style conversion, which has solved the problems in migration speed and the number of styles transferred, but the quality of migration needs to be improved. For example, the image after style transfer is distorted and the edges are deformed. In order to avoid the key information in the original image being too abrupt when performing direct style transfer, some preprocessing is performed to extract contour features, but this will affect the visual effect of the stylized image to a certain extent. Summary of the invention

[0004] The present invention provides an image style transfer method, device, equipment, medium and program product to solve the problem that the existing image style transfer has poor transfer quality and cannot meet the user's demand for high-quality transferred images.

[0005] According to one aspect of the present invention, there is provided an image style transfer method, comprising:

[0006] Obtain sample images to be migrated and style sample images;

[0007] Input the sample image to be migrated and the style sample image into the original image generation network to obtain the initial migration sample image;

[0008] Calculate image content loss, image style loss and image edge loss based on the sample image to be migrated, the style sample image and the initial migration sample image;

[0009] According to the image content loss, image style loss and image edge loss, the original image generation network is adjusted to obtain the target image generation network, so as to perform image style transfer through the target image generation network.

[0010] According to another aspect of the present invention, there is provided an image style transfer apparatus, comprising:

[0011] The original image sample acquisition module is used to obtain the sample images to be migrated and the style sample images;

[0012] A migration sample image generation module is used to input the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image;

[0013] An image loss calculation module, used to calculate image content loss, image style loss and image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image;

[0014] The model parameter adjustment module is used to adjust the original image generation network according to the image content loss, the image style loss and the image edge loss to obtain the target image generation network so as to perform image style transfer through the target image generation network.

[0015] According to another aspect of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image style transfer method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image style transfer method described in any embodiment of the present invention when executed.

[0020] According to another aspect of the present invention, a computer program product is provided, characterized in that the computer program product comprises a computer program, and when the computer program is executed by a processor, the image style transfer method described in any embodiment of the present invention is implemented.

[0021] The technical solution of the embodiment of the present invention obtains the sample image to be transferred and the style sample image, thereby inputting the sample image to be transferred and the style sample image into the original image generation network to obtain the initial transfer sample image, and then calculates the image content loss, image style loss and image edge loss according to the sample image to be transferred, the style sample image and the initial transfer sample image. In this solution, the loss of the initial transfer sample image is evaluated from the three dimensions of image content, style and edge, and the original image generation network is adjusted based on the image loss of the three dimensions. The target image generation network finally obtained can realize the rapid conversion of any style of the image, and can also make the edge of the image content after style transfer consistent with the edge of the original image content, avoiding the deformation of the image edge after style transfer, solving the problem that the transfer quality of the existing image style transfer is poor and cannot meet the user's demand for high-quality transfer images, and can improve the image quality of image style transfer and meet the customer's demand for high-quality style transfer images.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 A flowchart of an image style transfer method provided in Embodiment 1 of the present invention;

[0025] Figure 2 A flowchart of an image style transfer method provided in Embodiment 2 of the present invention;

[0026] Figure 3 A logical schematic diagram of an edge-preserving image style transfer provided in the second embodiment of the present invention;

[0027] Figure 4 A schematic diagram of the structure of an image style transfer device provided in Embodiment 3 of the present invention;

[0028] Figure 5 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Embodiment 1

[0032] Figure 1 This is a flowchart of an image style transfer method provided in the first embodiment of the present invention. This embodiment can be applied to the case of improving the quality of the transferred image. The method can be performed by an image style transfer device. The image style transfer device can be implemented in the form of hardware and / or software. The image style transfer device can be configured in an electronic device. The electronic device can be a computer or a server, etc. Figure 1 As shown, the method includes:

[0033] Step 110: Obtain sample images to be migrated and style sample images.

[0034] The sample images to be transferred may be sample images that carry non-style information (i.e., content information that needs to be retained, referred to as content information) during the image style transfer process. The style sample images may be sample images that carry style information during the image style transfer process. Exemplarily, the sample images to be transferred may be images in the COCO (Common Objects in Context, open source dataset for data recognition) dataset. The style sample images may be images in the WikiArt (encyclopedia of visual arts) dataset.

[0035] In the embodiment of the present invention, sample images to be migrated and style sample images may be obtained from an open source image set.

[0036] Optionally, based on the need to preserve content during image style migration, a sample image to be migrated may be obtained, and a style sample image may be obtained based on the style of the image to be migrated.

[0037] Step 120: input the sample image to be transferred and the style sample image into the original image generation network to obtain an initial transferred sample image.

[0038] The original image generation network may be a network that removes style information from one image and adds style information from another image. The initial migration sample image may be an image obtained by adding style information from the style sample image to the content information in the sample image to be migrated.

[0039] In an embodiment of the present invention, the sample image to be migrated and the style sample image can be input into the original image generation network, so that the sample image to be migrated and the style sample image can be recognized by the original image generation network, and the content information in the sample image to be migrated can be added to the style information of the style sample image to obtain the initial migration sample image.

[0040] Step 130: Calculate image content loss, image style loss and image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image.

[0041] Among them, image content loss can be used to describe the degree of loss of content information of the sample image to be transferred after image style transfer. Image style loss can be used to describe the degree of loss of style information of the style sample image after image style transfer. Image edge loss can be used to describe the degree of loss of image edge of content information of the sample image to be transferred after image style transfer.

[0042] In an embodiment of the present invention, based on a predefined image content loss function, the loss of the content information in the sample image to be migrated of the initial migration sample image can be calculated to obtain the image content loss, and based on a predefined image style loss function, the loss of the style information in the style sample image of the initial migration sample image can be calculated to obtain the image style loss, and based on a predefined image edge loss function, the loss of the graphic edge of the content information in the sample image to be migrated of the initial migration sample image can be calculated to obtain the image edge loss.

[0043] Optionally, the image content loss function, the image style loss function, and the image edge loss function may be known loss calculation functions in the existing field of image processing technology, or may be custom loss calculation functions.

[0044] Exemplarily, the image content loss function and the image style loss function may be loss calculation functions configured in the VGG19 network.

[0045] Step 140: According to the image content loss, the image style loss, and the image edge loss, the original image generation network is adjusted to obtain a target image generation network, so as to perform image style transfer through the target image generation network.

[0046] The target image generation network may be a model obtained by optimizing the original image generation network based on the image loss caused by image style transfer.

[0047] In an embodiment of the present invention, the network parameters of the original image generation network can be adjusted based on the image content loss, image style loss and image edge loss to meet the style transfer quality requirements, and the adjusted original image generation network is used as the target image generation network, and then the image that currently needs to be style transferred and the current style image are input into the target image generation network to automatically perform image style transfer through the target image generation network.

[0048] The technical solution of the embodiment of the present invention obtains the sample image to be transferred and the style sample image, thereby inputting the sample image to be transferred and the style sample image into the original image generation network to obtain the initial transfer sample image, and then calculates the image content loss, image style loss and image edge loss according to the sample image to be transferred, the style sample image and the initial transfer sample image. In this solution, the loss of the initial transfer sample image is evaluated from the three dimensions of image content, style and edge, and the original image generation network is adjusted based on the image loss of the three dimensions. The target image generation network finally obtained can realize the rapid conversion of any style of the image, and can also make the edge of the image content after style transfer consistent with the edge of the original image content, avoiding the deformation of the image edge after style transfer, solving the problem that the transfer quality of the existing image style transfer is poor and cannot meet the user's demand for high-quality transfer images, and can improve the image quality of image style transfer and meet the customer's demand for high-quality style transfer images.

[0049] Embodiment 2

[0050] Figure 2 This is a flowchart of an image style transfer method provided in the second embodiment of the present invention. This embodiment is specific based on the above embodiment and provides a specific optional implementation method for calculating image content loss, image style loss and image edge loss based on the sample image to be transferred, the style sample image and the initial transfer sample image. Figure 2 As shown, the method includes:

[0051] Step 210: Obtain a sample image to be transferred and a style sample image.

[0052] Step 220: Input the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image.

[0053] Step 230: Input the sample image to be migrated, the style sample image and the initial migration sample image into a convolutional neural network to obtain image content features and image style features.

[0054] The image content feature may be an image feature of content information in the sample image to be transferred, and the image style feature may be an image feature of style information in the style sample image.

[0055] In an embodiment of the present invention, based on a convolutional neural network, image content features in the sample images to be migrated and the initial migration sample images are collected, and image style features in the style sample images and the initial migration sample images are collected.

[0056] Optionally, the convolutional neural network in this embodiment can select a trained VGG-19 network (a convolutional neural network with a depth of 19 layers) or the like.

[0057] Step 240: Calculate the image content loss and the image style loss based on the image content features and the image style features respectively.

[0058] In an embodiment of the present invention, the convolutional neural network can calculate the image content loss based on the image content features in the sample image to be migrated and the initial migration sample image, and can also calculate the image style loss based on the image style features in the style sample image and the initial migration sample image.

[0059] In an optional embodiment of the present invention, calculating the image content loss and the image style loss based on the image content features and the image style features, respectively, may include: determining a target feature map of the initial migration sample image based on a convolutional neural network; calculating the image content loss based on a decryptor, an encoder and the target feature map in the convolutional neural network; and calculating the image style loss based on the target feature map and the style sample image.

[0060] The target feature map may be a feature map of an initial migration sample image.

[0061] In an embodiment of the present invention, a target feature map after image style features are migrated to image content features, that is, a target feature map of the initial migration sample image, can be determined based on a convolutional neural network, thereby decoding and encoding the target feature map in sequence based on a decryptor and an encoder in the convolutional neural network, and based on the encoding and decoding results and the target feature map, the image content loss is determined, and then the image style loss is determined according to the mean and variance of the target feature map and the image style features according to each layer of the convolutional neural network. Through a convolutional neural network with an encoder-decoder structure, features can be automatically learned without manually designing a feature extractor, which greatly reduces the cost and difficulty of feature engineering, has image translation invariance, can well identify target objects in images, and can also automatically calculate image content loss and image style loss.

[0062] Step 250: Based on the edge detection network, identify the image edges of the sample image to be migrated and the initial migration sample image to obtain a first grayscale edge image and a second grayscale edge image.

[0063] The first grayscale edge image may be a grayscale edge image of a sample image to be migrated, and the second grayscale edge image may be a grayscale edge image of an initial migration sample image.

[0064] In an embodiment of the present invention, the image edge of the sample image to be migrated can be identified based on an edge detection network to obtain a first grayscale edge image, and the image edge of the initial migration sample image can be identified based on an edge detection network to obtain a second grayscale edge image.

[0065] Step 260: Calculate the image edge loss according to the first grayscale edge image and the second grayscale edge image.

[0066] In the embodiment of the present invention, the image edge loss of the second grayscale edge image compared with the first grayscale edge image may be calculated based on a user-defined image edge loss function.

[0067] In an optional embodiment of the present invention, calculating the image edge loss based on the first grayscale edge image and the second grayscale edge image may include: calculating the pixel difference point by pixel for the first grayscale edge image and the second grayscale edge image to obtain each target pixel difference; and calculating the image edge loss based on each target pixel difference.

[0068] The target pixel difference may be a pixel difference between the first grayscale edge image and the second grayscale edge image at the same pixel point.

[0069] In an embodiment of the present invention, the pixel difference between the first grayscale edge image and the second grayscale edge image at each pixel position can be calculated pixel by pixel to obtain the target pixel difference corresponding to each pixel, and then the square sum of each target pixel difference is used as the image edge loss. In this solution, the pixel difference between the two edge images is calculated pixel by pixel, which can maximize the accuracy of the calculation of the image edge loss and improve the image style transfer effect of the final target image generation network.

[0070] Step 270: According to the image content loss, the image style loss, and the image edge loss, the original image generation network is adjusted to obtain the target image generation network, so as to perform image style transfer through the target image generation network.

[0071] In an optional embodiment of the present invention, adjusting the original image generation network according to image content loss, image style loss and image edge loss to obtain a target image generation network may include: determining target optimization weight parameters of the original image generation network according to the image content loss, image style loss and image edge loss; and updating weight parameters of the original image generation network based on the target optimization weight parameters to obtain the target image generation network.

[0072] The target optimization weight parameter may be an optimized weight parameter determined based on image content loss, image style loss, and image edge loss.

[0073] In an embodiment of the present invention, the network parameters of the original image generation network can be optimized based on the image content loss, image style loss, and image edge loss, and the network parameters that meet the image style transfer quality requirements are used as target optimization weight parameters, and then the target optimization weight parameters are used to update the weight parameters of the original image generation network to obtain the target image generation network. By integrating the loss information of three dimensions (image content loss, image style loss, and image edge loss), the weight parameters of the original image generation network are optimized, which greatly improves the image transfer quality of the target image generation network.

[0074] In an optional embodiment of the present invention, determining the target optimization weight parameters of the original image generation network according to the image content loss, the image style loss and the image edge loss may include: based on the gradient descent method, reversely updating the loss weight parameters of the original image generation network according to the image content loss, the image style loss and the image edge loss to obtain the target optimization weight parameters.

[0075] Among them, the loss weight parameter can be a weight parameter corresponding to the image content loss, the image style loss and the image edge loss.

[0076] In an embodiment of the present invention, image content loss, image style loss and image edge loss can be used as loss feedback, and the gradient descent method can be used to reversely optimize the loss weight parameters of the original image generation network, so as to obtain the final target optimization weight parameters through continuous iterative training. That is, through the gradient descent method, rapid iterative optimization of the original image generation network can be achieved, and the target optimization weight parameters can be determined with higher efficiency.

[0077] In a specific example, see the logic diagram of image style transfer based on edge preservation. Figure 3 ,like Figure 3 The image style transfer logic shown can be applied to the residual banknote recognition scenario. The specific logic is as follows: the sample image to be transferred and the style sample image are input into the original image generation network. The original image generation network can be an encoder-decoder network, in which the encoder is the first few layers (until relu4_1) of the pre-trained VGG-19 network, mainly due to the extraction of image features. After the content information and style information are input into the encoder, the image content features and image style features are obtained respectively, and then the aforementioned features are input into the AdaIN layer, the image content features are transformed, and its style information is removed, and then the style information of the style sample image is added to form a new feature map (i.e., the target feature map), and the new feature map is further input into the decoder to generate an initial migration sample image with content information and style information.

[0078] The AdaIN layer, also called the instance normalization layer, aligns the mean and variance of the image content features of the sample image to be migrated with the mean and variance of the image style features. The decoder is symmetrical with the encoder and is used to transform the image features back to a normal feature space. Through the AdaIN layer, the image generation network based on the encoder-decoder structure can achieve stylization of any style of the image. The mean and variance of the image features are used to represent the style of the image. In the AdaIN layer, the channel-level mean and standard deviation of the image content features of the content information x are matched to the channel-level mean and standard deviation of the image style features of the style information y.

[0079]

[0080] Among them, σ represents the variance, μ represents the mean, x represents the content information of the sample image to be transferred, and y represents the style information of the style sample image.

[0081] like Figure 3As shown, the sample image to be migrated, the style sample image and the initial migration sample image are input into the VGG-19 network, the feature map of the Relu3_3 layer of VGG-19 is selected as the image content feature, and the feature map of the Relu1_2, Relu2_2, Relu3_3 and Relu4_3 layers of VGG19 is selected as the image style feature. The purpose of constructing the VGG-19 network is to input the sample image to be migrated, the style sample image and the initial migration sample image output in the original image generation network into the network respectively to obtain the image content feature and image style feature of the image, calculate the image content loss and image style loss, and reversely update the parameters in the original image generation network in a gradient descent manner.

[0082] The edge detection network is to construct a morphological edge detection method to convert color images into grayscale images, define a 3*3 rectangular structure element, and use the rectangular structure element to perform expansion operation on the grayscale image to obtain A d , use S E Perform corrosion operation on the grayscale image to obtain A e ; Then use S E A d First do the opening operation, then the closing operation, and get image A d1 , use S E A e First do the opening operation, then the closing operation, and get image A e1 ; Finally, image A d1 Subtract image A e1 , get the edge image. The purpose of building the edge detection network is to obtain the edges of the content image and the initial migration sample image generated in the original image generation network through the edge detection network, and define the edge loss, and reversely update the loss weight parameters in the image generation network together with the content loss and style loss in a gradient descent manner.

[0083] In an embodiment of the present invention, the loss function can be defined jointly by the output results of the VGG-19 network and the edge detection network. In the VGG-19 network, image content features and image style features are obtained in a specific layer, so the image content loss function and the image style loss function are defined according to the image content features and the image style features. The content loss is a measure of the difference between the image content features and the features of the image after style transfer. The output t of AdaIN is used as the content target, f represents the encoder, and g represents the decoder. The image content loss function is defined as follows:

[0084] L content =||f(g(t))-t||2; where ‖ ‖2 represents the second norm, L content Indicates the loss of image content.

[0085] The image style loss function is defined as follows:

[0086]

[0087] Among them, φ i represents the i-th layer used to calculate the image style loss in the VGG-19 network. t is the content target of the output of AdaIN, y represents the style information of the style sample image, f represents the encoder, and g represents the decoder.

[0088] In the edge detection network, in order to improve the edge deformation problem of stylized images, an image edge loss function is designed. c and F represent the sample image to be migrated and the initial migration sample image output by the original image generation network, respectively. The sample image to be migrated and the initial migration sample image are input into the edge detection network to obtain the first grayscale edge image E c , and the second grayscale edge image E y The specific definition of the image edge loss function is as follows:

[0089] L edge =∑ i,j (E c (i,j)-E y (i,j)) 2 ;

[0090] Where (i, j) is the pixel point in the grayscale edge image; L edge Represents the image edge loss.

[0091] The total loss consists of image content loss, image style loss and image edge loss. The specific definitions are as follows:

[0092] L total =αL content +βL style +γL edge

[0093] Among them, α, β and γ are the weights of image content loss, image style loss and image edge loss respectively.

[0094] Get the sample images F to be transferred from the training data set c and style sample image F s ; Then input it into the original image generation network to generate the initial migration sample image F; the sample image to be migrated F c , style sample image F s And the initial migration sample image F are input into the VGG-19 network, and the image content loss and image style loss are obtained through the VGG-19 network; the sample image to be migrated F cThe initial migration sample image F is transferred to the edge detection network, and the image edge loss is calculated through the edge detection network. Then, according to the feedback of the loss, the weight of the original image generation network is adjusted. Through continuous iterative training, the target image generation network is finally formed.

[0095] Only the target image generation network is needed for style transfer. When image A needs to generate the style of image B, image A and image B are input into the target image generation network together to quickly generate a stylized image C with the style of B.

[0096] This scheme uses deep learning technology and a designed edge-preserving target image generation network to transfer image style, so as to better preserve image edge information and reduce feature loss during the migration process.

[0097] The technical solution of the embodiment of the present invention obtains a sample image to be migrated and a style sample image, thereby inputting the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image, and then inputting the sample image to be migrated, the style sample image and the initial migration sample image into a convolutional neural network to obtain image content features and image style features, and further respectively calculates image content loss and image style loss based on the image content features and image style features, and identifies image edges of the sample image to be migrated and the initial migration sample image based on an edge detection network to obtain a first grayscale edge image and a second grayscale edge image, thereby calculating image edge loss based on the first grayscale edge image and the second grayscale edge image, and then adjusting the original image generation network based on the image content loss, image style loss and image edge loss to obtain a target image generation network, so as to perform image style migration through the target image generation network.

[0098] In this solution, the image loss is calculated automatically from the three dimensions of image content, style and edge without human intervention, and the original image generation network is adjusted based on the image loss in the three dimensions. The target image generation network finally obtained can realize the rapid conversion of any style of the image, and can also make the edge of the image content after style transfer consistent with the edge of the original image content, avoiding deformation of the image edge after style transfer, and can transfer the style of any image in real time and quickly. The stylized image generated after style transfer maintains good edges and high image quality, which solves the problem of poor migration quality of existing image style transfer and inability to meet users' demand for high-quality migrated images. It can improve the image quality of image style transfer and meet customers' demand for high-quality style transferred images.

[0099] Embodiment 3

[0100] Figure 4 A schematic diagram of the structure of an image style transfer device provided in Embodiment 3 of the present invention.

[0101] like Figure 4 As shown, the device comprises:

[0102] The original image sample acquisition module 310 is used to acquire the sample image to be migrated and the style sample image;

[0103] The migration sample image generation module 320 is used to input the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image;

[0104] An image loss calculation module 330, used to calculate image content loss, image style loss and image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image;

[0105] The model parameter adjustment module 340 is used to adjust the original image generation network according to the image content loss, the image style loss and the image edge loss to obtain the target image generation network, so as to perform image style transfer through the target image generation network.

[0106] The technical solution of the embodiment of the present invention obtains the sample image to be transferred and the style sample image, thereby inputting the sample image to be transferred and the style sample image into the original image generation network to obtain the initial transfer sample image, and then calculates the image content loss, image style loss and image edge loss according to the sample image to be transferred, the style sample image and the initial transfer sample image. In this solution, the loss of the initial transfer sample image is evaluated from the three dimensions of image content, style and edge, and the original image generation network is adjusted based on the image loss of the three dimensions. The target image generation network finally obtained can realize the rapid conversion of any style of the image, and can also make the edge of the image content after style transfer consistent with the edge of the original image content, avoiding the deformation of the image edge after style transfer, solving the problem that the transfer quality of the existing image style transfer is poor and cannot meet the user's demand for high-quality transfer images, and can improve the image quality of image style transfer and meet the customer's demand for high-quality style transfer images.

[0107] Optionally, the image loss calculation module 330 includes a feature extraction unit, a content and style loss calculation unit, an edge image determination unit and an image edge loss calculation unit. The feature extraction unit is used to input the sample image to be migrated, the style sample image and the initial migration sample image into a convolutional neural network to obtain image content features and image style features. The content and style loss calculation unit is used to calculate the image content loss and the image style loss based on the image content features and the image style features, respectively. The edge image determination unit is used to identify the image edges of the sample image to be migrated and the initial migration sample image based on an edge detection network to obtain a first grayscale edge image and a second grayscale edge image. The image edge loss calculation unit is used to calculate the image edge loss based on the first grayscale edge image and the second grayscale edge image.

[0108] Optionally, a content and style loss calculation unit is used to determine a target feature map of the initial migration sample image based on a convolutional neural network; calculate the image content loss according to the decryptor, encoder and the target feature map in the convolutional neural network; and calculate the image style loss according to the target feature map and the style sample image.

[0109] Optionally, the image edge loss calculation unit is specifically used to calculate the pixel difference between the first grayscale edge image and the second grayscale edge image pixel by pixel to obtain each target pixel difference; and calculate the image edge loss according to each target pixel difference.

[0110] Optionally, the model parameter adjustment module 340 is specifically used to determine the target optimization weight parameters of the original image generation network based on the image content loss, the image style loss and the image edge loss; based on the target optimization weight parameters, update the weight parameters of the original image generation network to obtain the target image generation network.

[0111] Optionally, the model parameter adjustment module 340 is specifically used to reversely update the loss weight parameters of the original image generation network based on the gradient descent method according to the image content loss, the image style loss and the image edge loss to obtain the target optimization weight parameters.

[0112] The image style transfer device provided in the embodiment of the present invention can execute the image style transfer method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0113] Embodiment 4

[0114] Figure 5A schematic diagram of an electronic device that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0115] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0116] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as the image style transfer method.

[0118] In some embodiments, the image style transfer method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image style transfer method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the image style transfer method in any other appropriate manner (e.g., by means of firmware).

[0119] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of traditional physical hosts and VPS servers, which are difficult to manage and have weak business scalability.

[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for image style transfer, characterized in that: include: Obtain sample images to be migrated and style sample images; Inputting the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image; Calculating image content loss, image style loss and image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image; According to the image content loss, the image style loss and the image edge loss, the original image generation network is adjusted to obtain a target image generation network, so as to perform image style transfer through the target image generation network.

2. The method according to claim 1, characterized in that The calculating the image content loss, the image style loss and the image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image includes: Inputting the sample image to be migrated, the style sample image and the initial migration sample image into a convolutional neural network to obtain image content features and image style features; Calculating the image content loss and the image style loss based on the image content feature and the image style feature respectively; Based on an edge detection network, identifying image edges of the sample image to be migrated and the initial migrated sample image to obtain a first grayscale edge image and a second grayscale edge image; The image edge loss is calculated according to the first grayscale edge image and the second grayscale edge image.

3. The method according to claim 2, characterized in that The calculating the image content loss and the image style loss based on the image content feature and the image style feature respectively includes: Determining a target feature map of the initial migration sample image based on a convolutional neural network; Calculating the image content loss according to the decryptor, the encoder and the target feature map in the convolutional neural network; The image style loss is calculated according to the target feature map and the style sample image.

4. The method according to claim 2, characterized in that: The calculating the image edge loss according to the first grayscale edge image and the second grayscale edge image includes: Calculating pixel differences pixel by pixel for the first grayscale edge image and the second grayscale edge image to obtain target pixel differences; The image edge loss is calculated according to each of the target pixel differences.

5. The method according to claim 1, characterized in that The adjusting the original image generation network according to the image content loss, the image style loss and the image edge loss to obtain the target image generation network includes: Determining target optimization weight parameters of the original image generation network according to the image content loss, the image style loss, and the image edge loss; Based on the target optimization weight parameters, the weight parameters of the original image generation network are updated to obtain the target image generation network.

6. The method according to claim 5, characterized in that The step of determining the target optimization weight parameters of the original image generation network according to the image content loss, the image style loss, and the image edge loss includes: Based on the gradient descent method, the loss weight parameters of the original image generation network are reversely updated according to the image content loss, the image style loss and the image edge loss to obtain the target optimization weight parameters.

7. An image style transfer device, characterized in that: include: The original image sample acquisition module is used to obtain the sample images to be migrated and the style sample images; A migration sample image generation module, used for inputting the sample image to be migrated and the style sample image into the original image generation network to obtain an initial migration sample image; An image loss calculation module, used to calculate image content loss, image style loss and image edge loss according to the sample image to be migrated, the style sample image and the initial migration sample image; A model parameter adjustment module is used to adjust the original image generation network according to the image content loss, the image style loss and the image edge loss to obtain a target image generation network, so as to perform image style transfer through the target image generation network.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the image style transfer method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image style transfer method according to any one of claims 1 to 6 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the image style transfer method according to any one of claims 1 to 6.

Images