Image Art Style Transfer Method, System, Electronic Device, Storage Medium

Abstract

The present invention provides an image art style transfer method, system, electronic device, and storage medium. The transfer method includes: extracting the content features of a content image and a style image; based on the content features of the content image and the style image, passing the style of the style image to the content image through a style transfer network to form a generated image; wherein, the style transfer network is trained with a contrastive loss based on the contrastive learning theory in machine learning, and the objects of the contrastive loss are the style encodings of the style image, the generated image, and other style art images generated by a multi-layer style projector. The style transfer network trained through contrastive learning can transfer the style of art pictures into realistic pictures to obtain high-quality stylized pictures.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to an image artistic style transfer method, system, electronic device, and storage medium. Background Art

[0002] Artistic style transfer aims to transfer the style from artworks to natural scenes. An ideal artistic style transfer model should capture and transfer the local stroke features and overall appearance of the style image while preserving the structural information of the content image. Traditional style transfer methods such as stroke rendering, image analogy, or image filtering usually use low-level handcrafted features. With the development of deep learning methods, neural network-based methods have been introduced and dominate this field. Image features are obtained by using pre-trained neural networks, and the style is represented as multi-level feature correlations (e.g., Gram matrix, mean, and variance) and an optimization procedure based on style matching is applied. Following this framework, many variants have been proposed based on improvements and modifications in content loss, style loss, and network architecture.

[0003] Although the above methods have achieved reasonable results, there are still obvious artifacts in the stylized output. The main reason is that their style representation and optimization rely on the second-order statistics of features, which have drawbacks in two aspects: First, the second-order statistics can distinguish different styles to a certain extent, but they are not the best way to represent styles (e.g., images of different styles may produce equivalent Gram matrices or variances) because they focus on the distribution of the entire image and ignore the details of objects; Second, arbitrary stylization usually heuristically simulates the transferred style through artificially designed image features and loss functions (e.g., Gram matrix). Summary of the Invention

[0004] The purpose of the present invention is to provide an image artistic style transfer method. Some embodiments can be used to solve the defect that there are still obvious artifacts in the stylized output in the prior art. Instead of only measuring the difference between a single reference image and a stylized output, a large number of available art pictures are utilized to effectively model the distribution of multiple styles and their relationships. To this end, a novel style transfer framework for arbitrary artistic style transfer is proposed based on machine learning methods and deep learning algorithms.

[0005] An image artistic style transfer method provided by the present invention, the transfer method includes:

[0006] Extract the content features of the content image and the style image;

[0007] Based on the content features of the content image and the style image, pass the style of the style image to the content image through a style transfer network to form a generated image;

[0008] Among them, the style transfer network is trained with a contrastive loss designed based on the contrastive learning theory in machine learning. The objects of the contrastive loss are the style maps, generated maps, and style encodings of other style arts generated by a multi-layer style projector. Figure 3 Those of other style arts.

[0009] According to an image art style transfer method provided by the present invention, the multi-layer style projector is trained with a contrastive loss designed based on the contrastive learning theory in machine learning. The objects of the contrastive loss are the style maps generated by the multi-layer style projector, the image enhancement versions corresponding to the style maps, and the style encodings of other style arts. Figure 3 Those of other style arts.

[0010] According to an image art style transfer method provided by the present invention, when the style transfer network is trained, it is constrained by cycle consistency, including:

[0011] Taking the first image as the style map and the second image as the content map, and inputting them into the style transfer network to form a third image;

[0012] Then taking the third image as the content map and the second image as the style map, and inputting them into the style transfer network to obtain a restored fourth image;

[0013] When training the style transfer network, keep the second image and the fourth Figure 1 Consistent.

[0014] According to an image art style transfer method provided by the present invention, when the style transfer network is trained, it is simultaneously constrained by domain enhancement, including:

[0015] Setting a realistic image discriminator and an artistic image discriminator;

[0016] Taking the fifth image as the real sample of the realistic image discriminator and the sixth image as the real sample of the artistic image discriminator;

[0017] Taking the fifth image as the content map and the sixth image as the style map, and inputting them into the style transfer network to form a seventh image, which is used as a fake sample of the artistic image discriminator;

[0018] Taking the fifth image as the style map and the sixth image as the content map, and inputting them into the style transfer network to form an eighth image, which is used as a fake sample of the realistic image discriminator;

[0019] Training the style transfer network so that the artistic image discriminator and the realistic image discriminator cannot determine the authenticity of the generated image.

[0020] According to an image art style transfer method provided by the present invention, based on the content features of the content map and the style map, the style of the style map is transferred to the content map through the style transfer network to form a generated image, including:

[0021] Adopt an adaptive normalization method to align the mean and variance of the content features of the content map with the mean and variance of the content features of the style map;

[0022] Adopt a preset autoencoder to form a generated map based on the content features of the content map after the mean and variance alignment.

[0023] According to an image art style transfer method provided by the present invention, the multi-layer style projector includes: a style feature extractor and a multi-layer projector. The style feature extractor can obtain the style features of an image, and the multi-layer projector converts the style features into style codes.

[0024] The present invention also provides an image art style transfer system, and the transfer system includes:

[0025] An extraction module that extracts the content features of the content map and the style map;

[0026] A transfer module that transfers the style of the style map to the content map based on the content features of the content map and the style map through a style transfer network to form a generated map;

[0027] Among them, the style transfer network is designed and trained with a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style codes of the style map, the generated map, and other style arts generated by the multi-layer style projector Figure 3 and the generated map.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the image art style transfer method described in any one of the above.

[0029] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the image art style transfer method described in any one of the above.

[0030] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the steps of the image art style transfer method described in any one of the above.

[0031] The image art style transfer method, system, electronic device, and storage medium provided by the present invention can transfer the style of an art picture into a realistic picture through a style transfer network trained by contrast learning to obtain a high-quality stylized picture. Description of the Drawings

[0032] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0033] Figure 1 It is a schematic flow chart of the image art style transfer method provided by the present invention;

[0034] Figure 2 It is a schematic structural diagram of the multi-layer style projector provided by the present invention;

[0035] Figure 3 It is a schematic structural diagram of the multi-layer projector provided by the present invention;

[0036] Figure 4 It is a schematic flow chart of optimizing the multi-layer style projector and the style transfer network provided by the present invention;

[0037] Figure 5 It is a schematic structural diagram of the image art style transfer system provided by the present invention;

[0038] Figure 6 It is a schematic physical structure diagram of an electronic device provided by the present invention. Detailed implementation manners

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0040] The following will, in conjunction with the drawings, through specific embodiments and their application scenarios, provide a detailed description of the image art style transfer method provided by the embodiments of the present application.

[0041] Figure 1 It is a schematic flow chart of the image art style transfer method provided by the present invention. As Figure 1 shown, an image art style transfer method provided by the present invention, the transfer method includes:

[0042] Step 101, extract the content features of the content map and the style map;

[0043] Optionally, based on the convolutional neural network in deep learning, extract the content features of the content map and the style map;

[0044] Optionally, the convolutional neural network uses the pre-trained convolutional neural network VGG-19 on the ImageNet database;

[0045] It should be noted that generally, the content map uses a realistic picture, such as a photo; the style map is an artistic picture, such as a painting, that is, the purpose of the transfer can be to make a daily photo into a painting in a certain artistic style;

[0046] Step 102: Based on the content features of the content map and the style map, through the style transfer network, transfer the style of the style map to the content map to form a generated map;

[0047] It should be noted that the principle of transferring the style based on the content features is that the mean and variance of the content features can represent the style to a certain extent, such as color and texture information;

[0048] Among them, the style transfer network is trained by designing a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style map, the generated map, and the style encodings of other style arts generated by the multi-layer style projector Figure 3 of others;

[0049] Optionally, after the style transfer network stylizes the content map according to the style map, a generated map is formed; based on the multi-layer style projector, the style encodings of the style map, the generated map, and other style maps are formed; based on the contrast learning theory in machine learning, the above three style encodings are designed to be contrasted, so that the generated map and the style Figure 1 are consistent;

[0050] Optionally, the generated map and the style map are in an attractive relationship, and the generated map and other style maps are in a repulsive relationship. The visual style of the image is learned by maximizing the mutual information between the generated map and the style map.

[0051] This embodiment adopts deep learning technology, based on the contrast learning theory, represents the style in the form of style encoding and designs a contrast style loss to achieve a better style transfer effect.

[0052] Furthermore, in another embodiment, this embodiment provides an image artistic style transfer method. The multi-layer style projector is trained by designing a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style map, the generated map, and the style encodings of other style arts generated by the multi-layer style projector Figure 3 of others.

[0053] Optionally, the multi-layer style projector includes: a style feature extractor and a multi-layer projector. The style feature extractor can obtain the style features of the image, and the multi-layer projector converts the style features into style encodings.

[0054] Optionally, the style feature extractor is the VGG19 network used for style classification;

[0055] Optionally, the multi-layer projector consists of pooling, convolution, and multiple multi-layer perceptron layers. M layers of features in VGG19 are selected as the input of the multi-layer projector to project the style features into a set of k-dimensional latent style encodings;

[0056] In this embodiment, the multi-layer style projector is trained through contrastive learning to improve the conversion effect of the style encoding of the multi-layer style projector.

[0057] Further, in another embodiment, this embodiment provides an image art style transfer method. When the style transfer network is trained, it is constrained by cycle consistency, including:

[0058] Taking the first image as the style image and the second image as the content image, inputting them into the style transfer network to form a third image;

[0059] Then taking the third image as the content image and the second image as the style image, inputting them into the style transfer network to obtain a restored fourth image;

[0060] When training the style transfer network, keep the second image and the fourth Figure 1 consistent.

[0061] Taking the artistic image A as the style image and the realistic image R as the content image as the input to obtain the generated image G. Conversely, taking the generated image G as the content image and the realistic image R as the style image as the input to obtain the restored realistic image R'. This process is a cycle, and the cycle consistency loss is to constrain R = R'.

[0062] The cycle consistency loss in this embodiment enables the content information of the content image to be maintained during the style transfer process.

[0063] Further, in another embodiment, this embodiment provides an image art style transfer method. When the style transfer network is trained, it is simultaneously constrained by domain enhancement, including:

[0064] Setting up a realistic image discriminator and an artistic image discriminator;

[0065] Taking the fifth image as the real sample of the realistic image discriminator and the sixth image as the real sample of the artistic image discriminator;

[0066] Taking the fifth image as the content image and the sixth image as the style image, inputting them into the style transfer network to form a seventh image, which is used as the fake sample of the artistic image discriminator;

[0067] Taking the fifth image as the style image and the sixth image as the content image, inputting them into the style transfer network to form an eighth image, which is used as the fake sample of the realistic image discriminator;

[0068] Train a style transfer network so that the artistic image discriminator and the photorealistic image discriminator cannot determine whether the generated image is real or fake.

[0069] Optionally, the generative adversarial network consists of a generator G and discriminators D (including D A and D R ), and the generator is the style transfer network. Input the content image and the style image into the generator to obtain the generated image. Label the generated image (generate an artistic image for D A , generate a photorealistic image for D R ) with the "forged" label, and label the real image (artistic image for D A , photorealistic image for D R ) with the "real" label, and send them together to the discriminator D. The goal of the discriminator is to judge the authenticity. The goal of the generator is to make the discriminator unable to judge the authenticity.

[0070] Based on the generative adversarial network in deep learning, this embodiment proposes a domain enhancement module to enhance the brushstrokes and detailed texture of the generated stylized results.

[0071] Furthermore, in another embodiment, this embodiment provides an image artistic style transfer method. The style transfer network transfers the style of the style image to the content image based on the content features of the content image and the style image to form a generated image, including:

[0072] Adopt the adaptive normalization method to align the mean and variance of the content features of the content image with the mean and variance of the content features of the style image;

[0073] Adopt a preset autoencoder to form a generated image based on the content features of the content image after the mean and variance alignment;

[0074] Optionally, the autoencoder includes a decoder for forming a generated image based on the content features of the content image after the mean and variance alignment.

[0075] This embodiment realizes the transfer of color and texture information in the style by aligning the mean and variance of the content features of the content image and the style image.

[0076] Furthermore, in another embodiment, this embodiment provides an image artistic style transfer method, including the following steps:

[0077] Step S1: Extract the content features of any style image and any content image based on the convolutional neural network in deep learning.

[0078] To ensure that the extracted features can better represent the image information, the deep features of the image should be extracted. When depicting scenes with different contents, the painting styles of artists usually vary when using different strokes and colorings. The convolutional neural network VGG-19 pre-trained on the ImageNet database is used to calculate the feature map of the image. For an input image x, F j (x) is the feature map of the j-th convolutional layer of the VGG19 network, with a size of C j ×H j ×W j . Among them, C j is the number of convolutional kernels, H j is the height of the image, and W j is the width of the image. As an object-based representation of the image, the feature map of the VGG19 network pre-trained with object detection in the image contains the content features of the image, and the feature map of the VGG19 network pre-trained with image style classification contains the style features of the image.

[0079] Step S2: Based on the content features, a style transfer network is constructed using the adaptive instance normalization method and the autoencoder structure. According to the content features of the style map and the content map, the style of the style map is transferred to the content map, and a generated map with consistent content and the same style as the style map is obtained according to the transferred features. Figure 1 In the adaptive instance normalization method of the present invention, the process of transferring the style of the style map to the content map is expressed as:

[0080] Let I

[0081]

[0082] I c be the input content map, I s be the reference style map, F j (I c ) h,w,c be the feature map of the input content map, σ(F j (I c ) h,w,c ) be the mean of the feature map of the input content map, σ(F j (I s ) h,w,c ) be the mean of the feature map of the reference style map, μ(F j (I c ) h,w,c ) be the variance of the feature map of the input content map, μ(F j (I s ) h,w,c) is the variance of the feature map of the reference style map. During training, j = 4 is taken. This formula aligns the channel-level mean and standard deviation of the content map with those of the style map. Therefore, the adaptive instance normalization method helps to transfer the style of the style image to the content image.

[0083] Step S3: Based on the convolutional neural network in deep learning, extract the style features of any style map; according to the style features, design a contrastive style loss based on the contrastive learning theory in machine learning, and apply it to the multi-layer style projector and the style transfer network respectively, so that the style of the generated map is consistent with the style. Figure 1 According to the content features, constrain the content features of the generated image to be consistent with the input content through the cyclic consistency loss. Figure 1 Consistent.

[0084] The style information of an image includes its texture, color, and stroke features. The style information of the style map is obtained through the designed multi-layer style projector. Figure 2 This is the structural schematic diagram of the multi-layer style projector provided by the present invention. Figure 3 This is the structural schematic diagram of the multi-layer projector provided by the present invention, as Figure 2 , 3 shown. The multi-layer style projector includes a style feature extractor and a multi-layer projector. Specifically: Collect 18,000 style maps of 30 categories to train the VGG19 network for style classification, and use it as the style feature extractor to extract style features; the multi-layer projector consists of pooling, convolution, and multiple multi-layer perceptron layers. Select the M-layer features in VGG19 as the input of the multi-layer projector to project the style features into a set of k-dimensional latent style encodings. After training, the multi-layer style projector can encode a style map into a set of latent style encodings {z i |i ∈ [1, M], z i ∈ R K}.

[0085] Figure 4 This is the process schematic diagram of optimizing the multi-layer style projector and the style transfer network provided by the present invention, as Figure 4 shown. In order to optimize the contrastive style projector module and the style transfer network, a contrastive style loss is designed based on the contrastive learning theory in machine learning, and it is applied to the training of the multi-layer style projector and the training of the style transfer network respectively.

[0086] When training the multi-layer style projector, send the image I and its enhanced version I + into the style feature extractor of M layers, that is, the VGG-19 network pre-trained for the style classification task. Then send the extracted style features to the multi-layer projector, and the projector is an M-layer neural network that maps the style features to a set of k-dimensional vectors {z}. Compared with the negative samples considered {I- Compared with other style graphs in the dataset of}, the contrast representation learns the visual style of the image by maximizing the mutual information between I and I + . Map the image I, I + and N negative samples to M groups of k-dimensional vectors z, z + ∈R K and {z - ∈R K}. The vectors are normalized to prevent collapse.

[0087] The loss function for training the MSP module can be written as:

[0088]

[0089] where · represents the dot product of two vectors, and τ is the temperature scaling factor, which is set to 0.07 in all experiments. At the same time, a memory bank architecture is used to maintain a large dictionary of negative examples.

[0090] The contrast representation provides appropriate guidance for the style transfer network to transfer style between images. Use the contrast representation of the generated graph I cs = G(I c , I s ) and the style graph I s to calculate the loss, so that I cs will have a style similar to I s :

[0091]

[0092] where and represent the style representations of I cs and I s respectively. The negative examples are sampled from the same dictionary used for training the multi-layer style projector.

[0093] To keep the generated stylized result consistent with the content of the input content graph, use the cycle consistency loss, whose principle is expressed as:

[0094] L cyc = E[‖I c - G(I cs , I c )‖1] + E[‖I s - G(I sc , I s )‖1]

[0095] where I sc = G(I s , I c) Its purpose is to maintain the content information of the content image during the style transfer process between two domains, where E represents expectation.

[0096] Step S4: Design a content map and style map domain enhancement module based on the adversarial generative network in deep learning to enhance the realism of the generated image.

[0097] To learn the distributions of the style map domain and the content map domain, align the distribution of the generated image with that of a specific style map based on the adversarial generative network. Divide the images in the training set into the content map domain and the style map domain, and use discriminators D R and D A for enhancement respectively. During the training process, first randomly select an image from the content map domain as the content map I c , and randomly select an image from the style map domain as the style map I s . I c and I s serve as the real samples of D R and D A respectively. The generated image I cs = G(I c , I s ) serves as the fake sample of D A . Then, swap the content image and the style image, and generate the image I sc = G(I s , I c ) serves as the fake sample of D R . The adversarial loss is expressed as:

[0098] L adv = E[log D R (I c )] + E[log(1 - D R (I cs ))] + E[log D A (I s )] + E[log(1 - D A (I sc ))]

[0099] Step S5: Iterate steps S1 - S4, and finally obtain high-quality stylized results of any style in step S2.

[0100] To evaluate the quality of the generated results in terms of style categories, the deception rate evaluation was introduced. First, a VGG-19 network was trained to classify 10 styles on WikiArt, namely Western classical painting, Neoclassical painting, Impressionist painting, Post-Impressionist painting, semi-abstract, non-figurative, sketch, ink painting, stick figure, and comic. Then, the pre-trained network was used to predict which style the stylized image belongs to. This style classification network was used to classify real images, stylized images generated by the model of this technical solution, and stylized images of seven baseline models. Finally, the deception rate was calculated as the score of the correct artist predicted by the network. As shown in Table 1 below, the deception rates of the model of this technical solution and the seven baseline models are reported in the second column of Table 1. It can be observed that the model of this technical solution achieved the highest deception rate and greatly exceeded other methods. For reference, the average accuracy of the network for real images of artists from WikiArt is 78%.

[0101] Meanwhile, in a user study, this technical solution was compared with seven state-of-the-art style transfer methods in terms of user preference, that is, to determine which method's results are most popular among humans. Specifically, 100 content-style pairs were selected for evaluation (A / B test). For each participant, 50 content-style pairs were selected, and the stylized results of this technical solution method and one of the other methods were shown in random order. Then, the participants were asked to select the image that learned the most features from the style images. Finally, 3000 votes were collected from 60 participants. The voting percentages of each method are reported in the third column of Table 1. Taking the comparison between Linear and this solution as an example, 20% of the users preferred the results of Linear, and 80% of the users preferred the results of this solution. These results indicate that this technical solution method achieved better stylized results.

[0102] In addition, in a user study to quantitatively evaluate stylized images, it is called stylized authenticity detection. Specifically, 25 groups of images were collected, each group containing 10 images with similar styles, including 2 to 4 stylized images generated by this technical solution method or one of the seven baseline methods and some real painting images. The participants were asked to select all the images they thought were fake paintings from each group. Finally, 1875 sets of results from 75 participants were collected, and the accuracy and recall rate of each method were measured. As shown in the fourth and fifth columns of Table 1, the images generated by this technical solution method are least likely to be judged as fake paintings by people.

[0103] Table 1

[0104]

[0105] The following describes the image art style transfer system provided by the present invention. The image art style transfer system described below can be correspondingly referred to the image art style transfer method described above.

[0106] Figure 5 It is a schematic structural diagram of the image art style transfer system provided by the present invention. As Figure 5 shown, the present invention also provides an image art style transfer system. The transfer system includes:

[0107] An extraction module that extracts the content features of the content map and the style map;

[0108] A transfer module that, based on the content features of the content map and the style map, transfers the style of the style map to the content map through a style transfer network to form a generated map;

[0109] Among them, the style transfer network is trained by designing a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style encoding of the style map, the generated map, and other style arts generated by a multi-layer style projector Figure 3 and the generated map.

[0110] Through the style transfer network trained by contrast learning in this embodiment, the style of the art picture can be transferred into the realistic picture to obtain a high-quality stylized picture.

[0111] Figure 6 It is a schematic physical structure diagram of an electronic device provided by the present invention. As Figure 6 shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logical instructions in the memory 830 to execute the image art style transfer method, which includes:

[0112] Extract the content features of the content map and the style map;

[0113] Based on the content features of the content map and the style map, transfer the style of the style map to the content map through a style transfer network to form a generated map;

[0114] Among them, the style transfer network is trained by designing a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style encoding of the style map, the generated map, and other style arts generated by a multi-layer style projector Figure 3 and the generated map.

[0115] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image art style transfer method provided by the above-mentioned various methods. The method includes:

[0117] Extract the content features of the content map and the style map;

[0118] Based on the content features of the content map and the style map, through a style transfer network, transfer the style of the style map to the content map to form a generated map;

[0119] Among them, the style transfer network is designed and trained with a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style map, the generated map, and the style encodings of other style arts generated by a multi-layer style projector. Figure 3 of others.

[0120] On another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the image art style transfer method provided by the above-mentioned various methods. The method includes:

[0121] Extract the content features of the content map and the style map;

[0122] Based on the content features of the content map and the style map, through a style transfer network, transfer the style of the style map to the content map to form a generated map;

[0123] Among them, the style transfer network is designed and trained with a contrast loss based on the contrast learning theory in machine learning. The object of the contrast loss is the style map, the generated map, and the style encodings of other style arts generated by a multi-layer style projector. Figure 3The style encoding of the user.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product, and this computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of each embodiment of the present invention.

Claims

1. An image art style transfer method, characterized in that, The migration method includes: Extracting the content features of the content graph and the style graph; Based on the content features of the content graph and the style graph, passing the style of the style graph to the content graph through a style transfer network to form a generated graph; Among them, the style transfer network is trained by designing a contrast loss based on the contrast learning theory in machine learning, and the object of the contrast loss is the style encodings of the style graph, the generated graph, and other style art graphs generated by a multi-layer style projector; The style transfer network is constrained by cycle consistency during training, including: Taking the first graph as the style graph and the second graph as the content graph, inputting them into the style transfer network to form a third graph; Then taking the third graph as the content graph and the second graph as the style graph, inputting them into the style transfer network to obtain a restored fourth graph; When training the style transfer network, keeping the second graph and the fourth graph consistent; The style transfer network is also constrained by domain enhancement during training, including: Setting a realistic image discriminator and an art image discriminator; Taking the fifth graph as the real sample of the realistic image discriminator and the sixth graph as the real sample of the art image discriminator; Taking the fifth graph as the content graph and the sixth graph as the style graph, inputting them into the style transfer network to form a seventh graph, which is used as the fake sample of the art image discriminator; Taking the fifth graph as the style graph and the sixth graph as the content graph, inputting them into the style transfer network to form an eighth graph, which is used as the fake sample of the realistic image discriminator; Training the style transfer network, and the optimization goal is to make the art image discriminator and the realistic image discriminator unable to determine the authenticity of the generated graph; The process of passing the style of the style graph to the content graph through the style transfer network based on the content features of the content graph and the style graph to form a generated graph includes: Using an adaptive normalization method to align the mean and variance of the content features of the content graph with the mean and variance of the content features of the style graph; Using an autoencoder to form a generated graph based on the content features of the content graph after the mean and variance alignment.

2. The image art style transfer method according to claim 1, wherein The multi-layer style projector is trained by designing a contrast loss based on the contrast learning theory in machine learning, and the object of the contrast loss is the style encodings of the style graph generated by the multi-layer style projector, the image enhancement version of the corresponding style graph, and other style art graphs.

3. The image art style transfer method according to claim 1, wherein The multi-layer style projector includes: a style feature extractor and a multi-layer projector. The style feature extractor is used to obtain the style features of the image, and the multi-layer projector is used to convert the style features into style encodings.

4. An image artistic style transfer system, characterized in that, The migration system includes: An extraction module that extracts the content features of the content graph and the style graph; A migration module that, based on the content features of the content graph and the style graph, passes the style of the style graph to the content graph through a style transfer network to form a generated graph; Among them, the style transfer network is trained by designing a contrast loss based on the contrast learning theory in machine learning, and the object of the contrast loss is the style encodings of the style graph generated by the multi-layer style projector, the generated graph, and other style art graphs; The style transfer network is constrained by cycle consistency during training, including: Take the first figure as the style figure and the second figure as the content figure, and input them into the style transfer network to form a third figure; Then take the third figure as the content figure and the second figure as the style figure, and input them into the style transfer network to obtain a restored fourth figure; When training the style transfer network, keep the second figure and the fourth figure consistent; When the style transfer network is trained, it is simultaneously constrained by domain enhancement, including: Set up a realistic figure discriminator and an artistic figure discriminator; Take the fifth figure as the real sample of the realistic figure discriminator and the sixth figure as the real sample of the artistic figure discriminator; Take the fifth figure as the content figure and the sixth figure as the style figure, and input them into the style transfer network to form a seventh figure, which is used as the fake sample of the artistic figure discriminator; Take the fifth figure as the style figure and the sixth figure as the content figure, and input them into the style transfer network to form an eighth figure, which is used as the fake sample of the realistic figure discriminator; Train the style transfer network, and the optimization goal is to make the artistic figure discriminator and the realistic figure discriminator unable to determine the authenticity of the generated figure; Based on the content features of the content figure and the style figure, through the style transfer network, transfer the style of the style figure to the content figure to form a generated figure, including: Adopt the adaptive normalization method to align the mean and variance of the content features of the content figure with the mean and variance of the content features of the style figure; Adopt an autoencoder to form a generated figure based on the content features of the content figure after the mean and variance alignment.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the image art style transfer method according to any one of claims 1-3.

6. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image art style transfer method according to any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the image art style transfer method according to any one of claims 1-3.

Images