An arbitrary style transfer method based on semantic segmentation

Abstract

The present invention provides an arbitrary style transfer method based on semantic segmentation. In view of the problem that style transfer in the prior art can only transfer the style of one photo or one object in a photo, the artistic effect of style transfer is realized by using multiple styles to realize the function of transferring different styles of different types of objects on the same picture, so that different objects in a picture have different style information, making the picture more artistic in the original technology. The present invention solves the problem of selecting different artistic style pictures for different semantic information, and can transfer the styles of different types of objects on the original picture accordingly, so as to achieve different categories in one picture, and fuse different styles of multiple targets at the same time as needed, so as to achieve unexpected artistic effects. The present invention can also express the semantic information of the picture, and is independent of the type of style picture, and does not need to retrain the network for each input.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an arbitrary style migration method based on semantic segmentation. Background Art

[0002] Deep learning has attracted great attention from researchers in the field of image processing due to its high prediction accuracy in recognition applications. Solutions using deep neural networks such as convolutional neural networks (CNNs) can gradually replace traditional image processing based on algorithm descriptions. Although image preprocessing, postprocessing, and signal processing are still performed using existing methods, deep learning has become increasingly important in image classification applications (defect, object, and feature classification). It is easier and more effective to use deep learning to process some tasks, and some tasks can only be solved using deep learning methods. Deep learning is gradually threatening the status of traditional image processing methods, especially when the processing tasks contain complex variables, such as image segmentation, image conversion, and detection. Deep learning has the advantage of translation invariance, which requires a lot of investment using traditional processing methods. Traditional style transfer is only a global transfer of images. It is not possible to achieve simultaneous style transfer of a certain type or several types of objects in an image, so that different objects in an image have different style information. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide an arbitrary style transfer method based on semantic segmentation, which is used to realize the function of transferring different styles of different types of objects on the same picture using multiple styles.

[0004] The technical solution adopted by the present invention to solve the above technical problems is: an arbitrary style transfer method based on semantic segmentation, comprising the following steps:

[0005] S1: Let the source image I be the content image, put the content image and different style images into the style transfer network AdaIN for training, and generate content images I with different styles s1 ,I s2 ,…,I sn ;

[0006] S2: By semantically segmenting the content image, we obtain a mask I with semantic information of different target regions. m1 ,I m2 ,…,I mn ,The number of masks is consistent with the number of content images after migration;

[0007] S3: Using masks with different semantic information I m1 ,I m2 ,…,Imn Processing content image after style transfer I s1 ,I s2 ,…,I sn , get the target area I of style transfer with semantic information r1 ,I r2 ,…,I rn and the background area I of the content image v ; Set the target area I r1 ,I r2 ,…,I rn With background area I v Add together to get a multi-style transfer picture (I r1 +I r2 +…+I rn+ I v ).

[0008] According to the above scheme, in step S1, the specific steps are:

[0009] S11: Use VGG to extract feature information of content image x and style image y before training;

[0010] S12: Input the information source including content image x and style image y to the style transfer network AdaIN, and use the mean and variance of style image y as affine parameters; let the mean of the feature of the input content image be μ(x) and the standard deviation be σ(x), and the mean of the feature of the input style image be μ(y) and the standard deviation be σ(y), and match the channel-level mean and standard deviation of the content image x to the channel-level mean and standard deviation of the style image y:

[0011]

[0012] It means that the content image x is first de-stylized and then stylized to the style of the style image y.

[0013] Furthermore, in step S21, the specific steps are:

[0014] S21: Use the lightweight segmentation model DABNet as the segmentation network;

[0015] S22: Use the semantic image labeled corresponding to the content image x to train the segmentation network and obtain the mask I with semantic information m1 ,I m2 ,…,I mn .

[0016] According to the above scheme, in step S3, the target area I r1 ,I r2 ,…,I rn With background area I v The specific steps of addition are:

[0017] S31: Extract the semantic region image after style fusion, using mask I m1 ,I m2 ,…,I mn Respectively with the content image I after style transfer s1 ,I s2 ,…,I sn Corresponding to the Hadamard product:

[0018] I r1 =I m1 ⊙I s1 ,

[0019] I r2 =I m2 ⊙I s2 ,

[0020] …,

[0021] I rn =I mn ⊙I sn ,

[0022] In the obtained image, only the target area I of the style transfer with semantic information is r1 ,I r2 ,…,I rn Keep the original pixel value, and set the pixel value of other areas to 0;

[0023] S32: Mask I with semantic information m1 ,I m2 ,…,I mn Add and block the target area of ​​the source image I, then perform matrix multiplication with the source image I, let ⊙ be the Hadamard product, and get the background area I with semantic information removed v :

[0024] I v =(1-I m1 -I m2 -…-I mn )⊙I;

[0025] S33: Obtain the final composite image I by adding pixel values f :

[0026] I f =(I r1 +I r2 +…+I rn+ I v ).

[0027] Furthermore, in step S3,

[0028] Target Area I r1 ,I r2 ,…,I rn The pixel value and the style image I after migration s1 ,I s2 ,…,I sn The pixel value of each style-transferred image that does not contain semantic information is 0;

[0029] The pixel value of the target area of ​​the binary image of the mask is 255, which is white, and the pixel value of other areas is 0, which is black.

[0030] A computer storage medium stores a computer program executable by a computer processor, wherein the computer program executes an arbitrary style transfer method based on semantic segmentation.

[0031] The beneficial effects of the present invention are:

[0032] 1. The arbitrary style transfer method based on semantic segmentation of the present invention aims to solve the problem that style transfer in the prior art can only be performed on one photo or one object in a photo. Through the artistic effect of style transfer, the function of using multiple styles to transfer different styles of different types of objects on the same picture is realized, so that different objects in a picture have different style information, making the picture more artistic based on the original technology.

[0033] 2. The present invention solves the problem of selecting different artistic style pictures for different semantic information, so that the styles of different types of objects in the original picture can be transferred accordingly, so as to achieve different categories in one picture. Different styles can be integrated for multiple targets at the same time as needed, which can achieve unexpected artistic effects.

[0034] 3. The present invention can also express the semantic information of the image, and is independent of the type of style image, and does not require retraining the network for each input. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is the content image of the embodiment of the present invention, that is, the original image.

[0036] Figure 2 This is the first style diagram of the embodiment of the present invention.

[0037] Figure 3 It is a second style diagram of the embodiment of the present invention.

[0038] Figure 4 This is the third style diagram of the embodiment of the present invention.

[0039] Figure 5 is the mask I of the category tree of the embodiment of the present invention m1, Mask I of category house m2 and the mask I of the class car m3 picture.

[0040] Figure 6 This is a picture after style transfer is performed between the content picture and the style picture in the embodiment of the present invention.

[0041] Figure 7 It is a migration diagram after mask processing according to an embodiment of the present invention.

[0042] Figure 8 It is the background image after the original image is processed by masking in the embodiment of the present invention.

[0043] Fig. 9 The final effect image is obtained by adding the migration image processed by mask and the background image processed by mask in the embodiment of the present invention.

[0044] Fig.10 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0046] See also Fig.10 , an arbitrary style transfer method based on semantic segmentation according to an embodiment of the present invention comprises the following steps:

[0047] S1: Put the content image and different style images into the style transfer network AdaIN to generate content images of different styles I s1 ,I s2 and I s3 ;

[0048] During training, VGG is first used to extract feature information of the content map and style map. The difference between AdaIN and IN is that the affine parameters come from the samples, that is, the samples as conditions. In AdaIN, there are no affine parameters that need to be learned, that is, the affine parameters are not obtained through learning, which is different from BN, IN, LN, and GN. In the usual fast style transfer network, a model must be retrained for each style map. In style transfer, the style is closely related to the affine parameter IN. AdaIN expands the capabilities of IN and uses the mean and variance of the style image as affine parameters, so given any affine parameters, any style of image can be synthesized.

[0049] After training, a model can support the migration of any style, rather than being limited to a specific style. Different migration images can be obtained by simply passing the content map and style map into AdaIN.

[0050] The method for obtaining content graphs and different style graphs is as follows:

[0051] AdaIN receives two information sources: content input x and style input y. Let μ(x) and σ(x) represent the mean and standard deviation of the features of the input content image, μ(y) and σ(y) represent the mean and standard deviation of the features of the input style image, respectively. According to formula (1), the channel-level mean and standard deviation of x are matched to the channel-level mean and standard deviation of y:

[0052]

[0053] The formula means (1) first de-stylize (subtract its own mean and divide it by its own standard deviation), and then stylize it to the style of the style picture (multiply the standard deviation of the style picture and add the mean).

[0054] S2: The content image is segmented by semantics to obtain a mask I with semantic information of different target regions. m1 ,I m2 ,I m3 ,The number of masks is consistent with the number of content maps after migration;

[0055] The segmentation network is trained by using semantic images corresponding to the content images, and the lightweight segmentation model DABNet is used to obtain the mask I with semantic information through this neural network. m1 ,I m2 ,I m3 .

[0056] S3: Using masks with different semantic information of the target m1 ,I m2 ,I m3 To process the content images of different styles after style transfer I s1 , I s2 , I s3 , get the target area I of style transfer with semantic information r1 , I r2 , I r3 and the background area I of the content image v , and then add the target area and the background area to get a multi-style transfer image (I r1 +I r2 +I r3+ I v ). The specific calculation method for adding the image target area and background area is:

[0057] Extract the semantic region image after style fusion and use the pre-made mask I m1 ,I m2 ,I m3 Respectively with the style transferred picture I s1 , I s2 , I s3Perform Hadamard product as shown in formula (2):

[0058] I r1 =I m1 ⊙I s1

[0059] I r2 =I m2 ⊙I s2 (2)

[0060] I r3 =I m3 ⊙I s3

[0061] The obtained image only retains the original pixel value in the target area, and the pixel values ​​in other areas are 0; the image after semantic segmentation has many areas with 0 pixels (the pixel value of each segmentation migration image without semantic information is 0). Although these areas do not carry information, they have no effect on image fusion. It only needs the pixel value of the target area to be consistent with the pixel value of the migrated style image. The pixel value of the target area of ​​the masked binary image is 255, which is white, and the pixel value of other areas is 0, which is black.

[0062] Then, the target area of ​​the source image is blocked using a mask, the mask with semantic information is added, and matrix multiplication is performed with the source image I. Let ⊙ be the Hadamard product, and the background image with semantic information removed is obtained as I v , the final composite image I is obtained by simply adding pixel values f As shown in (3):

[0063] I v =(1-I m1 -I m2 -I m3 )⊙I (3)

[0064] I f =(I r1 +I r2 +I r3+ I v )

[0065] The present invention is further described below with reference to the accompanying drawings by taking the AdaIN style transfer network and the DABNet semantic segmentation network as examples:

[0066] In one embodiment of the present invention, the two main neural networks are DABNet and AdaIN networks. DABNet is a lightweight segmentation network that builds a deeper network without increasing model parameters. In this way, the optimal is achieved under the same order of magnitude of network parameters. The highlight is the proposal of factored dilated depthwise separable convolution (FDDWC), which actually draws on the previous work of depthwise separable convolution and split convolution to reduce convolution parameters. There are no learnable affine parameters in AdaIN, that is, affine parameters are not obtained through learning. AdaIN receives two information sources: content input x and style input y, and matches the channel-level mean and standard deviation of x to the channel-level mean and standard deviation of y: AdaIN achieves style transfer by changing the data distribution of features at the feature map level. It has low computational and storage overheads and is easy to implement.

[0067] Figure 1 It is a street view image, which contains four categories: buildings, trees, cars, and people. We select buildings, cars, and trees for different style transfers, and use pixels that do not belong to these three categories as the background.

[0068] Figure 2 There are 3 pictures of different styles, which are fused with buildings, trees and cars respectively.

[0069] Figure 3 It is the category mask obtained by semantic segmentation by densely predicting each pixel and predicting the classification, which are tree (I m1 ), House(I m3 ), the car's mask (I m3 ).

[0070] Figure 4 This is the picture after style transfer.

[0071] Figure 5 It is to take the semantic region image after style fusion and use the pre-made mask I m1 ,I m2 ,I m3 Respectively with the style transferred picture I s1 , I s2 , I s3 Figure I obtained by doing Hadamard product r1 , I r2 , I r3 .

[0072] Figure 6 Then, the target area of ​​the source image is blocked using a mask, the mask with semantic information is added, and matrix multiplication is performed with the source image I to obtain the background image I without semantic information. v .

[0073] Figure 7 This is the effect diagram of segmentation and fusion.

[0074] Fig. 9 The final effect image is obtained by adding the migration image processed by the mask and the background image processed by the mask in the embodiment of the present invention.

[0075] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for arbitrary style transfer based on semantic segmentation, characterized by: The following steps are involved: S1: Let the source image I be the content image, put the content image and different style images into the style transfer network AdaIN for training, and generate content images I with different styles s1 ,I s2 ,…,I sn ; S2: By semantically segmenting the content image, we obtain a mask I with semantic information of different target regions. m1 ,I m2 ,…,I mn , the number of masks is consistent with the number of content images after migration; the specific steps are: S21: Use lightweight segmentation model DABNet as the segmentation network; S22: Use the semantic image corresponding to the content image x to train the segmentation network and obtain the mask I with semantic information m1 ,I m2 ,…,I mn ; S3: Using masks with different semantic information I m1 ,I m2 ,…,I mn Processing content image after style transfer I s1 ,I s2 ,…,I sn , get the target area I of style transfer with semantic information r1 ,I r2 ,…,I rn and the background area I of the content image v ; Set the target area I r1 ,I r2 ,…,I rn With background area I v Add together to get a multi-style transfer picture (I r1 +I r2 +…+I rn+ I v ) ; Target area I r1 ,I r2 ,…,I rn With background area I v The specific steps of addition are: S31: Extract the semantic region image after style fusion, using mask I m1 ,I m2 ,…,I mn Respectively with the content image I after style transfer s1 ,I s2 ,…,I sn Corresponding to the Hadamard product: I r1 =I m1 ⊙I s1 , I r2 =I m2 ⊙I s2 , …， I rn =I mn ⊙I sn , In the obtained image, only the target area I of the style transfer with semantic information is r1 ,I r2 ,…,I rn Keep the original pixel value, and set the pixel value of other areas to 0; S32: Mask I with semantic information m1 ,I m2 ,…,I mn Add and block the target area of ​​the source image I, then perform matrix multiplication with the source image I, let ⊙ be the Hadamard product, and get the background area I with semantic information removed v : I v =(1-I m1 -I m2 -…-I mn )⊙I; S33: Obtain the final composite image I by adding pixel values f : I f =(I r1 +I r2 +…+I rn+ I v )。 2. The arbitrary style transfer method based on semantic segmentation according to claim 1, characterized in that: In the step S1, the specific steps are: S11: Use VGG to extract feature information of content image x and style image y before training; S12: Input the information source including content image x and style image y to the style transfer network AdaIN, and use the mean and variance of style image y as affine parameters; let the mean of the feature of the input content image be μ(x) and the standard deviation be σ(x), and the mean of the feature of the input style image be μ(y) and the standard deviation be σ(y), and match the channel-level mean and standard deviation of the content image x to the channel-level mean and standard deviation of the style image y: It means that the content image x is first de-stylized and then stylized to the style of the style image y.

3. The arbitrary style transfer method based on semantic segmentation according to claim 1, characterized in that: In the step S3, Target Area I r1 ,I r2 ,…,I rn The pixel value and the style image I after migration s1 ,I s2 ,…,I sn The pixel values ​​are consistent; The pixel value of each style-transferred image that does not contain semantic information is 0; The pixel value of the target area of ​​the binary image of the mask is 255, which is white, and the pixel value of other areas is 0, which is black.

4. A computer storage medium, characterized in that: A computer program executable by a computer processor is stored therein, and the computer program executes an arbitrary style transfer method based on semantic segmentation as described in any one of claims 1 to 3.

Images