High-quality lifelike image style migration method and system based on composite filtering

Abstract

The invention relates to the technical field of computer image processing, in particular to a high-quality vivid image style migration method based on composite filtering, which comprises the following steps of: semantic segmentation of an image; performing channel-by-channel color migration and boundary feature restoration on the input image and the reference image to obtain a feature restoration image, performing channel-by-channel feature loss estimation on an ILS structure layer of the feature restoration image, converting the ILS structure layer into texture layer feature loss, and performing texture loss compensation to obtain a final ILS texture layer; the structural layer of the feature repairing image and the final ILS texture layer are fused to obtain a high-quality image with vivid image style migration, the problem of contradiction between complete image texture feature keeping and original image color style information removing is solved, input images or reference style images with bright colors are processed in a high-quality mode, and the image quality is improved. And the reference styles containing the complex styles can be migrated to the input image with high quality. The invention further discloses a high-quality vivid image style migration system based on composite filtering.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a high-quality realistic image style transfer method and system based on composite filtering. Background Art

[0002] Realistic image style transfer technology refers to the technology that uses various algorithms to transfer the style (such as color, etc.) of a reference image to an input image while keeping the features of the original image unchanged, and regenerates a realistic image style transfer result image. The essence of realistic image style transfer technology is to transfer the color of a reference image to the input image.

[0003] There are two mainstream technologies in the existing realistic image color transfer technology: one is the image style transfer method based on the loss function of reference style loss and input content feature loss. This type of method often needs to make a trade-off between the saliency of style transfer and feature preservation, resulting in the two being incompatible. The second is to whiten (de-style) the input image, which can also be regarded as a feature extraction process. Then the reference image style is transferred on this basis. This type of method adds a reference style on the basis of feature extraction, which is also prone to feature loss. Therefore, both methods are prone to cause the image style transfer and content feature preservation modules to merge and interfere with each other, resulting in color overflow, feature blur and other problems in the final generated image style transfer results.

[0004] Another method is to decouple the image based on the WLS image filter, perform color migration on the structure layer, and process the feature layer separately. This method can keep the image features unaffected by style migration. However, when the WLS filter extracts image features, the feature layer extracted by the WLS filter usually contains the color information of the image, resulting in incomplete style migration in the final output. The greater the correlation between the input image color and the reference image color, the more serious the phenomenon of incomplete style migration will be. Fig. 9 (c) Fig.10 (d) and Fig.11 (e) as shown. Summary of the invention

[0005] The purpose of the present invention is to provide a high-quality realistic image style transfer method and system based on composite filtering to solve the contradiction between maintaining complete image features and removing the color style information of the original image, and to generate a method for complex color transfer of realistic images without losing or even enhancing the features of the input image.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is:

[0007] A high-quality realistic image style transfer method based on composite filtering comprises the following steps:

[0008] S1. Perform semantic segmentation on the input image and the reference image;

[0009] S2. Decoupling the input image and the reference image using an ILS filter to obtain an ILS structure layer and an ILS texture layer of the input image and the reference image;

[0010] S3. Perform R, G, B channel-by-channel color migration on the ILS structure layer of the reference image in step S2 to the structure layer of the input image to obtain a new ILS structure layer, and combine the new ILS structure layer with the ILS texture layer of the input image to obtain a new image;

[0011] S4. Using the TE module to perform boundary feature repair on the ILS structure layer of the new image in step S3 to obtain the ILS structure layer of the final image;

[0012] S5. Estimating the loss of texture features caused by the restoration of the boundary features in step S4, and converting the loss caused in the structural layer of the final image into the loss of texture features in the texture layer of the final image;

[0013] S6. Performing texture feature loss compensation on the texture layer of the final image in step S5 to obtain an ILS texture layer of the final image;

[0014] S7. Fusing the ILS structure layer of the final image in step S4 with the final ILS texture layer in step S6 to obtain a high-quality image with realistic image style transfer.

[0015] Furthermore, in step S3, by representing C as the input image and R as the reference image, C and R and their semantic segmentation are input into the encoder-decoder, and color migration is performed based on the WCT principle to extract the texture features of C as follows:

[0016]

[0017] Where D C is a covariance matrix t' C (t' C ) T ∈R Ch×Ch Diagonal matrix of eigenvalues; t C is the vectorized VGG feature of C; Ch is the number of channels; E C is the orthogonal matrix corresponding to the eigenvector, then:

[0018] t' C (t' C ) T =E CD C (E C ) T Formula (3.2)

[0019] The color of S is transferred to C:

[0020] t” C =E S (D S ) -1 / 2 (E S ) T t' C . Formula (3.3)

[0021] where t C " is the color migration result; D S is a covariance matrix t' S (t' S ) T ∈R Ch×Ch The diagonal matrix of eigenvalues; E S is the orthogonal matrix corresponding to the eigenvector, satisfying:

[0022] t C ” = t C '+m Formula (3.4)

[0023] Where m is t C "The average value; input t C The decoder obtains the color migration result of the image in the RGB channel;

[0024] In formula (3.1)-formula (3.4), replace C with the structure layer I(C) obtained by filtering the j channel of C with ILS filter j ), replace S with the j channel of S and filter it with ILS filter to obtain the structure layer I(S j ) to obtain the color migration result structure layer S′ of the RGB channel j , where j∈{R,G,B}; the new structure layer is combined with the texture layer of the input image to obtain a new image W.

[0025] Further, in step S4, the boundary feature repair is performed on the structure layer of the new image W as follows:

[0026] W″=W(W)+T(C)=W(W)+CW(C) Formula (4.1)

[0027] Where W′ is the feature restoration result; W(·) represents the WLS filter filtering operation, W(W) represents the structure layer obtained by filtering W with the WLS filter, and T(C) represents the texture layer obtained by filtering C with the WLS filter; the AWLS texture layer of C is:

[0028] T(C)=k1×Ck′1×W(C) Formula (4.2)

[0029] Among them, k1 and k′1 are the weights of C and W(C) respectively;

[0030] According to formula (4.2), for the new image W in step S3, the AWLS texture layer of W is obtained by using the AWLS filter:

[0031] T(W)=k1×Wk′1×W(W) Formula (4.3)

[0032] According to formula (4.2), the ILS texture layer of C without the original image color is obtained through the ILS filter:

[0033] T′(C)=k2×Ck′2×I(C) Formula (4.4)

[0034] Where I(·) represents the ILS filter operation, I(C) and T′(C) represent the structure layer and texture layer obtained by filtering C with the ILS filter, respectively; k2 and k′2 are the weights of C and I(C), respectively, and are set to k2=k′2=1; according to formula (4.3) and formula (4.4), we get:

[0035] T′(W)=k2×Wk′2×I(W) Formula (4.5)

[0036] Where I(W) and T′(W) represent the structure layer and texture layer obtained by filtering W with an ILS filter, respectively;

[0037] According to formula (4.1), when the j channel of S′ is repaired, S′ can be obtained j The repair result is:

[0038] S j =W(I(W j ))+I(C j )-W(I(C j )) Formula (4.6)

[0039] Among them, S j The TE module is the WCT 2 Output the repair result of the ILS structure layer of W in channel j; W(I(W j )) is for W j The structure layer obtained by filtering with ILS filter is then filtered with AWLS filter to obtain the structure layer; I(C j ) represents the structure layer obtained by filtering the j channel of C with an ILS filter; W(I(C j )) indicates the I(Cj ) is filtered by AWLS filter to obtain the structural layer.

[0040] Furthermore, in step S2, the AWLS filter is a filter designed specifically for image style transfer based on the WLS filter.

[0041] The WLS filter controls the degree of image smoothing:

[0042]

[0043]

[0044] In formula (1.1), f is the input image; l is the brightness information of image f; s is the pixel position; ε′ is a very small constant value; α′ is the smoothing weight;

[0045] The ILS filter controls the degree of image smoothing:

[0046]

[0047] Where p is the smoothing factor;

[0048] When ε and ε′ are ignored, we obtain:

[0049]

[0050] The smoothness of the controlled image in the WLS filter is compared with the smoothness of the controlled image in the ILS filter, and by judging the edge-preserving smoothing capabilities of the ILS filter and the WLS filter, the input image and the reference image are decoupled using the ILS filter to obtain an ILS structural layer with a stronger edge-preserving effect and an ILS texture layer with a better effect of removing the color of the input image.

[0051] Furthermore, in step S5, the loss estimate is calculated as follows:

[0052] δ=I(W j )-S j =I(W j )-W(I(W j ))-I(C j )+W(I(C j )) Formula (5.1)

[0053] The loss caused in the structural layer of the final image is converted into the texture feature loss of the final image:

[0054] W′=W(W)+T(C)=WT(W)+T(C) Formula (5.2)

[0055] Among them, WT(W) is the structural layer of W extracted by WLS filter and is obtained according to formula (5.1):

[0056] W(W)+T(W)=I(W)+T′(W) Formula (5.3)

[0057] According to formula (1.3), when ε and ε′ are ignored, the result of filtering the image with the WLS filter is approximately equal to the result of filtering the image with the ILS filter and then with the WLS filter, and according to formula (5.3), we get:

[0058] W(I(W))≈W(W), W(I(C))≈W(C) Formula (5.4)

[0059] Substituting formula (5.4) into formula (5.1) we have:

[0060] δ≈I(W j )-W(W j )-I(C j )+W(C j )

[0061] ≈W j -T`'(W j )-W(W j )-(C j -T'(C j ))+W(C j )

[0062] ≈T(W j )-T'(W j )-T(C j )+T'(C j ) Formula (5.5)

[0063] Among them, δ represents the feature loss of the final image structure layer.

[0064] Furthermore, in step S6, the texture feature loss compensation is performed on the texture of the final image in step S5:

[0065] W′ j Extract the WLS filter characteristics and obtain them through formula (5.1):

[0066] T(W' j )=T(W j -T(9W j )+T(9C j ))

[0067] ≈T(W j )-T(T(9W j ))+T(T(9C j))Formula (6.1)

[0068] According to formula (5.4) and formula (6.1), we can approximate δ and obtain:

[0069] δ'=K1×T(W' j )+K”1×T′(C j ) Formula (6.2)

[0070] Among them, δ' is the compensated texture feature loss, K1 and K'1 are weights respectively. The difference between δ and δ' can be understood as the difference in texture features between the problem boundary and the natural boundary.

[0071] Further, in step S6, the texture layer of the final image is subjected to texture feature loss compensation to obtain an ILS texture layer of the final image;

[0072] F'=K1×T(W' j )+K′1×T′(C j ) Formula (6.3)

[0073] K′1×T′(C j ) is T′(C j ) and T′(C j ) combination; suppose K1 is less than or equal to 0.4, and K′1 is greater than or equal to 0.6.

[0074] Furthermore, in step S7, the calculation method for merging the ILS structure layer of the final image with the ILS texture layer of the final image is:

[0075] O j =γ1S j +γ2F j Formula (7.1)

[0076] Among them, γ1 and γ2 are S j and F j The weight of .

[0077] Further, according to formula (7.1)(6.3)(4.2) we can obtain

[0078]

[0079] Among them, the sum of the weights of the two structural layers is 0, and the weights of other items are 1, then: γ1-γ2K1k′1=0, γ2K1k1+γ2K′1=0, γ1=k′1, γ2=K1=1, k1=0.8, K′1=0.2; by adjusting the values ​​of γ1 and -k′1, different degrees of color migration results can be obtained.

[0080] A realistic image style transfer system based on composite filtering includes a processor and a storage device, wherein the processor is provided with one or more and the storage device is used for storing one or more programs to execute the method of step S1 to step S7.

[0081] The beneficial effects of the present invention are:

[0082] The present invention uses ILS to extract the feature layer of the input image that has fewer features and does not contain color information, and extracts the structural layer that contains most of the image information, performs color migration on the structural layer, and uses WLS to repair the boundary of the structural layer, and estimates and compensates for the feature loss caused by the repair, thereby solving the contradictory problem of removing the color style information of the original image while highly maintaining the original image features, and generating a method for realistic complex image color migration. Compared with existing image style migration methods, the present invention can not only process input images or reference style images with bright colors with high quality, but also migrate reference styles containing complex styles (such as containing multiple colors) to input images with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flow chart of the migration method steps of a preferred embodiment of the present invention.

[0084] Figure 2 It is a migration method algorithm framework diagram of a preferred embodiment of the present invention.

[0085] Figure 3 It is a feature preservation and ECSL algorithm framework diagram of a migration method according to a preferred embodiment of the present invention.

[0086] Figure 4 This is a diagram showing the ablation experiment effect of a migration method according to a preferred embodiment of the present invention.

[0087] Figure 5 This is a comparison diagram of the filtering results of various filters of the migration method of a preferred embodiment of the present invention.

[0088] Figure 6 This is a comparison diagram of the feature preservation and style transfer effects of the migration method of a preferred embodiment of the present invention before and after adding ECSL.

[0089] Figure 7 It is a style transfer effect diagram of different weights of the transfer method according to a preferred embodiment of the present invention.

[0090] Figure 8 This is a color migration effect diagram of different weights of the migration method according to a preferred embodiment of the present invention.

[0091] Fig. 9This is a comparison diagram of a migration method according to a preferred embodiment of the present invention and the effect of solving the problem of incomplete migration.

[0092] Fig.10 This is a schematic diagram of incomplete style transfer caused by the high correlation between the input image color and the reference image color.

[0093] Fig.11 This is a comparison chart of the migration effects of different migration methods. DETAILED DESCRIPTION

[0094] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0096] A high-quality realistic image style transfer method based on composite filtering in a preferred implementation manner of this embodiment includes the following steps:

[0097] S1. Perform semantic segmentation on the input image and the reference image.

[0098] The semantic segmentation method of the image is as follows: the DeepLab semantic segmentation algorithm is used to perform semantic segmentation on the image and the reference style image. There are 150 basic semantic categories, and the merge and simplify method is used to produce streamlined categories. For example, lakes, rivers, oceans, and water flows are classified into one category, thereby generating a set of streamlined categories to produce clearer and simpler segmentation, thereby generating more stable output.

[0099] S2. Decouple the input image and the reference image using an ILS filter to obtain an ILS structure layer and an ILS texture layer of the input image and the reference image.

[0100] In step S2, a weighted least squares filter is used. The weighted least squares filter is an edge-preserving filter whose goal is to make the filtering result as close to the original image as possible, while being as smooth as possible in the area with small gradients and preserving the edges of strong gradients as much as possible. Let the original image be f, and the filtering result to be solved be u, a x 、a yare the weight matrices of the gradients in the x and y directions respectively, then the loss function can be expressed as:

[0101]

[0102] Where f and u are the vector representations of the input image and the image filtering result; s is the pixel position.

[0103] According to formula (1.4):

[0104]

[0105] Where l is the brightness information of image f. The value of α′ is between 1.2 and 2.0. ε′ is a very small constant value. In fact, we can use other matrices to replace the parameter l.

[0106] When L(f) is used to represent a matrix closely related to the source image f, which has the same dimension as f, the solution of formula (1.4) is

[0107] (I+λL(f))u=fFormula (1.6)

[0108] Where u is the image smoothing filter result. The default value of L(f) is log(f). I is the identity matrix. λ is used to balance the data term and the smoothing term.

[0109] The adaptive filter is as follows: When f is C, let L(C) be the sum of the logarithmic values ​​of L(C) (lighting information of C) and C:

[0110] L(C)=α×log(L(C))+(1-α)×log(C) formula (1.7)

[0111] in, ΔL=|mean(L(C))-mean(L(R))|. mean(·) refers to the mean value of the variable. τ is the threshold that controls the use of log(L(C)) and log(C). β is a trigger with a value of 0 or 1. Among them, when ΔL>τ, β=1; ΔL<τ, β=0; let τ=0.5, when f is R, calculate L(R) like calculating L(C).

[0112] The ILS filter energy equation is as follows:

[0113]

[0114]

[0115] Where f and u are vector representations of the input image and the image filtering result, and s is the pixel position. represents the gradient of u in the x and y directions. The smoothing factor p is 0<p<1. ε=0.0001.

[0116] The solution of formula (1.8) is:

[0117]

[0118] Among them, F(·) and F -1 (·) represent FFT and IFFT operations respectively. represents the complex conjugate of F(·),

[0119] Therefore, the degree of image smoothing controlled by the WLS filter is:

[0120]

[0121]

[0122] Among them, f is the input image; l is the brightness information of image f; s is the pixel position; ε′ is a very small constant value; α′ is the smoothing weight;

[0123] The ILS filter controls the degree of image smoothing:

[0124]

[0125] Where p is the smoothing factor;

[0126] Considering that the filtering under L guidance can only obtain information about brightness changes, and L is only a part of the image (because the image is composed of L, A, and B), the image filtering result under L guidance is not as close to the input image as the image filtering result under self guidance. In other words, the ILS result retains more information about the input image than the WLS result. In addition, when the WLS filter is guided by the image itself, when ε and ε′ are ignored, the following is obtained:

[0127]

[0128] From this we get φ p (x)>a x,s , which shows that compared with a in the WLS filter, φ in the ILS p (x) It has a stronger edge-preserving and smoothing effect.

[0129] Therefore, in step S2, the smoothness of the controlled image in the WLS filter is compared with the smoothness of the controlled image in the ILS filter, and the edge-preserving smoothing capabilities of the ILS filter and the WLS filter are judged so as to decouple the input image and the reference image using the ILS filter.

[0130] The smoothness of the controlled image in the WLS filter is compared with that in the ILS filter, and by judging the edge-preserving smoothing capabilities of the ILS filter and the WLS filter, the ILS filter is used to decouple the input image and the reference image to obtain an ILS structural layer with a stronger edge-preserving effect and an ILS texture layer with a better effect of removing the color of the input image.

[0131] S3. Perform R, G, B channel-by-channel color migration on the ILS structure layer of the reference image in step S2 to the structure layer of the input image to obtain a new ILS structure layer, and combine the new ILS structure layer with the ILS texture layer of the input image to obtain a new image.

[0132] In step S3, by representing C as the input image and R as the reference image, C and R and their semantic segmentation are input into the encoder-decoder, and color migration is performed based on the WCT principle to extract the texture features of C as follows:

[0133]

[0134] Where D C is a covariance matrix t' C (t' C ) T ∈R Ch×Ch The diagonal matrix of eigenvalues; t C is the vectorized VGG feature of C; Ch is the number of channels; E C is the orthogonal matrix corresponding to the eigenvector, then:

[0135] t' C (t' C ) T =E C D C (E C ) T Formula (3.2)

[0136] The color of S is transferred to C:

[0137] t” C =E S (D S ) -1 / 2 (E S ) T t' C . Formula (3.3)

[0138] where t C " is the color migration result; D S is a covariance matrix t' S (t' S ) T ∈RCh×Ch The diagonal matrix of eigenvalues; E S is the orthogonal matrix corresponding to the eigenvector, satisfying:

[0139] t C ” = t C '+m Formula (3.4)

[0140] Where m is t C "The average value; input t C The decoder obtains the color migration result of the image in the RGB channel;

[0141] In formula (3.1)-formula (3.4), replace C with the structure layer I(C) obtained by filtering the j channel of C with ILS filter j ), replace S with the j channel of S and filter it with ILS filter to obtain the structure layer I(S j ) to obtain the color migration result structure layer S′ of the RGB channel j , where j∈{R,G,B}; the new structure layer is combined with the texture layer of the input image to obtain a new image W.

[0142] S4. Use the TE module to perform boundary feature repair on the ILS structure layer of the new image in step S3 to obtain the ILS structure layer of the final image. The TE module mainly includes an AWLS filter.

[0143] like Figure 4 As shown, in step S4, a feature repair module is designed to improve the unnatural boundary problem easily generated by the original network structure.

[0144] In step S4, the boundary features of the structure layer of the new image W are repaired as follows:

[0145] W″=W(W)+T(C)=W(W)+CW(C) Formula (4.1)

[0146] Where W′ is the feature restoration result; W(·) represents the WLS filter filtering operation, W(W) represents the structure layer obtained by filtering W with the WLS filter, and T(C) represents the texture layer obtained by filtering C with the WLS filter; the AWLS texture layer of C is:

[0147] T(C)=k1×Ck′1×W(C) Formula (4.2)

[0148] Among them, k1 and k′1 are the weights of C and W(C) respectively;

[0149] According to formula (4.2), for the new image W in step S3, the AWLS texture layer of W is obtained by using the AWLS filter:

[0150] T(W)=k1×Wk′1×W(W) Formula (4.3)

[0151] According to formula (4.2), the ILS texture layer of C without the original image color is obtained through the ILS filter:

[0152] T′(C)=k2×Ck′2×I(C) Formula (4.4)

[0153] Where I(·) represents the ILS filter operation, I(C) and T′(C) represent the structure layer and texture layer obtained by filtering C with the ILS filter, respectively; k2 and k′2 are the weights of C and I(C), respectively, and are set to k2=k′2=1; according to formula (4.3) and formula (4.4), we get:

[0154] T′(W)=k2×Wk′2×I(W) Formula (4.5)

[0155] Where I(W) and T′(W) represent the structure layer and texture layer obtained by filtering W with an ILS filter, respectively;

[0156] According to formula (4.1), when the j channel of S′ is repaired, S′ can be obtained j The repair result is:

[0157] S j =W(I(W j ))+I(C j )-W(I(C j )) Formula (4.6)

[0158] Among them, S j Is the TE module to WCT 2 Output the repair result of the ILS structure layer of W in channel j; W(I(W j )) is for W j The structure layer obtained by filtering with ILS filter is then filtered with AWLS filter to obtain the structure layer; I(C j ) represents the structure layer obtained by filtering the j channel of C with an ILS filter; W(I(C j )) indicates the I(C j ) is filtered by AWLS filter to obtain the structural layer.

[0159] S5. Estimating the loss of texture features caused by the restoration of boundary features in step S4, and converting the loss caused in the structural layer of the final image into the texture feature loss of the texture layer of the final image.

[0160] In step S5, the loss estimate is calculated as:

[0161] δ=I(W j )-S j =I(W j )-W(I(W j ))-I(C j )+W(I(C j )) Formula (5.1)

[0162] The loss caused in the structural layer of the final image is converted into the texture feature loss of the final image:

[0163] W′=W(W)+T(C)=WT(W)+T(C) Formula (5.2)

[0164] Among them, WT(W) is the structural layer of W extracted by WLS filter and is obtained according to formula (5.1):

[0165] W(W)+T(W)=I(W)+T′(W) Formula (5.3)

[0166] According to formula (1.3), when ε and ε′ are ignored, the result of filtering the image with the WLS filter is approximately equal to the result of filtering the image with the ILS filter and then with the WLS filter, and according to formula (5.3), we get:

[0167] W(I(W))≈W(W), W(I(C))≈W(C) Formula (5.4)

[0168] Substituting formula (5.4) into formula (5.1) we have:

[0169] δ≈I(W j )-W(W j )-I(C j )+W(C j )

[0170] ≈W j -T`'(W j )-W(W j )-(C j -T'(C j ))+W(C j )

[0171] ≈T(W j )-T'(W j )-T(C j )+T'(C j ) Formula (5.5)

[0172] Among them, δ represents the feature loss of the final image structure layer.

[0173] In step S6, the texture feature loss compensation is performed on the texture of the final image in step S5:

[0174] W′ j Extract the WLS filter characteristics and obtain them through formula (5.1):

[0175] T(W' j )=T(W j -T(9W j )+T(9C j ))

[0176] ≈T(W j )-T(T(9W j ))+T(T(9C j )) Formula (6.1)

[0177] According to formula (5.4) and formula (6.1), we can approximate δ and obtain:

[0178] δ'=K1×T(W' j )+K”1×T′(C j ) Formula (6.2)

[0179] Among them, δ' is the compensated texture feature loss, K1 and K'1 are weights respectively. The difference between δ and δ' can be understood as the difference in texture features between the problem boundary and the natural boundary.

[0180] S6. Perform texture feature loss compensation on the texture layer of the final image in step S5 to obtain the ILS texture layer of the final image.

[0181] In step S6, the texture layer of the final image is subjected to texture feature loss compensation to obtain the ILS texture layer of the final image;

[0182] F'=K1×T(W' j )+K′1×T′(C j ) Formula (6.3)

[0183] K′1×T′(C j ) is T′(C j ) and T′(C j ) combination; suppose K1 is less than or equal to 0.4, and K′1 is greater than or equal to 0.6.

[0184] According to formula (5.5), I(W j ) indicates WCT 2The structural layer of the j channel component is combined with formula (5.3) to obtain the approximate feature loss caused by feature restoration in step S4. The structural layer of the feature restoration image is extracted by the WLS filter to compensate for the feature loss and obtain T(W″ j ), extract the texture layer of the input image through the ILS filter to obtain T′(C j ), T(W″ j ) and T′(C j ) Combined to obtain the final ILS texture layer F j .

[0185] Figure 6 To increase the comparison of feature preservation and style transfer before and after ECSL.

[0186] The image texture layer extracted by WLS has more information about image brightness changes, while the image texture layer extracted by ILS has almost no brightness information. Therefore, we can obtain the image style transfer results of different brightness by adjusting the weights of K1 and K′1 in the above formula, as follows: Figure 7 shown.

[0187] S7. Fuse the ILS structure layer of the final image in step S4 with the final ILS texture layer in step S6 to obtain a high-quality image with realistic image style transfer.

[0188] In step S7, the calculation method for fusion of the structure layer and the final texture layer of the feature repair image is:

[0189] O j =γ1S j +γ2F j Formula (7.1)

[0190] Among them, γ1 and γ2 are S j and F j The weight of .

[0191] According to formula (7.1)(6.3)(4.2) we can get

[0192]

[0193] Among them, the sum of the weights of the two structural layers is 0, and the weights of other items are 1, then: γ1-γ2K1k′1=0, γ2K1k1+γ2K′1=0, γ1=k′1, γ2=K1=1, k1=0.8, K′1=0.2; by adjusting the values ​​of γ1 and -k′1, different degrees of color migration results can be obtained. Figure 8 shown.

[0194] This embodiment also includes a realistic image style transfer system based on composite filtering, which includes a processor and a storage device, the processor is provided with one or more, and the storage device is used to store one or more programs to execute the method of steps S1 to S7.

Claims

1. A high-quality realistic image style transfer method based on composite filtering, characterized in that: The steps include: S1. Perform semantic segmentation on the input image and the reference image; S2. Decoupling the input image and the reference image using an ILS filter to obtain an ILS structure layer and an ILS texture layer of the input image and the reference image; S3. Perform R, G, B channel-by-channel color migration on the ILS structure layer of the reference image in step S2 to the structure layer of the input image to obtain a new ILS structure layer, and combine the new ILS structure layer with the ILS texture layer of the input image to obtain a new image; S4. Using the TE module to perform boundary feature repair on the ILS structure layer of the new image in step S3 to obtain the ILS structure layer of the final image; S5. Estimating the loss of texture features caused by the restoration of the boundary features in step S4, and converting the loss caused in the structural layer of the final image into the loss of texture features in the texture layer of the final image; S6. Performing texture feature loss compensation on the texture layer of the final image in step S5 to obtain an ILS texture layer of the final image; S7. Fusing the ILS structure layer of the final image in step S4 with the final ILS texture layer in step S6 to obtain a high-quality image with realistic image style transfer.

2. The high-quality realistic image style transfer method based on composite filtering according to claim 1, characterized in that: In step S3, by Denote as input image, R as reference image, With R and its semantic segmentation input encoder-decoder, color transfer is performed based on the WCT principle to extract The texture features are as follows: where Dc is the covariance matrix diagonal matrix of eigenvalues; for Vectorized VGG features; Ch is the number of channels; is the orthogonal matrix corresponding to the eigenvector, then: Will The color of the have: in is the color migration result; is a covariance matrix diagonal matrix of eigenvalues; is the orthogonal matrix corresponding to the eigenvector, satisfying: Where m is The average value of The decoder obtains the color migration result of the image in the RGB channel; In formula (3.1)-formula (3.4), Replace with The structure layer obtained by filtering the j channel with ILS filter Will Replace with The structure layer obtained by filtering the j channel with ILS filter To obtain the color migration result structure layer S′ of the RGB channel j , where j∈{R,G,B}; the new structure layer is combined with the texture layer of the input image to obtain a new image W.

3. The high-quality realistic image style transfer method based on composite filtering according to claim 2, characterized in that: In step S4, the boundary features of the structure layer of the new image W are repaired as follows: Among them, W′ is the feature restoration result; represents the WLS filter filtering operation, represents the structure layer obtained by filtering W with a WLS filter; Express The texture layer obtained by filtering with WLS filter; The texture layers of the AWLS are: Among them, k1 and k′1 are and The weight of According to formula (4.2), for the new image W in step S3, the AWLS texture layer of W is obtained by using the AWLS filter: According to formula (4.2), the ILS filter is used to obtain The ILS texture layer without the original image color has: in, represents the ILS filter filtering operation, and Respectively express The structure layer and texture layer obtained by ILS filter filtering; k2 and k′2 are and The weight of is set to k2=k′2=1; according to formula (4.3) and formula (4.4), we get: in, and They represent the structure layer and texture layer obtained by filtering W with an ILS filter respectively; According to formula (4.1), when the j channel of S′ is repaired, S′ can be obtained j The repair result is: Among them, S j The TE module is the WCT 2 Output the repair result of the ILS structure layer of W in channel j; Yes W j The structural layer obtained by filtering with an ILS filter is then filtered with an AWLS filter to obtain a structural layer; Express The structure layer is obtained by filtering the j channel of the ILS filter; Express The structure layer obtained by AWLS filtering.

4. The high-quality realistic image style transfer method based on composite filtering according to claim 3, characterized in that: In step S2, the AWLS filter is a filter designed specifically for image style migration based on the WLS filter. The WLS filter controls the degree of image smoothing: In formula (1.1), f is the input image; l is the brightness information of image f; s is the pixel position; ε′ is a very small constant value; α′ is the smoothing weight; The ILS filter controls the degree of image smoothing: Where p is the smoothing factor; When ε and ε′ are ignored, we obtain: The smoothness of the controlled image in the WLS filter is compared with the smoothness of the controlled image in the ILS filter, and by judging the edge-preserving smoothing capabilities of the ILS filter and the WLS filter, the input image and the reference image are decoupled using the ILS filter to obtain an ILS structural layer with a stronger edge-preserving effect and an ILS texture layer with a better effect of removing the color of the input image.

5. The high-quality realistic image style transfer method based on composite filtering according to claim 4, characterized in that: In step S5, the loss estimate is calculated as: The loss caused in the structural layer of the final image is converted into the texture feature loss of the final image: in, The WLS filter is used to extract the structural layer of W, and according to formula (5.1) it is obtained: According to formula (1.3), when ε and ε′ are ignored, the result of filtering the image with the WLS filter is approximately equal to the result of filtering the image with the ILS filter and then with the WLS filter, and according to formula (5.3), we get: Substituting formula (5.4) into formula (5.1) we have: Among them, δ represents the feature loss of the final image structure layer.

6. The high-quality realistic image style transfer method based on composite filtering according to claim 5, characterized in that: In step S6, the texture feature loss compensation is performed on the texture of the final image in step S5: W′ j Extract the WLS filter characteristics and obtain them through formula (5.1): According to formula (5.4) and formula (6.1), we can approximate δ and obtain: Among them, δ' is the compensated texture feature loss, K1 and K'1 are weights respectively. The difference between δ and δ' can be understood as the difference in texture features between the problem boundary and the natural boundary.

7. The high-quality realistic image style transfer method based on composite filtering according to claim 6, characterized in that: In step S6, the texture layer of the final image is subjected to texture feature loss compensation to obtain an ILS texture layer of the final image; is the Same as in step 2 Decoupled by the ILS filter combination; suppose K1 is less than or equal to 0.4, and K′1 is greater than or equal to 0.

6.

8. The high-quality realistic image style transfer method based on composite filtering according to claim 7, characterized in that: In step S7, the calculation method for merging the ILS structure layer of the final image with the ILS texture layer of the final image is: O j =γ1S j +γ2F j Formula (7.1) Among them, γ1 and γ2 are S j and F j The weight of .

9. The method for realistic image style transfer based on composite filtering according to claim 8, characterized in that: According to formula (7.1)(6.3)(4.2) we can get Among them, the sum of the weights of the two structural layers is 0, and the weights of other items are 1, then: γ1-γ2K1k′1=0, γ2K1k1+γ2K′1=0, γ1=k′1, γ2=K1=1, k1=0.8, K1′=0.2; by adjusting the values ​​of γ1 and -k′1, different degrees of color migration results can be obtained.

10. A realistic image style transfer system based on composite filtering, characterized by: The method comprises a processor and a storage device, wherein the processor is provided with one or more and the storage device is used for storing one or more programs to execute the method of step S1 to step S7.

Images