AI generation method of target stylized image

Abstract

The invention discloses an AI generation method for a target stylized image, and the method comprises the steps: firstly introducing a neural arrangement network to rearrange the feature maps of a single style image, enabling the feature maps to adapt to the feature maps of a content image, and generating different styles of contents; measuring the quality of the stylized content by using indexes of a content structure, style perception and artistic aesthetics; comparing the creative stylized image with an output generated by a conventional style migration method; high-quality stylized contents are sampled from different stylized images generated by the random arrangement matrix; a high quality stylized image is used to train PerNet. According to the method, the original content image and the style image are converted into a more creative style migration result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an AI generation method for a target stylized image. Background Art

[0002] Developing computer vision techniques to create human-like visual art is a long-standing goal in the long history of artificial intelligence and art. One of the core features of this endeavor is creativity, which is defined as the generation of novel and valuable artifacts in a process of simultaneous evaluation. Recently, neural style transfer (NST) technology has shown its potential in many successful applications as a means to achieve this goal by transferring the artistic style of one image to another, such as creating artistic images, videos, and 3D objects / scenes. Although NST has produced many impressive visual artworks, it still struggles to create truly creative outputs because these methods merely imitate the style image and lack creativity.

[0003] In terms of style transfer, some works have tried to increase the diversity of styles using different methods. Zixuan Huang et al. introduced multi-level feature concatenation and patch attention to achieve better semantic correspondence and maintain richer style details. Yijun Li et al. proposed a diversity loss to make the feedforward network produce different outputs. Zhizhong Wang proposed deep feature perturbation (DFP) to perturb the deep age feature map using an orthogonal random noise matrix. In addition, Haibo Chen et al.'s diverse image style transfer achieves a reversible cross-space mapping and achieves significant diversity. Spatial control Tai-Yin Chiu et al. transfers different styles to different regions of the content image through masking and segmentation. In addition, as mentioned in the introduction, Songhua Liu et al. proposed style interpolation to interpolate a set of style features and assign corresponding weights to them. Xin Xie et al. proposed stylediscover to also construct a set of style components and linearly combine them to cover new artistic styles. These methods increase the diversity of style transfer by introducing different technical means, but they also have some common shortcomings. First, many methods have high computational complexity and resource requirements, especially when processing large-scale images, which may require a lot of computing resources, affecting the efficiency of practical applications. Second, although they can increase the diversity of styles, there are problems with the stability and quality control of the generated results, especially when random perturbations are involved or multiple outputs are generated, which may lead to unstable image quality or distortion, and it is difficult to strike a balance between style details and content semantics. Finally, some methods rely too much on predefined style components or simple linear combinations, which limits the expressiveness of the style, and are not sophisticated enough in style transitions or detail control, which may cause the generated style effects to be too single or unnatural. Summary of the invention

[0004] The purpose of the present invention is to provide an AI generation method for a target stylized image, generate a new and meaningful artistic style, learn the conversion of style vectors in a supervised manner to a more valuable and surprising style, and achieve a more creative style migration result based on the original content image and style image, which is called creative style transfer (CSFer).

[0005] The technical solution to achieve the purpose of the present invention is: an AI generation method for a target stylized image, comprising the following steps:

[0006] Step 1: Given a content image I C and style image I S , use the convolutional image encoder E to extract the feature f C =E(I C ) and f S =E(Is ), and calculate the features f respectively. C and f S The mean and variance μ of the channel C , σ C and μ S , σ S ;

[0007] Step 2: Randomly sample the two-dimensional permutation matrix P and compare it with the mean and variance μ of the style image. S , σ S Fusion and Then With f C Perform adaptive instance normalization to obtain And input into the decoder D, depending on the generated result For a sampling example, repeat 1000 times to obtain a sampling set

[0008] Step 3: Perform structural similarity screening and style difference screening. First, use SSIM to dynamically calculate the set The similarity between the style information and the content image is then calculated using SPD to obtain the difference between the style information and the content image style and the original output image style.

[0009] Step 4: Perform aesthetic screening and use the aesthetic assessment method SAAN to select Select the image with the highest rating

[0010] Step 5. Repeat steps 1 to 4 to get Create a data set, where O is the output result of the original model, and use the created data set to train the neural replacement network PerNet;

[0011] Step 6: The original output O is extracted by encoder E and the mean is calculated, which is recorded as where μ s = μ(E(O)) and Minimize the loss of supervised training using distance loss

[0012] Step 7: For the new content image and style images Use the neural permutation network PerNet to generate the corresponding permutation matrix P, which is consistent with the new style features. Fusion It is obtained by adaptive normalization with content features and passes through decoder D to generate an artistic image.

[0013] Preferably, step 1 uses relu4_1 of the VGG19 network to extract content images I and C and style image I S The characteristic f C =E(I C ) and f S =E(I S ), which contains content information and style information, and calculates the mean and variance μ of each channel of the feature C =μ(f C ), σ C =ρ(f C ) and μ S =μ(f s ), σ S =ρ(f S ), which represents the style information of the image.

[0014] Preferably, step 2 generates and Sampling method: For the feature f extracted by encoder E S , calculate the mean variance μ of the channel S , σ S , perform matrix multiplication on the randomly generated permutation matrix P to obtain get Represents the new style features obtained by one sampling.

[0015] Preferably, step 3 will collect First pass Select images with similar content information to the content image. The parameter α represents the 30% largest number of structure calculations. This value is obtained through dynamic calculations. σ(φ i (y))|| 2 , calculate the mean and variance of different relu layers of encoder E, using Calculate the sampling image and content image I C The degree of style difference from the original style transfer O, where SPD(x, y) indicates the degree of difference in style between the two images, L is equal to 4, indicating different relu layers, μ, σ indicate mean and variance, β 1 , β 2 Indicates the maximum 50% and 30% of the two image styles, β 1 , β 2 Through dynamic calculation, φ iRepresents different Relu layers of Encoder (specifically Relu1_1, Relu2_1, Relu3_1, Relu41); the final sampling is the set

[0016] Preferably, in step 4, the aesthetic assessment method SAAN is used to select the Select the image with the highest rating

[0017] Preferably, during the training process of step 5, firstly execute steps 1 and 2 to obtain the corresponding mean and variance μ S , σ S , and then through the permutation convolutional neural network PerNet, input μ S , σ S Get the permutation matrix Where φ represents the neural network, τ represents the temperature coefficient, and N r , N c Normalize the rows and columns of the matrix, and add the softmax operation with the temperature coefficient to approach argmax, that is, only one value of the vector is 1, and the rest are 0. Through this operation, the soft permutation matrix PerNet (μ s ;φ).

[0018] Preferably, during the training process of step 5, the soft permutation matrix PerNet (μ s ; φ) and μ S , σ S Perform matrix multiplication to generate That is, the sampled target style vector; and f C Perform adaptive normalization operation to obtain the final image output

[0019] Preferably, The original output O is extracted by encoder E and the mean is calculated, which is recorded as where μ s = μ(E(O)) and Minimize the loss of supervised training using distance loss Where D represents the sampled data set.

[0020] Preferably, step 8 needs to change the soft matrix PerNet (μ s ;φ) is transformed into a hard permutation matrix H-PerNet(μ s ; φ), use the Hungarian algorithm to operate on each row and column of the matrix, and μ S , σ SMatrix multiplication is performed to ensure the invariance of style information in high-dimensional space.

[0021] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method of AI generating artistic style images when executing the program.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method of generating artistic style images by AI.

[0023] Compared with the existing techniques, the significant advantages of our method are: (1) We propose a neural permutation method that enables us to capture the uniqueness and surprising elements inherent in style images by ranking the mean and variance of the feature maps of a single style image. (2) By combining the permutation method, we explore a creative style transfer method to generate novel and meaningful works of art. Our method is the first to efficiently integrate permutation and evaluation to create innovative aesthetic styles. (3) Experiments show that our method outperforms the style images generated by existing style transfer techniques in terms of creativity and diversity, and exhibits strong generalization capabilities while maintaining a simple integration process.

[0024] The present invention is described in further detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the process of the present invention.

[0026] Figure 2 This is the structural diagram of CSFer of the present invention.

[0027] Figure 3 This is a process diagram for sampling of the present invention.

[0028] Figure 4 This is a comparison chart of the effects of the present invention and other style transfer methods.

[0029] Figure 5 This is a test effect diagram of the present invention applied to other style transfer methods.

[0030] Figure 6 This is a diagram showing the effect of applying the present invention to the CAN and Creative Decoder models.

[0031] Figure 7 It is a box plot of the SPD indicator of the present invention in WikiArt and CAN data.

[0032] Figure 8 This is the final sampling result image of the sampling process of the present invention. DETAILED DESCRIPTION

[0033] The present invention proposes an AI generation method for a target stylized image, which can be widely used in the fields of art creation, design, advertising, fashion, social media, etc. by generating a style transfer result that is different from the original style image and has novel and meaningful effects. This technology can combine classic or modern art styles with target content to create unique visual effects and enhance the expressiveness and appeal of the work. Through this method, innovative style generation solutions can be provided for various creative works, promoting artistic creation and content production in various industries.

[0034] Our invention is able to generate and transfer creative artistic styles to content images using only a single style image. This approach is significantly different from previous works that use linear combinations of style components, and we introduce a novel neural permutation method that operates on the features of a single style image, producing meaningful and surprising stylization results that stand out from previous works that often fall on similar style boundaries.

[0035] The present invention considers three core criteria for evaluating creativity: novelty, surprise, and value. By combining style transfer techniques, we quantify the degree of deviation of the new style from a given style to evaluate novelty, and its difference from the original style in terms of unfamiliarity to evaluate surprise. Meanwhile, value evaluates the aesthetic appeal of the new style relative to the original style. Based on these criteria, we design a new and surprising set of features through style transfer, and adjust these features in a single style image to create content with aesthetic style.

[0036] To achieve this goal, we draw inspiration from permutation and combination theory to effectively reconfigure the features of a single style image to generate a diverse set of style features and new style generation. Based on these newly generated styles, we use Style Perceptual Distance (SPD) as a score to evaluate their novelty and surprise. Meanwhile, we calculate aesthetic scores using a specific style art assessment network (SAAN) to evaluate their artistic value. Higher SPD and SAAN scores indicate higher creativity (i.e., novelty, surprise, and aesthetic value). In addition, we determine the most creative output by selecting the generated version with the highest SPD and SAAN scores.

[0037] We develop a neural creative process for style transfer, named Creative Style Transfer (CSFer). Given an artistic style image and a content image, the process is performed alternately to generate new style images that are different from the given style and evaluate their creativity. First, we design a neural permutation network (PerNet) to predict a permutation matrix to sort the mean and variance of the style image feature map, and then adjust the feature map of the content image to generate a new style image. Second, we evaluate the novelty, surprise, and aesthetic value of the newly generated style images using a combination of content structure, style perception, and aesthetic metrics: structural similarity (SSIM), style-perceptual distance (SPD), and style-specific art assessment network (SAAN) scores. These metrics help us highlight the differences between the new style images and traditional style images (such as those generated by conventional neural style transfer methods such as AdaIN) in terms of structural information and aesthetics. Finally, we train PerNet by randomly sampling permutation matrices to meet the criteria of generating novel and aesthetically pleasing style images.

[0038] The present invention first introduces a neural permutation network (PerNet) to rearrange the feature maps of a single style image so that they adapt to the feature maps of the content image, thereby generating content of different styles. In essence, this permutation operation enriches the basis of a single style in a high-dimensional feature space, rather than the traditional linear combination of multiple styles. To measure the quality of the stylized content, we use indicators including content structure, style perception, and artistic aesthetics; these indicators enable us to evaluate creative stylized content by comparing creative stylized images with the outputs produced by traditional style transfer methods; using this evaluation, we sampled high-quality stylized content from different stylized images generated by randomly permuting matrices. Finally, we use this high-quality stylized image to train PerNet. The accompanying figure shows that CSFer can create novel stylized results that are different from the original. In addition, CSFer demonstrates robust generalization capabilities by seamlessly inserting PerNet into existing style transfer methods.

[0039] The present invention enriches the basis of a single style in a high-dimensional feature space by applying permutation operations, which is superior to the traditional linear combination of multiple styles. To measure the quality of stylized images, we use indicators including content structure, style perception, and artistic aesthetics. These indicators enable us to evaluate the stylized content by comparing the stylized images with the outputs produced by traditional style transfer methods. Using this evaluation, we sampled high-quality stylized content from different stylization results generated by randomly permuting matrices. Finally, we used this high-quality stylization result to train PerNet.

[0040] Combination Figure 1 to Figure 3 , an AI generation method for a target stylized image, the specific steps are as follows:

[0041] Step 1: Given a content image I C and style image I S , use the convolutional image encoder E to extract the feature f C =E(I C ) and f S =E(I s ), and calculate the features f respectively. C and f S The mean and variance μ of the channel C , σ C and μ S , σ S , μ ,σ∈R 512 ;

[0042] Step 2: Generate and Sampling method: For the feature f extracted by encoder E S , calculate the mean variance μ of the channel S , σ S , the randomly generated permutation matrix P∈R 512×512 , where each row and column of the permutation matrix has only one 1, and the rest are 0. Perform matrix multiplication to get Then With f C Perform adaptive instance normalization:

[0043]

[0044] get And input into the decoder D, depending on the generated result For a sampling example, repeat 1000 times to obtain a sampling set

[0045] Step 3: Assemble First pass Select images with similar content information to the content image, and then Calculate the mean and variance of different relu layers of encoder E using Calculate the sampling image and content image I C The degree of style difference with the original style migration O is obtained by

[0046] Step 4: Use the aesthetic evaluation method SAAN from the collection Select the image with the highest rating

[0047] Step 5. Repeat steps 1-4 to get several Create a data set and train the neural replacement network. During the training process, first execute steps 1 and 2 to obtain the corresponding mean and variance μ S , σ S , and then through the permutation convolutional neural network PerNet, input μ S , σ S Get the permutation matrix The softmax operation with the temperature coefficient is close to argmax. The same operation is performed on each row and column of the matrix to obtain the soft permutation matrix PerNet (μ s ;φ). The soft permutation matrix PerNet(μ s ; φ) and μ S , σ S Perform matrix multiplication to generate That is, the sampled target style vector. and f C Perform adaptive normalization operation to obtain the final image output

[0048] Step 6: The original output O is extracted by encoder E and the mean is calculated, which is recorded as where μ s = μ(E(O)) and Minimize the loss of supervised training using distance loss

[0049] Step 7: During the test, the soft matrix PerNet (μ s ;φ) is transformed into a hard permutation matrix H-PerNet(μ s ; φ), use the Hungarian algorithm to operate on each row and column of the matrix, and μ S , σ S Matrix multiplication is performed to ensure the invariance of style information in high-dimensional space, and ultimately produce creative generation results.

[0050] Table 1 User survey.

[0051]

[0052] We conducted a user study to evaluate the effectiveness of our CSFer algorithm compared to other methods. The evaluation was divided into three categories, each analyzing from different perspectives: the most popular results among the five NST methods, the creative artistic effect comparison between CAN and CSFer, and the visual effect comparison between CD and CSFer. A total of 338 votes were collected in each category, with a total of 1,014 votes cast by 169 users. The results showed that CSFer received the most votes, with 28.4% of users selecting it as the favorite method, while ArtINS received only 4.7% of the votes. In addition, about 57% of users believed that CSFer provided a more creative style than CAN, and 58% of users believed that CSFer provided a better visual experience than CD. Overall, CSFer became the most popular one among all novel creative methods.

[0053] Table 2 Aesthetic evaluation of style transfer methods on three different style datasets (WikiArt, CAN and CD).

[0054]

[0055]

[0056] We demonstrate the value of CSFer in generating aesthetically meaningful styles by evaluating its aesthetic quality. Two methods are used for the evaluation: SAAN and TANet, and the results are presented in the corresponding tables. The results show that CSFer's styles outperform the current state-of-the-art methods (SOTA) on both SAAN and TANet scores, except on the CD dataset of SAAN. For example, IEC+CSFer and MST+CSFer achieve the highest SAAN scores of 4.782 and 4.789 on the WikiArt and CAN datasets, respectively. Similarly, LST+CSFer and IEC+CSFer achieve the highest TANet scores of 5.216, 5.261, and 4.986 on the WikiArt, CAN, and CD datasets, respectively. In summary, CSFer is not only able to improve or match the performance of AdaIN, MST, LST, and ArtINS, but also to create valuable styles.

[0057] Figure 4Various visual images of style transfer achieved by different methods are as follows. We have the following four observations. First, the AdaIN method transfers the artistic style of the style image to the content image, and the generated style is similar to the given artistic style, but lacks diversity, not to mention creativity. Second, although the DFP method generates diverse styles, it is still closer to the given style due to the small perturbation applied. When using larger perturbations, DFP may not be able to successfully transfer the artistic style. Third, the style generated by the ArtINS method is more diverse than AdaIN, LST, and DFP, but the cardinality of its independent components is about 10, which is somewhat limited. ArtINS also requires manual setting of the parameters of the linear combination. Finally, unlike these methods, our CSFer method generates novel and meaningful artistic styles through the permutation and combination of neural networks, which is able to create new style bases. Even when applied to ArtINS, ArtINS+CSFer can generate richer and more creative results.

[0058] Figure 5 The images generated by CSFer combined with LST, IEC, and MST are shown, indicating that CSFer can combine other methods to enhance their visual generation effects. Combined with Table 2, the aesthetic evaluation of their effects is higher than the original generation results.

[0059] Figure 6 We compare the artistic styles generated by CAN, Creative Decoder (CD), and our proposed CSFer method. To ensure a fair comparison, we apply each style to the same content using AdaIN. Our scheme generates novel styles with mixed colors, which is comparable to CAN and Creative Decoder (CD). e Decoder, with significant improvements. For example, in the fourth column, CSFer naturally creates a mixed green and red style that is visually better than the black and yellow style generated by CAN+AdaIN. Similarly, in the eighth column, CSFer generates a mixed green and yellow style that is more attractive than the yellow style of CD+AdaIN.

[0060] Figure 7 The results generated by AdaIN, MST, LST and ArtINS methods are compared with those generated by CSFer. Greater than the result SPD(O,I S ). Therefore, CSFer increases the degree of style differentiation and creates new style features.

[0061] Figure 8Figure 2 shows some examples of sampled images from the dataset, from left to right, they are represented as content images, segmented images, original results, and sampled result images. We determined the formula based on 200 pairs of style and content images.

[0062]

[0063] The values ​​of the three parameters α, β1 and β2 in . In order to generate the image set for each pair of images We have S , σ S ) were randomly permuted 1000 times. We chose to set the parameters to α = 0.2, β1 = 0.5, and β2 = 0.3. These parameter values ​​were chosen because they performed well in the average aesthetic assessment SNNA score. Using these parameter settings, we obtained a subset of 10 images, denoted as And determine an optimal image

[0064] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI generation method for a target stylized image, characterized in that: The steps include: Step 1: Given a content image I C and style image I S , use the convolutional image encoder E to extract the feature f C =E(I C ) and f S =E(I s ), and calculate the features f respectively C and f S The mean and variance μ of the channel C ,σ C and μ S ,σ S ; Step 2: Randomly sample the two-dimensional permutation matrix P and compare it with the mean and variance μ of the style image. S ,σ S Fusion and Then With f C Perform adaptive instance normalization to obtain And input into the decoder D, depending on the generated result For a sampling example, repeat 1000 times to obtain a sampling set Step 3: Perform structural similarity screening and style difference screening. First, use SSIM to dynamically calculate the set The similarity between the style information and the content image is then calculated using SPD to obtain the difference between the style information and the content image style and the original output image style. Step 4: Perform aesthetic screening and use the aesthetic assessment method SAAN to select Select the image with the highest rating Step 5. Repeat steps 1 to 4 to get Create a data set, where O is the output result of the original model, and use the created data set to train the neural replacement network PerNet; Step 6: The original output O is extracted by encoder E and the mean is calculated, which is recorded as where μ s = μ(E(O)) and Minimize the loss of supervised training using distance loss Step 7: For the new content image and style images Use the neural permutation network PerNet to generate the corresponding permutation matrix P, which is consistent with the new style features. Fusion It is obtained by adaptive normalization with content features and passes through decoder D to generate an artistic image.

2. The method for generating an artistic style image by AI according to claim 1, characterized in that: Step 2 is generated by sampling and The specific method is as follows: For the feature f extracted by encoder E S , calculate the mean variance μ of the channel S ,σ S , perform matrix multiplication on the randomly generated permutation matrix P to obtain 3. The AI ​​generation method of a target stylized image according to claim 2, characterized in that: Step 3: Assemble First, by the formula Select images with similar structural information to the content image, and the parameter α represents the 30% largest number of structure calculations. Then, Calculate the mean and variance of different relu layers of encoder E using Calculate the sampling image and content image I C The degree of style difference from the original style transfer O, where SPD(x,y) indicates the degree of difference in style between the two images, L indicates different relu layers, μ, σ indicates mean and variance, β1, β2 indicates the maximum 50% and 30% of the styles of the two images, φ i Represents the different Relu layers of the Encoder, specifically Relu1_1, Relu2_1, Relu3_1, Relu4_1; the final sampling is the set 4. The AI ​​generation method of a target stylized image according to claim 3, characterized in that: In step 4, we use the aesthetic assessment method SAAN to extract Select the image with the highest rating 5. The AI ​​generation method of a target stylized image according to claim 4, characterized in that: During the training process of step 5, first execute steps 1 and 2 to obtain the corresponding mean and variance μ S ,σ S , and then through the permutation convolutional neural network PerNet, input μ S ,σ S Get the permutation matrix Where φ represents the neural network, τ represents the temperature coefficient, and N r , N c Normalize the rows and columns of the matrix, and add the softmax operation with the temperature coefficient to approach argmax, that is, only one value of the vector is 1, and the rest are 0. Through this operation, the soft permutation matrix PerNet (μ s ;φ).

6. The AI ​​generation method of a target stylized image according to claim 1, characterized in that: Step 5 During the training process, the soft permutation matrix PerNet (μ s ; φ) and μ S ,σ S Perform matrix multiplication to generate That is, the target style vector that needs to be optimized; and f C Perform adaptive normalization operation to obtain the final image output 7. The AI ​​generation method of a target stylized image according to claim 1, characterized in that: Step 6 The original output O is extracted by encoder E and the mean is calculated, which is recorded as where μ s = μ(E(O)) and Minimize the loss of supervised training using distance loss Where D represents the sampled data set.

8. The AI ​​generation method of a target stylized image according to claim 1, characterized in that: Step 7 During the test, the soft permutation matrix PerNet (μ s ;φ) is transformed into a hard permutation matrix H-PerNet(μ s ; φ), use the Hungarian algorithm to operate on each row and column of the matrix, and S ,σ S Perform matrix multiplication to ensure the invariance of style information in high-dimensional space.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the AI ​​generation method of the target stylized image as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the AI ​​generation method of the target stylized image as described in any one of claims 1 to 8 is implemented.

Images