An ai generation method of a target stylized image

Abstract

The application discloses an AI generation method of a target stylized image. The method first introduces a neural arrangement network to rearrange the feature maps of a single style image, so that they adapt to the feature maps of a content image to generate different styles of content. The quality of the stylized content is measured by using indexes of content structure, style perception and artistic aesthetics. The creative stylized image is compared with the output generated by a traditional style transfer method. High-quality stylized content is sampled from different stylized images generated by a random arrangement matrix. PerNet is trained using the high-quality stylized image. The method converts the original content image and the style image into a more creative style transfer result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to an AI method for generating target stylized images. Background Technology

[0002] Developing computer vision techniques to create humanoid visual art has been a long-standing goal in the history of artificial intelligence and art. One of the core characteristics of this endeavor is creativity, defined as the generation of novel and valuable works of art through simultaneous evaluation. Recently, neural style transfer (NST) technology, as a means of achieving this goal, has shown potential in many successful applications, such as creating artistic images, videos, and 3D objects / scenes, by transferring the artistic style of one image to another. While NST has produced many impressive visual artworks, it still struggles to create truly creative outputs because these methods merely imitate stylistic images, lacking originality.

[0003] In style transfer, some works have attempted to increase style diversity using different methods. Zixuan Huang et al. introduced multi-level feature concatenation and patch attention to achieve better semantic correspondence and preserve richer style details. Yijun Li et al. proposed a diversity loss to enable the feedforward network to produce different outputs. Zhizhong Wang proposed Deep Feature Perturbation (DFP), which uses an orthogonal random noise matrix to perturb the deep age feature map. Furthermore, Haibo Chen et al.'s diverse image style transfer achieved a reversible cross-spatial mapping, resulting in significant diversity. Spatial control Tai-Yin Chiu et al. transferred different styles to different regions of the content image through masking and segmentation. Additionally, as mentioned in the introduction, Songhua Liu et al. proposed style interpolation, which interpolates a set of style features and assigns them corresponding weights; Xin Xie et al.'s stylediscover also constructs a set of style components and linearly combines them to cover new artistic styles. These methods increase the diversity of style transfer by introducing different techniques, but they also share some common drawbacks. First, many methods are computationally complex and resource-intensive, especially when processing large-scale images, potentially requiring significant computational resources and impacting efficiency in practical applications. Second, while they can increase stylistic diversity, they present challenges in the stability and quality control of generated results, particularly when dealing with random perturbations or generating multiple outputs, which can lead to unstable image quality or distortion, making it difficult to maintain a balance between stylistic details and semantic content. Finally, some methods rely too heavily on predefined style components or simple linear combinations, limiting stylistic expressiveness and lacking finesse in style transitions or detail control, potentially resulting in overly simplistic or unnatural style effects. Summary of the Invention

[0004] The purpose of this invention is to provide an AI generation method for target stylized images, which generates new and meaningful artistic styles, and learns style vectors in a supervised manner to transform them into more valuable and surprising styles. This achieves the transformation of the original content image and style image into a more creative style transfer result, known as Creative Style Transfer (CSFer).

[0005] The technical solution for achieving the objective of this invention is: an AI method for generating target stylized images, comprising the following steps:

[0006] Step 1, Given content image I C and style image I S Feature f is extracted using a convolutional image encoder E. C =E(I C ) and f S =E(Is ); and calculate feature f respectively. C and f S The mean and variance μ of the channels C , σ C and μ S , σ S ;

[0007] Step 2: Randomly sample the two-dimensional permutation matrix P, and then compare it with the mean and variance μ of the style image. S , σ S Fusion and Then with f C Perform adaptive instance normalization to obtain And input it into decoder D, depending on the generated result. To obtain a sample set, repeat the process 1000 times for each sample instance.

[0008] Step 3: Perform structural similarity filtering and style difference filtering. First, use SSIM to dynamically calculate the set. The similarity between the content image and the style information is calculated, and then the SPD is used to calculate the difference between the style information and the style of the content image and the style of the original output image, thus obtaining a set.

[0009] Step 4: Perform aesthetic screening using the SAAN aesthetic evaluation method from the set. Select the image with the highest rating

[0010] Step 5: Repeat steps 1-4 to obtain... Create a dataset, where O represents the output of the original model, and use the created dataset to train the neural permutation network PerNet.

[0011] Step 6, The mean of the feature extraction results of the original output O and the encoder E is calculated and denoted as . Where μ s =μ(E(O)) and Minimize the loss during supervised training using distance loss.

[0012] Step 7: For the new content image and style images The PerNet neural permutation network is used to generate the corresponding permutation matrix P, and the permutation matrix P is compared with the new style features. Fusion The content features are adaptively normalized to obtain the image, which is then decoded by the decoder D to generate the artistic image.

[0013] Preferably, in step 1, the content image I is extracted using the ReLU4_1 function of the VGG19 network. C and style image I S Features f C =E(I C ) and f S =E(I S This feature contains both content and style information, and calculates the mean and variance μ for each channel of the feature. C =μ(f C ), σ C =ρ(f C ) and μ S =μ(f s ), σ S =ρ(f S (), representing the style information of the image.

[0014] Preferably, step 2 generates and Sampling method: For the features f extracted by encoder E S Calculate the mean and variance μ of the channels. S , σ S The randomly generated permutation matrix P is obtained by matrix multiplication. get This represents the new style features obtained from a single sampling.

[0015] Preferably, step 3 will collect the set Firstly, through The selected image has content information similar to the selected image. The parameter α represents the largest 30% of the structural calculations, which is obtained through dynamic calculation. Then, through... σ(φ i (y))||2, calculate the mean and variance of different ReLU layers of encoder E, and use Calculate the sampled image and the content image I C The degree of style difference between the original style transfer O and the original style transfer, where SPD(x,y) represents the degree of style difference between the two images, L equals 4 to indicate different ReLU layers, μ and σ represent the mean and variance, β1 and β2 represent the maximum 50% and 30% of the two image styles, respectively, β1 and β2 are obtained dynamically, and φ i This represents the different ReLU layers of the Encoder (specifically Relu1_1, Relu2_1, Relu3_1, and Relu41); the final sampled result is a set.

[0016] Preferably, in step 4, the aesthetic evaluation method SAAN is used from the set Select the image with the highest rating

[0017] Preferably, during the training process in step 5, steps 1 and 2 are first executed to obtain the corresponding mean and variance μ. S , σ S Then, by permuting the convolutional neural network PerNet, the input μ S , σ S Obtain the permutation matrix Where φ represents the neural network, τ represents the temperature coefficient, and N... r N c Normalizing the rows and columns of the matrix, and then applying a softmax operation with a temperature coefficient to approximate argmax, results in a vector with only one value of 1 and the rest of the values ​​being 0. This operation yields the soft permutation matrix PerNet(μ). s ;φ).

[0018] Preferably, during the training process in step 5, the soft permutation matrix PerNet(μ) is used. s ;φ) and μ S , σ S Perform matrix multiplication to generate That is, the target style vector being sampled; and f C The final image output is obtained by performing adaptive normalization.

[0019] Preferably, the The mean of the feature extraction results of the original output O and the encoder E is calculated and denoted as . Where μ s =μ(E(O)) and Minimize the loss during supervised training using distance loss. Where D represents the dataset obtained from sampling.

[0020] Preferably, step 8 requires the soft mass transfer matrix PerNet(μ) to be applied during the testing process. s ;φ) is transformed into a hard permutation matrix H-PerNet(μ) s ;φ), using the Hungarian algorithm to operate on each row and column of the matrix, and summing with μ S , σ S Performing matrix multiplication ensures the invariance of style information in high-dimensional space.

[0021] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for generating artistic images using AI.

[0022] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the above-described method for generating artistic images using AI.

[0023] Compared with existing technologies, the significant advantages of this invention are: (1) We propose a neural permutation method that, by ranking the mean and variance of feature maps of a single style image, enables us to capture the inherent uniqueness and surprise elements in the style image. (2) By combining the permutation method, we explore a creative style transfer method to generate novel and meaningful works of art. This invention is the first to efficiently integrate permutation and evaluation to create innovative aesthetic styles. (3) Experiments show that our method outperforms existing style transfer techniques in terms of creativity and diversity, and exhibits strong generalization ability with a simple integration process.

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the process of the present invention.

[0026] Figure 2 This is a structural diagram of the CSFer of the present invention.

[0027] Figure 3 This is a diagram illustrating the sampling process of the present invention.

[0028] Figure 4 This is a comparison diagram of the effects of the present invention with other style transfer methods.

[0029] Figure 5 This is a test result diagram of the application of the present invention to other style transfer methods.

[0030] Figure 6 This is a diagram showing the effect of applying the present invention to the CAN and Creative Decoder models.

[0031] Figure 7 Box plot of the SPD index in WikiArt and CAN data for this invention.

[0032] Figure 8 This is the final sampling result image of the sampling process of this invention. Detailed Implementation

[0033] This invention proposes an AI-generated method for stylized images, which generates novel and meaningful style transfer results that differ from the original style image. This method has wide applications in fields such as art creation, design, advertising, fashion, and social media. It can combine classic or modern art styles with target content to create unique visual effects, enhancing the expressiveness and appeal of the work. This method can provide innovative style generation solutions for various creative endeavors, promoting artistic creation and content production across industries.

[0034] This invention enables the generation and transfer of creative artistic styles into content images using a single-style image. This method differs significantly from previous works that used linear combinations of style components. We introduce a novel neural arrangement method that manipulates the features of a single-style image to produce meaningful and surprising stylized results that stand out from previous works that often fall within similar style boundaries.

[0035] This invention considers three core criteria for evaluating creativity: novelty, surprise, and value. By combining style transfer techniques, we quantify the degree of deviation of the new style from a given style to assess novelty, and its difference in unfamiliarity from the original style to assess surprise. Simultaneously, value assesses the aesthetic appeal of the new style relative to the original style. Based on these criteria, we design a new and surprising set of features through style transfer, adjusting these features within a single-style image to create content with an aesthetically pleasing style.

[0036] To achieve this goal, we draw inspiration from permutation and combinatorics theory to effectively reconfigure the features of single-style images, thereby generating diverse sets of style features and new styles. Based on these newly generated styles, we use Style Perceived Distance (SPD) as a score to evaluate their novelty and surprise. Simultaneously, we utilize the Style Specific Art Evaluation Network (SAAN) to calculate aesthetic scores to assess their artistic value. Higher SPD and SAAN scores indicate greater creativity (i.e., novelty, surprise, and aesthetic value). Furthermore, we determine the most creative output by selecting the generated version with the highest SPD and SAAN scores.

[0037] This invention develops a neural creative process for style transfer, named Creative StyleTransfer (CSFer). Given an art style image and a content image, the process alternates to generate new style images different from the given style and evaluate their creativity. First, we design a neural permutation network (PerNet) to predict a permutation matrix to rank the mean and variance of the style image feature maps, thereby adjusting the content image feature maps to generate new style images. Second, we utilize a combination of content structure, style perception, and aesthetic metrics to evaluate the novelty, surprise, and aesthetic value of the newly generated style images: Structural Similarity (SSIM), Style Perceptual Distance (SPD), and Style-Specific Art Evaluation Network (SAAN) score. These metrics help highlight the differences in structural information and aesthetic significance between the new style images and conventional style images (such as those generated by conventional neural style transfer methods like AdaIN). Finally, we train PerNet by randomly sampling the permutation matrix to meet the criteria for generating novel and aesthetically pleasing style images.

[0038] This invention first introduces a PerNet neural network to rearrange the feature maps of individual style images, thereby adapting them to the feature maps of content images and generating content with different styles. Essentially, this permutation operation enriches the foundation of a single style in a high-dimensional feature space, rather than a traditional linear combination of multiple styles. To measure the quality of stylized content, we utilize metrics including content structure, style perception, and artistic aesthetics; these metrics enable us to evaluate creative stylized content by comparing the creatively stylized images with the outputs of traditional style transfer methods; using this evaluation, we sample high-quality stylized content from different stylized images generated by a random permutation matrix. Finally, we use these high-quality stylized images to train PerNet. The accompanying figures demonstrate that CSFer can create novel stylized results different from the originals. Furthermore, CSFer demonstrates robust generalization capabilities by seamlessly integrating PerNet into existing style transfer methods.

[0039] This invention enriches the foundation of single-style representation in high-dimensional feature spaces by applying permutation operations, outperforming traditional multi-style linear combinations. To measure the quality of stylized images, we utilize metrics including content structure, style perception, and artistic aesthetics. These metrics enable us to evaluate stylized content by comparing the stylized images with the outputs of traditional style transfer methods. Using this evaluation, we sample high-quality stylized content from different stylization results generated by random permutation matrices. Finally, we use this high-quality stylized result to train PerNet.

[0040] Combination Figures 1-3 An AI-based method for generating targeted stylized images, with the following specific steps:

[0041] Step 1, Given content image I C and style image I S Feature f is extracted using a convolutional image encoder E. C =E(I C ) and f S =E(I s ); and calculate feature f respectively. C and f S The mean and variance μ of the channels C , σ C and μ S , σ S , μ , σ∈R 512 ;

[0042] Step 2, Generate and Sampling method: For the features f extracted by encoder E S Calculate the mean and variance μ of the channels. S , σ S The randomly generated permutation matrix P∈R 512×512 The permutation matrix has only one 1 in each row and column, with all other values ​​being 0. Matrix multiplication yields... Then with f C Perform adaptive instance normalization:

[0043]

[0044] get And input it into decoder D, depending on the generated result. To obtain a sample set, repeat the process 1000 times for each sample instance.

[0045] Step 3: Set Firstly, through Select images with similar content information, then... Calculate the mean and variance of different ReLU layers in encoder E using Calculate the sampled image and the content image I C The degree of style difference between the transferred style O and the original style is used to obtain the set.

[0046] Step 4: Use the aesthetic evaluation method SAAN to evaluate the set Select the image with the highest rating

[0047] Step 5: Repeat steps 1-4 to obtain several... Create a dataset and train the neural permutation network. During training, first execute steps 1 and 2 to obtain the corresponding mean and variance μ. S , σ S Then, by permuting the convolutional neural network PerNet, the input μ S , σ S Obtain the permutation matrix The softmax operation, which incorporates a temperature coefficient, approximates the argmax operation. This same operation is performed on each row and column of the matrix to obtain the soft permutation matrix PerNet(μ). s ;φ). The soft permutation matrix PerNet(μ) s ;φ) and μ S , σ S Perform matrix multiplication to generate That is, the target style vector being sampled. and f C The final image output is obtained by performing adaptive normalization.

[0048] Step 6, The mean of the feature extraction results of the original output O and the encoder E is calculated and denoted as . Where μ s =μ(E(O)) and Minimize the loss during supervised training using distance loss.

[0049] Step 7: During the testing process, the soft mass transfer matrix PerNet(μ) needs to be applied. s ;φ) is transformed into a hard permutation matrix H-PerNet(μ) s ;φ), using the Hungarian algorithm to operate on each row and column of the matrix, and summing with μ S , σ S Matrix multiplication ensures the invariance of style information in high-dimensional space, ultimately producing creative generative results.

[0050] Table 1 User Survey.

[0051]

[0052] We conducted a user study to evaluate the effectiveness of our CSFer algorithm compared to other methods. The evaluation was divided into three categories, analyzed from different perspectives: the most popular results among the five NST methods, a comparison of creative artistic effects between CAN and CSFer, and a comparison of visual effects between CD and CSFer. A total of 338 votes were collected in each category, with 169 users casting 1,014 votes in total. The results showed that CSFer received the most votes, with 28.4% of users selecting it as their favorite method, while ArtINS only received 4.7%. Furthermore, approximately 57% of users felt that CSFer offered a more creative style than CAN, and 58% felt that CSFer provided a superior visual experience compared to CD. Overall, CSFer emerged as the most popular among all novel creative methods.

[0053] Table 2 shows the aesthetic evaluation of style transfer methods for three different style datasets (WikiArt, CAN, and CD).

[0054]

[0055]

[0056] We demonstrate the value of CSFer in generating aesthetically meaningful styles by evaluating its aesthetic quality. Two methods were used for evaluation: SAAN and TANet, and the results are presented in the corresponding tables. The results show that, except on the SAAN CD dataset, CSFer's styles outperform state-of-the-art (SOTA) methods on both SAAN and TANet. For example, IEC+CSFer and MST+CSFer achieved the highest SAAN scores of 4.782 and 4.789 on the WikiArt and CAN datasets, respectively. Similarly, LST+CSFer and IEC+CSFer achieved the highest TANet scores of 5.216, 5.261, and 4.986 on the WikiArt, CAN, and CD datasets, respectively. In conclusion, CSFer not only improves upon or matches the performance of AdaIN, MST, LST, and ArtINS, but also creates valuable styles.

[0057] Figure 4The following are various visual images of style transfer achieved through different methods. We have the following four observations. First, the AdaIN method transfers the artistic style of a style image to a content image, generating styles similar to the given artistic style but lacking diversity, let alone creativity. Second, although the DFP method generates diverse styles, it still closely resembles the given style due to the small perturbation applied. With larger perturbations, DFP may fail to successfully transfer artistic styles. Third, the ArtINS method generates more diverse styles than AdaIN, LST, and DFP, but its cardinality of independent components is approximately 10, which is a limitation. ArtINS also requires manual setting of the parameters for linear combinations. Finally, unlike these methods, our CSFer method generates novel and meaningful artistic styles through permutations and combinations of neural networks, creating new style bases. Even when applied to ArtINS, ArtINS+CSFer generates richer and more creative results.

[0058] Figure 5 The images generated by CSFer in combination with LST, IEC, and MST are shown, demonstrating that CSFer can enhance the visual generation effect by combining with other methods. According to Table 2, the aesthetic evaluation of their effects is higher than that of the original generation results.

[0059] Figure 6 We compared the art styles generated by CAN, Creative Decoder (CD), and our proposed CSFer method. To ensure a fair comparison, we applied each style to the same content using AdaIN. Our scheme generated novel styles with mixed colors, compared to CAN and Creative Decoder. e The decoder shows significant improvements. For example, in the fourth column, CSFer naturally creates a mixed green and red style, which is visually superior to the black and yellow style generated by CAN+AdaIN. Similarly, in the eighth column, CSFer generates a mixed green and yellow style that is more appealing than the yellow style generated by CD+AdaIN.

[0060] Figure 7 This demonstrates a comparison of the results generated by the AdaIN, MST, LST, and ArtINS methods with those generated by CSFer. The result SPD(O, I) is greater than that generated by the original method. S The box plot. Therefore, CSFers increase the degree of style differentiation and create new style features.

[0061] Figure 8The diagram shows a partial example of sampled images from the dataset, represented from left to right as the content image, segmented image, original result, and sampled result image. We determined the formula based on 200 pairs of style and content images.

[0062]

[0063] The values ​​of the three parameters α, β1, and β2 are used to generate the image set for each pair of images. We are about (μ) S , σ S 1000 random permutations were performed. We chose to set the parameters to α = 0.2, β1 = 0.5, and β2 = 0.3. These parameter values ​​were chosen because they performed well in the average aesthetic evaluation SNNA score. Using these parameter settings, we obtained a subset of 10 images, denoted as . And an optimal image was determined.

[0064] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AI generation method of a target stylized image, characterized by, Comprising the steps of: Step 1, given a content image I C and a style image I S , extract features f C = E(I C ) and f S = E(I s ) using a convolutional image encoder E; and compute the mean and variance of the channels of f C and f S , μ C ,σ C and μ S ,σ S , respectively; Step 2, randomly sample two-dimensional permutation matrix P, respectively with the mean and variance of the style image μ S ,σ S fusion and again with f C adaptive instance normalization operation to obtain and input into the decoder D, the generated results as a sampling example, repeat 1000 times to obtain a sampling set Step 3, structure similarity screening and style difference screening, first use SSIM to dynamically calculate the similarity of the set with the content image, and then use SPD to sequentially calculate the difference degree of the style information with the content image style and the original output image style, to obtain the set Specifically: set First, through the formula Select images with similar structural information to the content images, where parameter α represents the largest 30% of the structural calculations, and then... Calculate the mean and variance of different ReLU layers in encoder E using Calculate the sampled image and the content image I C The degree of style difference between the original style transfer O and the original style transfer, where SPD(x,y) represents the degree of style difference between the two images, L represents different ReLU layers, μ and σ represent the mean and variance, β1 and β2 represent the maximum 50% and 30% of the two image styles, respectively, and φ i This represents the different ReLU layers of the encoder, specifically Relu1_1, Relu2_1, Relu3_1, and Relu4_1; the final sampled result is a set. Step 4, Aesthetic selection is performed using the aesthetic evaluation method SAAN to select the highest scoring images from the set ​ Step 5, repeating Step 1-Step 4, to obtain A data set is made, where O is the output result of the original model, and the neural replacement network PerNet is trained using the data set made; In the process of training, first, step 1 and step 2 are executed to obtain the corresponding mean and variance μ S ,σ S , then the permutation convolutional neural network PerNet is inputted with μ S ,σ S to obtain the permutation matrix Wherein φ represents a neural network, τ represents a temperature coefficient, N r , N c Normalization is performed on the rows and columns of the matrix, and the softmax operation with the temperature coefficient is added to approach argmax, that is, the vector has only one value of 1 and the remaining values of 0. Through the operation, the soft permutation matrix PerNet(μ s ; φ); Step 6, the and original output O pass through the encoder E to extract feature results, and the mean value is calculated, denoted as where μ s = μ(E(O)) and Loss of supervised training using distance loss minimization Step 7, for the new content image and style image A corresponding permutation matrix P is generated using a neural permutation network PerNet, and the permutation matrix P is fused with the new style feature to obtain Adaptive normalization is performed with the content feature to obtain, and an artistic image is generated through a decoder D.

2. The AI generation method of a target stylized image according to claim 1, characterized in that, Step 2 generated by sampling and The specific method is as follows: for the feature f extracted by the encoder E S , the mean variance μ of the channel is calculated S ,σ S , the randomly generated permutation matrix P is multiplied to obtain 3.The AI generation method of a target stylized image according to claim 1, characterized in that, In the step 5 training process, the soft permutation matrix PerNet(μ s ; φ) and μ S ,σ S are multiplied to generate , i.e., the target style vector to be optimized; and f C is subjected to adaptive normalization operation to obtain the final image output 4.The AI generation method of a target stylized image according to claim 1, wherein, Step 6 and original output O pass through encoder E to extract feature results, and calculate the mean, denoted as where μ s = μ(E(O)) and Loss of supervised training using distance loss minimization where D represents the sampled data set. 5.The AI generation method of a target stylized image according to claim 1, wherein, Step 7 requires the transformation of the soft permutation matrix PerNet(μ s ; φ) into the hard permutation matrix H-PerNet(μ S ; φ) using the Hungarian algorithm for each row and each column of the matrix and matrix multiplication with μ S , σ S to guarantee the invariance of the style information in the high-dimensional space.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the AI generation method of the target stylized image as claimed in any one of claims 1-5 when executing the program.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the AI generation method of the target stylized image as claimed in any one of claims 1-5.

Images