Unpaired multi-scene sketch synthesis method and device

Abstract

The present application relates to a kind of non-matching multi-scene sketch synthesis method and device.The non-matching multi-scene sketch synthesis method described in the present application includes the following steps: feature extraction is carried out to content image and sketch image, obtain content feature and style feature;Content feature and style feature are fused, and content-style feature is obtained;According to content-style feature, generate sketch image.The present application is fused by a new feature fusion method, when the feature of non-matching content feature and style feature is fused, content structure and style texture are highly retained, and content-style feature that can be used to generate non-matching multi-scene sketch is obtained.And, the present application also provides a kind of non-matching multi-scene sketch synthesis device, for executing the non-matching multi-scene sketch synthesis method.Compared with prior art, the present application can generate sketch image using non-matching data, avoids the tedious operation of artificial pairing data, and greatly improves work efficiency.

Description

Technical Field

[0001] This invention relates to the application of artificial intelligence in computer vision, and in particular to a method and apparatus for non-paired multi-scene sketch synthesis. Background Technology

[0002] Sketch generation technology is an artificial intelligence technique designed to transform digital images into an art style similar to hand-drawn sketches. This technology involves many background techniques, including computer vision, deep learning, and image processing. Sketching is a powerful art form. Unlike other painting arts that rely on rich colors, sketching primarily uses black, white, and gray tones, outlining forms with lines and employing contrast and shadow to express the three-dimensionality of shapes. This requires artists to accurately grasp the structure and contours of forms, as well as the proportions of the subject, and possess excellent perspective.

[0003] In recent years, significant progress has been made in computer graphics, enabling the automatic generation of high-fidelity digital sketches. Sketch generation can also be viewed as a domain transfer problem—a hot topic in computer vision. Especially with the development of deep learning, convolutional neural networks have been widely applied in image style transfer, and generative adversarial networks (GANs) have achieved great success in solving image style transfer problems. This has enabled existing style transfer methods to generate multi-scene images using unpaired datasets. However, existing techniques are mainly applied to messy styles, such as oil painting styles, whose stylized images are full of fragmented brushstrokes and have low quality requirements for each element. Sketch styles are very different; they are primarily black, white, and gray tones, based on lines and relying on shading. Therefore, existing sketch generation models are rarely applied to unpaired multi-scene data, but rather mostly use paired training datasets to generate single-scene sketches. Even some improved style transfer methods or traditional image conversion methods cannot generate high-quality multi-scene sketch images. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a method and apparatus for non-paired multi-scene sketch synthesis.

[0005] A method for non-paired multi-scene sketch synthesis includes the following steps: S1. Extract features from the content image and the sketch image to obtain content features. and style characteristics ; S2, Content Features and style characteristics Perform feature fusion to obtain content-style features ; S3. Based on content-style characteristics Generate a sketch image; Specifically, step S2 includes: S21. Based on content characteristics and style characteristics Generate Content - Style Attention Map ; S22. Based on style characteristics Generate style self-attention maps ; S23. Based on Content-Style Attention Map Style Self-Attention Map Perform feature rearrangement and summation to obtain feature matrix A; S24. Based on feature matrix A and content features Acquire content - style characteristics .

[0006] The non-paired multi-scene sketch synthesis method described in this invention proposes a novel feature fusion approach. By performing operations such as generating attention maps, feature rearranging and summing, and normalization on content features and style features respectively, it obtains content-style features that highly preserve content structure and style texture. This method enables the generation of high-quality sketch synthesis images even with non-paired data, particularly achieving non-paired multi-scene sketch synthesis. This reduces the tedious manual data pairing process, significantly improves the efficiency of sketch style image synthesis, and optimizes the overall effect of the synthesized image.

[0007] Further, step S21 specifically involves... Content features and style characteristics Perform calculations to generate a content-style attention map. ,in , , This represents bilinear interpolation; f and g represent the 1 used for feature embedding. 1. Convolutional layer, where Norm represents the mean-variance normalization function. ,in This represents the average value of x. The variance of x is represented by x. , This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

[0008] Further, step S22 specifically involves, according to On style characteristics Perform calculations to generate style self-attention maps. .

[0009] Further, step S23 specifically involves... Content-style attention map Perform feature rearrangement to obtain content-style features. , where h represents the number of 1s used for feature embedding. 1. Convolutional layer; according to Style Self-Attention Map Perform feature rearrangement to obtain style features. The obtained style features are then summed to obtain the feature matrix A, i.e. .

[0010] Further, step S24 specifically involves calculating the mean of the feature matrix A. and standard deviation and according to Calculate and obtain content-style features .

[0011] Meanwhile, the present invention also provides a non-paired multi-scene sketch synthesis device, comprising: The feature extraction module is used to extract features from the content image and the sketch image to obtain content features. and style characteristics ; The feature fusion module is used to integrate content features. and style characteristics Perform feature fusion to obtain content-style features ; The generation module is used to generate content based on style characteristics. Generate a sketch image; The feature fusion module includes: Content-style attention unit, used to determine content characteristics and style characteristics Generate Content - Style Attention Map ; Style self-attention units are used to determine style characteristics. Generate style self-attention maps ; Feature rearrangement summation units, based on content-style attention map Style Self-Attention Map Perform feature rearrangement and summation to obtain feature matrix A; Normalization unit, used to determine the content features based on feature matrix A and content features. Acquire content - style characteristics .

[0012] Furthermore, the content-style attention unit is based on Content features and style characteristics Perform calculations to generate a content-style attention map. ,in , , This represents bilinear interpolation; f and g represent the 1 used for feature embedding. 1. Convolutional layer, where Norm represents the mean-variance normalization function. ,in This represents the average value of x. The variance of x is represented by x. , This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

[0013] Furthermore, the style self-attention unit is based on On style characteristics Perform calculations to generate style self-attention maps. .

[0014] Furthermore, the feature rearrangement summation unit is based on Content-style attention map Perform feature rearrangement to obtain content-style features. , where h represents the number of 1s used for feature embedding. 1. Convolutional layer; according to Style Self-Attention Map Perform feature rearrangement to obtain style features. The obtained style features are then summed to obtain the structure preservation matrix A, i.e. .

[0015] Further, the normalization unit calculates the mean of the feature matrix A. and standard deviation and according to Calculate and obtain content-style features .

[0016] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart of the non-paired multi-scene sketch synthesis method described in this invention; Figure 2 This is a schematic diagram of the non-paired multi-scene sketch synthesis device described in this invention; Figure 3 This is a schematic diagram of the feature fusion module described in this invention. Detailed Implementation

[0018] Please see Figure 1 This invention proposes an unpaired multi-scene sketch synthesis method. By improving the feature fusion method, the feature fusion can highly preserve the content structure and style texture between unpaired data pairs, and realize the generation of stylized images by using feature fusion of unpaired data pairs.

[0019] Please see Figure 2 Based on the aforementioned unpaired multi-scene sketch synthesis method, this invention provides an unpaired multi-scene sketch synthesis device, including a feature extraction module, a feature fusion module, and a generation module. The feature extraction module is used to extract features from the input image; the feature fusion module is used to fuse the extracted image features, which can highly preserve the content structure and style texture between unpaired data pairs, enabling the generation of stylized images of multiple scenes using feature fusion of unpaired data pairs; the generation module is used to generate the final stylized image from the fused features.

[0020] The feature extraction module is a pre-trained VGG-19 network, which is used to perform step S1: extracting features from the content image and the sketch image respectively to obtain content features. and style characteristics .

[0021] Specifically, it includes the following sub-steps: S1A: Extract features from the content image to obtain content features. .

[0022] The VGG-19 network extracts features from the input content image, obtaining content features with a 5-layer content image feature matrix. :

[0023] in, This represents the extracted feature matrix of the x-th layer content image, where the feature matrix is... form.

[0024] Preferably, the content image is a grayscale image that has undergone grayscale processing to reduce the impact of color on subsequent operations.

[0025] S1B: Extract features from the sketch image to obtain style features. .

[0026] Similarly, the VGG-19 network extracts features from the input sketch image to obtain style features with a 5-layer style image feature matrix. :

[0027] in, This represents the extracted x-th layer style image feature matrix, where the feature matrix is... form.

[0028] Please see Figure 3 The feature fusion module is used to perform step S2: processing content features and style characteristics Feature fusion processing is performed to obtain several layers of content-style features. .

[0029] The feature fusion module includes a content-style attention unit, a style self-attention unit, a feature rearrangement summation unit, and a normalization unit.

[0030] The content-style attention unit is used to perform step S21: focusing on content features. and style characteristics Normalization, convolution, and classification are performed to obtain several layers of content-style attention maps. .

[0031] Specifically, it includes the following sub-steps: S211, respectively, regarding content features and style characteristics Perform upsampling to obtain and The and Satisfying the relation:

[0032]

[0033] in Indicates a bilinear interpolation layer; S212, respectively for and Based on the normalization function Perform normalization to obtain and ,in This represents the average value of x. Represents the variance of x; S213, respectively for and Use 1 The first convolutional layer performs convolution processing to obtain... and ; S214, will Perform transpose and union After performing matrix multiplication, substitute the values ​​into the matrix. Calculate and obtain the content-style attention map ,in This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

[0034] In summary, the content-style attention graph of layer x... Satisfying the relation:

[0035] For example, the content-style attention map of layer 4. ,in , The same applies to other layers.

[0036] The style self-attention unit is used to perform step S22: focusing on style features. Normalization, convolution, and classification are performed to obtain several layers of style self-attention maps. .

[0037] Specifically, it includes the following sub-steps: S221, Regarding stylistic features Perform upsampling to obtain The Satisfying the relation:

[0038] in Indicates a bilinear interpolation layer; S222, to Based on the normalization function Perform normalization to obtain ,in This represents the average value of x. Represents the variance of x; S223, to Use two 1s The first convolutional layer performs convolution processing to obtain... and ; S224, will Perform transpose and union After performing matrix multiplication, substitute the values ​​into the matrix. Calculate and obtain style self-attention map ,in This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

[0039] In summary, the style self-attention map of layer x Satisfying the relation:

[0040] For example, the style self-attention map of layer 4. ,in The same applies to other layers.

[0041] The feature rearrangement summation unit is used to perform step S23: processing the content-style attention map respectively. Style Self-Attention Map Normalization and convolution are performed to obtain several feature matrices A.

[0042] Specifically, it includes the following sub-steps: S231, Content-Style Attention Map Transposed style features and style features that have been normalized and convolutionally processed Perform matrix multiplication to obtain several layers of content-style features. Content-style features of layer x Satisfying the relation:

[0043] S232, Style Self-Attention Map Transposed style features and style features that have been normalized and convolutionally processed Perform matrix multiplication to obtain several layers of style features. Style features of the xth layer Satisfying the relation:

[0044] S233, each layer and By combining corresponding features, we obtain several layers of feature matrices A. The feature matrix A of the x-th layer satisfies the following relationship:

[0045] The normalization unit is used to perform step S24: calculate the mean and variance of the feature matrix A, and combine it with the instantiated content features. Derivation yields several layers of content-style characteristics. Content and style features of layer x Satisfying the relation:

[0046] in, This represents the mean of A. This represents the standard deviation of A.

[0047] The generation module is used to perform step S3: based on the input content-style features. Generate stylized images.

[0048] The generation module includes a generator, a feature extractor, a Gram manipulator, and a discriminator.

[0049] The generator is a deep neural network model, such as a Generative Adversarial Network (GAN) or a Variational Autoencoder (VAE). In this embodiment, the generator is a GAN, comprising four sequentially connected upsampling units: a first upsampling unit, a second upsampling unit, a third upsampling unit, and a fourth upsampling unit. Each of the first to third upsampling units includes a residual block, a convolutional block, and an upsampling layer. The fourth upsampling unit includes a convolutional block, an upsampling layer, and a padding layer. Furthermore, the first and second upsampling units are connected by skip connections, such that the output of the first upsampling unit is related to the content-style features. The combined input is then fed into the second upsampling unit, and the output of the second upsampling unit is compared with the content-style features. The combined data is then input into a third upsampling unit, which allows the structure of the content image to be preserved as much as possible while stylizing it.

[0050] The generator is used to perform step S31: processing content-style features. The image is then processed to obtain a stylized generated image.

[0051] Specifically, the fourth layer of content-style features generated in step S24 The input is fed into the generator and processed by the first upsampling unit to obtain the first modulation feature. This first modulation feature is then compared with the third-layer content-style feature generated in step S24. The second modulation feature is obtained by combining the input with the second upsampling unit and processing it. This second modulation feature is then combined with the second-layer content-style feature generated in step S24. The combined inputs are processed by the third upsampling unit to obtain the third modulation feature. The third modulation feature is then input to the fourth upsampling unit to generate the final stylized image.

[0052] The feature extractor is a pre-trained VGG-19 network, used to perform step S32: extracting features from the generated image to obtain generated features. .

[0053] The Gram operator is used to perform step S33: generating features. and style characteristics Perform Gram operations to obtain the Gram matrix.

[0054] The discriminator is also a deep neural network model; in this embodiment, it is a generative adversarial network, comprising two downsampling units and one instantiation layer. The discriminator is used to execute step S34: determining whether the generated image is a true sketch image based on the Gram matrix.

[0055] In the above steps, the generator and discriminator are two opposing forces. The generator aims to produce a generated image that can be easily confused with a real sketch image, while the discriminator aims to accurately distinguish between the real sketch image and the generated image. Their goals are opposite. This adversarial process continuously optimizes the parameters of the generator and discriminator, thereby improving the generator's performance and achieving better generation results. A loss function is used to measure the capabilities of the generator and discriminator, and their parameters are optimized based on this measurement.

[0056] In this invention, the loss function includes generative adversarial loss, sketch style loss, and content structure loss. The generative adversarial loss is...

[0057] The sketch style loss includes relaxed bulldozing distance (rEMD) loss and mean squared error (MSE) loss. Both loss functions can be used to measure the distance between the feature distributions of the generated image and the sketch image, effectively representing the degree of sketch stylization of the generated image. The MSE loss is... The rEMD loss is

[0058] in ,express and The pairwise cosine distance matrix between them.

[0059] The content structure loss includes rEMD loss and normalized perceptual loss, wherein the rEMD loss is:

[0060] The standardized perception is .

[0061] The objective function of the generator is: The objective function of the discriminator is The generator's objective function considers both sketch style loss and content structure loss, maximizing sketch style transfer while preserving content structure as much as possible.

[0062] The non-paired multi-scene sketch synthesis method described in this invention proposes a feature fusion module, which separately processes content features. and style characteristics The process obtains content-style attention maps and style self-attention maps, which fully consider the features of the content image and the sketch image from low to high levels. These features are then processed through feature rearrangement summation units and normalization units to obtain content-style features that highly preserve the content structure and style texture. In particular, in existing technologies, it is difficult to simultaneously preserve the content structure and style texture after feature fusion of unpaired content images and sketch images. However, the feature fusion module described in this invention has a good processing effect on unpaired data, enabling the generation of multi-scene sketches using unpaired data, effectively improving the efficiency and effect of sketch image synthesis.

[0063] The embodiments described above are merely preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A method for non-paired multi-scene sketch synthesis, characterized in that: Includes the following steps: S1. Extract features from the content image and the sketch image to obtain content features. and style characteristics ; S2, Content Features and style characteristics Perform feature fusion to obtain content-style features ; S3. Based on content-style characteristics Generate a sketch image; Specifically, step S2 includes: S21. Based on content characteristics and style characteristics Generate Content - Style Attention Map ; S22. Based on style characteristics Generate style self-attention maps ; S23. Content-Style Attention Map Style Self-Attention Map Perform feature rearrangement and summation to obtain feature matrix A; S24. Regarding the feature matrix A and content features Perform calculations to obtain content-style characteristics. Specifically, this involves calculating the mean of the feature matrix A. and standard deviation and according to Calculate and obtain content-style features .

2. The non-paired multi-scene sketch synthesis method according to claim 1, characterized in that: The specific step S21 is based on Content features and style characteristics Perform calculations to generate a content-style attention map. ,in , , This represents bilinear interpolation; f and g represent the 1 used for feature embedding.

1. Convolutional layer, where Norm represents the mean-variance normalization function. ,in This represents the average value of x. The variance of x is represented by x. , This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

3. The non-paired multi-scene sketch synthesis method according to claim 2, characterized in that: Step S22 is specifically based on On style characteristics Perform calculations to generate style self-attention maps. .

4. A method for synthesizing unpaired multi-scene sketches according to claim 3, characterized in that: Step S23 is specifically based on Content-style attention map Perform feature rearrangement to obtain content-style features. , where h represents the number of 1s used for feature embedding.

1. Convolutional layer; according to Style Self-Attention Map Perform feature rearrangement to obtain style features. The obtained style features are then summed to obtain the feature matrix A, i.e. .

5. A non-paired multi-scene sketch synthesis device, comprising: The feature extraction module is used to extract features from the content image and the sketch image to obtain content features. and style characteristics ; The feature fusion module is used to integrate content features. and style characteristics Perform feature fusion to obtain content-style features ; The generation module is used to generate content based on style characteristics. Generate a sketch image; Its characteristic is that the feature fusion module includes: Content-style attention unit, used to determine content characteristics and style characteristics Generate Content - Style Attention Map ; Style self-attention units are used to determine style characteristics. Generate style self-attention maps ; Feature rearrangement summation units, based on content-style attention map Style Self-Attention Map Perform feature rearrangement and summation to obtain feature matrix A; Normalization unit, used to determine the content features based on feature matrix A and content features. Acquire content - style characteristics Specifically, this involves calculating the mean of the feature matrix A. and standard deviation and according to Calculate and obtain content-style features .

6. The non-paired multi-scene sketch synthesis device according to claim 5, characterized in that: The content-style attention unit is based on Content features and style characteristics Perform calculations to generate a content-style attention map. ,in , , This represents bilinear interpolation; f and g represent the 1 used for feature embedding.

1. Convolutional layer, where Norm represents the mean-variance normalization function. ,in This represents the average value of x. The variance of x is represented by x. , This represents the raw output value of the i-th input channel. This represents the sum of the exponents of all channel output values.

7. The non-paired multi-scene sketch synthesis device according to claim 6, characterized in that: The style self-attention unit according to On style characteristics Perform calculations to generate style self-attention maps. .

8. The non-paired multi-scene sketch synthesis device according to claim 7, characterized in that: The feature rearrangement summation unit is based on Content-style attention map Perform feature rearrangement to obtain content-style features. , where h represents the number of 1s used for feature embedding.

1. Convolutional layer; according to Style Self-Attention Map Perform feature rearrangement to obtain style features. The obtained style features are then summed to obtain the structure preservation matrix A, i.e. .

Images