Painting image harmony method based on attention generative adversarial network

Abstract

The invention discloses a drawing image harmony method based on an attention generative adversarial network. The method comprises the following steps: firstly, establishing an image data set required by network training; constructing a drawing image harmony model based on the attention generative adversarial network, and using a generator comprising a style converter to take the composite image and the instance segmentation mask as input and output harmony drawing images; secondly, providing a pixel-level feature identifier to guide a generator to generate a more real and harmonious drawing image; training the model by adopting the training data; and finally, receiving a picture needing image harmony through the trained model, and outputting the picture after harmony processing is completed. According to the method, local and global textures are transferred by self-adaptively calculating mode repeatability, so that the phenomenon that an attention module repeatedly abuses a specific local patch in a style image to cause disharmony and obvious repeated artistic images in the existing method is solved, and a harmonious image which clearly expresses an artistic style by using different background style modes can be generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of image synthesis, and relates to an image harmony algorithm based on an attention-generated adversarial network, and in particular to the field of painting image harmony based on deep learning technology. Background Art

[0002] Image synthesis is an important operation in image processing. After inserting a realistic foreground object into another painting background image, the synthesized image may look unnatural and discordant. The goal of painting image harmonization is to transfer the style of the background painting to the foreground object, which is a challenging task due to the large domain gap between the foreground and background. In the process of achieving foreground style transfer, traditional attention mechanisms are used to map local regions of the style image to the corresponding regions of the foreground image to achieve artistic style transfer. However, due to the low semantic correspondence between arbitrary content and artwork, the attention modules in existing painting image harmonization methods often repeatedly use specific local regions in the style image, resulting in discordant and obviously repetitive artifacts. Summary of the invention

[0003] In view of the shortcomings in the prior art, the present invention provides a painting image harmony method based on an attention-generated adversarial network. The present invention proposes an adversarial learning method based on an enhanced attention mechanism and a captured style organization structure to bridge the domain gap between the foreground feature map and the background feature map to achieve natural harmony of the synthesized image. The generator consists of an encoder, a style converter and a decoder. The encoder adopts a pre-trained VGG-19 network, and the decoder is designed to reflect the architecture of the encoder by using upsampling operations. The style converter includes an attention style transfer module and a global feature statistics conversion module. The attention style transfer module is used to transfer the style according to the local patch correspondence, while the global feature statistics conversion module is used to capture the style information of the background painting from a global perspective and apply it to the foreground object. A global discriminator is used to evaluate the authenticity of the synthesized image generated by the generator. The goal of the discriminator is to make the improved foreground feature map generated by the generator indistinguishable from the background feature map, so that the synthesized image looks more natural.

[0004] By this method, the present invention can effectively solve the problem that the synthesized image may look unnatural and disharmonious after inserting a realistic foreground object into another painting background image. By bridging the domain gap between the foreground feature map and the background feature map through adversarial learning, the natural harmony of the synthesized image can be achieved, thereby achieving the purpose of painting image coordination.

[0005] A method for painting image harmony based on attention-generated adversarial network, specifically comprising the following steps:

[0006] Step 1: Dataset acquisition;

[0007] Create the image dataset required for network training in step 2.

[0008] Step 2: Construct a painting image harmony model based on attention-generated adversarial network;

[0009] Use the generator including the style converter to synthesize the image I c The instance segmentation mask M (i.e., foreground mask) is used as input to output a harmonious painting image. Secondly, a pixel-level feature discriminator is proposed to guide the generator to produce more realistic and harmonious painting images.

[0010] Step 3: Use the training data obtained in step 1 to train the constructed painting image harmony model based on the attention-generated adversarial network;

[0011] Step 4: The trained neural network model receives the image that needs to be harmonized and outputs the image after the harmonization process is completed.

[0012] Furthermore, the specific operations of step 1 are as follows:

[0013] Download the public datasets COCO and WikiArt datasets. Use images from the WikiaArt dataset as painting background images and use the instance segmentation masks provided by the WikiaArt dataset to extract photographic foreground objects from the COCO dataset. Randomly use instance segmentation masks to insert foreground objects into the painting background images to form synthetic images I c , since the number of foreground images is less than that of background images, the foreground images will be reused.

[0014] Furthermore, the painting image harmony model based on the attention-generated adversarial network includes a generator network model and a discriminator network model. The generator network model consists of a synthetic image (I c ), painting background image (I s ) and foreground mask (M) as input. The generator consists of an encoder, an attention style transfer module, and a decoder. The encoder Em contains the first four layers of the pre-trained VGG-19 (i.e., up to ReLU4_1), and the decoder structure is symmetrical to the encoder. When training the network, the encoder Em adds skip connections on ReLU1_1, ReLU2_1, and Relu3_1 to preserve the content details in the low-level feature maps. First, the encoder Em is extracted from the painting background image I s , Synthetic Image I c And foreground mask M, L = 4 layers of feature maps are extracted, which come from four encoder layers ReLU1_1, ReLU2_1, ReLU3_1 and ReLU4_1, respectively, to obtain background image features Synthetic Image Features and the resized foreground mask For the lth layer, the feature map and With the resized foreground mask M l are input into the attention style transfer module, which adjusts Statistics of the foreground area in the middle, making it The statistical information of

[0015] Attention style transfer module is a key component that transfers local and global textures adaptively according to pattern repeatability. s The degree of pattern repeatability in the paper considers two perspectives:

[0016] 1) The similarity between each patch and the entire image, i.e., the repetitiveness of patches within the image;

[0017] 2) The similarity between patches, i.e., the repetitiveness between patches.

[0018] For a given image I, features are collected from the lth (l=1,2,3,4)th encoder layer of encoder φ Then N feature patches are obtained by spatially dividing each feature with ratio r Right now In order to estimate the similarity between the local style and the global style, that is, the patch repetitiveness α within the image intra , measure each feature patch The Gram matrix of l The cosine similarity of is expressed as follows:

[0019]

[0020] In the formula, G represents the Gram matrix calculation, and L represents the number of convolutional layers in the encoder φ.

[0021] In addition, the texture similarity between different patches, i.e., the inter-patch repetitiveness α, is calculated inter , to estimate whether a specific local pattern occurs frequently. This process is formalized as follows

[0022]

[0023] where β is derived from A set of selected pairs is selected from the pairs, which is effectively calculated with probability p, that is, Where N represents the number of feature patches, Represents the number of combinations of selecting 2 elements from N elements. This value represents the number of all possible feature block pairs and is used to estimate the repeatability of local patterns in the image.

[0024] Finally, the pattern repeatability α style The calculation is as follows:

[0025]

[0026] For the lth layer of the encoder, the feature map and with the resized foreground mask M l are input together into the AdaIN layer, which adjusts Statistics of the foreground area in the middle, making it The statistical information of Stylized Attention Map It is calculated based on the feature maps output by the 3rd and 4th layers of the encoder. The formula is as follows:

[0027]

[0028] where h and Represent convolutional layers and upsampling, respectively.

[0029] Finally, calculate the final output of each layer of the encoder As shown below:

[0030]

[0031] The decoder is a symmetrical structure of VGG-19. The final generator output is obtained as the input of the decoder or connected with the decoder features through skip connection.

[0032] Discriminator Network Model D g The inputs are the painting background image I s And the harmonious image I output by the generator h , respectively, through seven convolutional layers and self-attention layers for feature extraction, and LeakyReLU layers are applied after all convolutional layers before the last convolutional layer to generate the probability D of the discriminator output prediction respectively. g (I s ) and D g (I h ) is used to calculate the adversarial loss.

[0033] Furthermore, the specific operations of step 3 are as follows:

[0034] The generator and discriminator are optimized by alternating training, using a hinged version of the adversarial loss function, which is formulated as follows:

[0035]

[0036] Where D g is the discriminator, responsible for judging whether the input image is a painting background image or a harmonious image (generated by the generator G). The loss function is minimized when the harmonious image can be accurately identified. Loss function, the generator is encouraged to generate images that can fool the discriminator, thereby improving the realism of the generated images.

[0037] The training is performed using the Adam optimizer with β1 = 0, β2 = 0.9. By default, the learning rate for the discriminator is 0.0004 and the learning rate for the generator is 0.0001. During training and testing, the input images are resized to 256×256.

[0038] Furthermore, the specific method of step 4 is as follows:

[0039] First, load the painting image harmony model weights trained in step 3 and update the parameters in the model. Secondly, the image to be harmonized is obtained by pasting the foreground image onto the background image according to the instance segmentation mask as input data into the trained network model. The input data continuously updates the parameters through the generator and the discriminator, and finally the generator obtains the harmonized output image.

[0040] The beneficial effects of the present invention are as follows:

[0041] The present invention transfers local and global textures by adaptively calculating pattern repeatability to solve the problem that the attention module in the existing method repeatedly abuses specific local patches in the style image, resulting in disharmony and obvious repetitive artifacts. The present invention can calculate the repeatability of the style and generate harmonious images that clearly express the artistic style with different background style patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of an attention-based generator network model according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the structure of the discriminator network model according to an embodiment of the present invention;

[0045] Figure 4 The figure shows the comparison between the method of the present invention and the existing method. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and examples.

[0047] The method of the present invention specifically comprises the following steps:

[0048] like Figure 1 As shown, a painting image harmony method based on attention-generated adversarial network specifically includes the following steps:

[0049] Step 1: Dataset acquisition;

[0050] Create the image dataset required for network training in step 2. Download the public datasets COCO and WikiArt datasets. The COCO dataset is a large-scale dataset containing 123,287 images with instance segmentation annotations for objects in 80 categories. The Wikiart dataset is a large digital art dataset containing 81,444 images from 27 different styles. Use images from the WikiaArt dataset as painting background images and extract photographic foreground objects from the COCO dataset using the instance segmentation masks provided by the WikiaArt dataset. Randomly use the instance segmentation masks to insert foreground objects into the painting background images to form synthetic images I c ,Since the number of foreground images is less than the background images, the foreground images will be reused. ,The final dataset includes 11,596 foreground images, 23,302 background painting images, 11,596 instance segmentation masks, and 23,302 synthetic images.

[0051] Step 2: Construct a painting image harmony model based on attention-generated adversarial network;

[0052] Step 2.1: Modeling the harmony theory of painting images based on attention-generated adversarial networks. Use the generator including the style converter to convert the synthetic image I c The instance segmentation mask M (i.e., foreground mask) is used as input to output a harmonious painting image. Secondly, a pixel-level feature discriminator is proposed to guide the generator to produce more realistic and harmonious painting images.

[0053] The theoretical model of the attention-enhanced generator is formulated as:

[0054] I c =G(I c ,M) (1)

[0055] Among them I c represents the synthetic image, the foreground mask M represents the area to be coordinated, and G represents the generator network, which is composed of continuously minimizing the loss function L rThe specific formula is:

[0056] L r =‖I c -I r ‖ (2)

[0057] Among them I r This is the output image after harmony.

[0058] Step 2.2: Construct an image harmony training network based on the attention-generated adversarial network, and train the network to form a network model that produces a painting image coordination effect.

[0059] The network model includes a generator network model and a discriminator network model. The generator network model consists of a synthetic image (I c ), painting background image (I s ) and foreground mask (M) as input. Figure 2 As shown in Figure 1, the generator consists of an encoder, an attention style transfer module, and a decoder. The encoder Em contains the first four layers of the pre-trained VGG-19 (i.e., up to ReLU4_1), and the decoder structure is symmetrical to the encoder. When training the network, the encoder Em adds skip connections on ReLU1_1, ReLU2_1, and Relu3_1 to preserve the content details in the low-level feature maps. First, the encoder Em extracts the background image I from the painting. s , Synthetic Image I c And foreground mask M, L = 4 layers of feature maps are extracted, which come from four encoder layers ReLU1_1, ReLU2_1, ReLU3_1 and ReLU4_1, respectively, to obtain background image features Synthetic Image Features and the resized foreground mask For the lth layer, the feature map and With the resized foreground mask M l are input into the attention style transfer module, which adjusts Statistics of the foreground area in the middle, making it The statistical information of

[0060] Attention style transfer module is a key component that transfers local and global textures adaptively according to pattern repeatability. s The present invention considers two perspectives regarding the degree of pattern repeatability:

[0061] 1) The similarity between each patch and the entire image, i.e., the repetitiveness of patches within the image;

[0062] 2) The similarity between patches, i.e., the repetitiveness between patches.

[0063] For a given image I, we collect features from the lth (l=1,2,3,4)th encoder layer of encoder φ Then N feature patches are obtained by spatially dividing each feature with ratio r Right now In order to estimate the similarity between the local style and the global style, that is, the patch repetitiveness α within the image intra , measure each feature patch The Gram matrix of l The cosine similarity of is expressed as follows:

[0064]

[0065] In the formula, G represents the Gram matrix calculation and L represents the number of convolutional layers in the encoder φ. intra indicates that the global style appears consistently in its local patches, while a lower α inter Meaning that the local style is different from the global style.

[0066] In addition, the texture similarity between different patches, i.e., the inter-patch repetitiveness α, is calculated inter , to estimate whether a specific local pattern occurs frequently. This process is formalized as follows

[0067]

[0068] where β is derived from A set of selected pairs is selected from the pairs, which is effectively calculated with probability p, that is, Where N represents the number of feature patches, Represents the number of combinations of selecting 2 elements from N elements. This value represents the number of all possible feature block pairs and is used to estimate the repeatability of local patterns in the image.

[0069] Finally, the pattern repeatability α style The calculation is as follows:

[0070]

[0071] For the lth layer of the encoder, the feature map and with the resized foreground mask M l are input together into the AdaIN layer, which adjusts Statistics of the foreground area in the middle, making it The statistical information of Stylized Attention Map It is calculated based on the feature maps output by the 3rd and 4th layers of the encoder. The formula is as follows:

[0072]

[0073] where h and Represent convolutional layers and upsampling, respectively.

[0074] Finally, calculate the final output of each layer of the encoder As shown below:

[0075]

[0076] The decoder is a symmetrical structure of VGG-19. The final generator output is obtained as the input of the decoder or connected with the decoder features through skip connection.

[0077] like Figure 3 As shown, the discriminator network model D g The inputs are the painting background image I s And the harmonious image I output by the generator h , respectively, through seven convolutional layers and self-attention layers for feature extraction, and LeakyReLU layers are applied after all convolutional layers before the last convolutional layer to generate the probability D of the discriminator output prediction respectively. g (I s ) and D g (I h ) is used to calculate the adversarial loss.

[0078] Step 3: Use the training data obtained in step 1 to train the constructed painting image harmony model based on the attention-generated adversarial network;

[0079] Specifically, the generator and discriminator are optimized through alternating training, using a hinged version of the adversarial loss function, which is formulated as follows:

[0080]

[0081] Where D g is the discriminator, responsible for judging whether the input image is a painting background image or a harmonious image (generated by the generator G). The loss function is minimized when the harmonious image can be accurately identified. The loss function encourages the generator to generate images that can fool the discriminator, thereby improving the realism of the generated images. The form of the hinge loss function encourages the discriminator to predict high scores for real images and low scores for generated images, thereby encouraging the generator to generate more realistic images. By alternately optimizing the loss functions of the discriminator and the generator, the generator and the discriminator can learn together to continuously improve the quality and realism of the generated images. This training strategy makes the generator and the discriminator compete with each other and eventually converge to an equilibrium point to produce high-quality synthetic images.

[0082] The training is performed using the Adam optimizer with β1 = 0, β2 = 0.9. By default, the learning rate for the discriminator is 0.0004 and the learning rate for the generator is 0.0001. During training and testing, the input images are resized to 256×256.

[0083] Step 4: The trained neural network model receives the image that needs to be harmonized and outputs the image after the harmonization process is completed.

[0084] First, load the painting image harmony model weights trained in step 3 and update the parameters in the model. Secondly, the image to be harmonized is obtained by pasting the foreground image onto the background image according to the instance segmentation mask as input data into the trained network model. The input data continuously updates the parameters through the generator and the discriminator, and finally the generator obtains the harmonized output image.

[0085] Experimental results: Figure 4 In the figure, we compare the visualization results of our method with the recent painting image harmony method PHDNet (AAAI2023) and ProPIH (AAAI2024). In the first row, the umbrella generated by our enhanced attention mechanism better integrates the style of the background to achieve satisfactory results. As shown in the second row, our method can better transfer the style of the background. In the third row, our method better preserves the foreground content (letters on the truck).

[0086] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0087] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A painting image harmony method based on attention-generated adversarial network, characterized in that: The specific steps include: Step 1: Dataset acquisition; Establish the image dataset required for network training in step 2; Step 2: Construct a painting image harmony model based on attention-generated adversarial network; Use the generator including the style converter to synthesize the image I c and instance segmentation mask M, i.e. foreground mask, as input to output a harmonious painting image; secondly, a pixel-level feature discriminator is proposed to guide the generator to produce more realistic and harmonious painting images; Step 3: Use the training data obtained in step 1 to train the constructed painting image harmony model based on the attention-generated adversarial network; Step 4: The trained neural network model receives the image that needs to be harmonized and outputs the image after the harmonization process is completed.

2. According to claim 1, a method for painting image harmony based on attention-generated adversarial network is characterized in that: Step 1: Download the public datasets COCO and WikiArt datasets; use images from the WikiaArt dataset as painting background images, and use the instance segmentation masks provided by the WikiaArt dataset to extract photographic foreground objects from the COCO dataset; randomly use instance segmentation masks to insert foreground objects into the painting background images to form synthetic images I c , since the number of foreground images is less than that of background images, the foreground images will be reused.

3. A method for painting image harmony based on attention-generated adversarial networks according to claim 1 or 2, characterized in that: The painting image harmony model based on the attention-generated adversarial network includes a generator network model and a discriminator network model; the generator network model is composed of a synthetic image I c , painting background image I s And foreground mask M as input; The generator consists of an encoder, an attention style transfer module, and a decoder; the encoder Em contains the first four layers of the pre-trained VGG-19, and the decoder structure is symmetrical to the encoder; when training the network, the encoder Em adds skip connections on ReLU1_1, ReLU2_1, and Relu3_1 to preserve the content details in the low-level feature maps; first, the encoder Em is extracted from the painting background image I s , Synthetic Image I c And foreground mask M, L = 4 layers of feature maps are extracted, which come from four encoder layers ReLU1_1, ReLU2_1, ReLU3_1 and ReLU4_1, respectively, to obtain background image features Synthetic Image Features and the resized foreground mask For the lth layer, the feature map and With the resized foreground mask M l are input into the attention style transfer module, which adjusts Statistics of the foreground area in the middle, making it The statistical information of Attention style transfer module is a key component that transfers local and global textures adaptively according to pattern repeatability; in order to measure the painting background image I s The degree of pattern repeatability in the paper considers two perspectives: 1) The similarity between each patch and the entire image, i.e., the repetitiveness of patches within the image; 2) the similarity between patches, i.e., inter-patch repetitiveness; For a given image I, features are collected from the lth (l=1,2,3,4)th encoder layer of encoder φ Then N feature patches are obtained by spatially dividing each feature with ratio r Right now In order to estimate the similarity between the local style and the global style, that is, the patch repetitiveness α within the image intra , measure each feature patch The Gram matrix of l The cosine similarity of is expressed as follows: In the formula, G represents the Gram matrix calculation, and L represents the number of convolutional layers in the encoder φ; In addition, the texture similarity between different patches, i.e., the inter-patch repetitiveness α, is calculated inter , to estimate whether a specific local pattern occurs frequently; this process is formalized as follows where β is derived from A set of selected pairs is selected from the pairs, which is effectively calculated with probability p, that is, Where N represents the number of feature blocks, Indicates the number of combinations of selecting 2 elements from N elements. This value represents the number of all possible feature block pairs and is used to estimate the repeatability of local patterns in the image. Finally, the pattern repeatability α style The calculation is as follows: For the lth layer of the encoder, the feature map and with the resized foreground mask M l are input together into the AdaIN layer, which adjusts Statistics of the foreground area in the middle, making it The statistical information of Stylized Attention Map It is calculated based on the feature maps output by the 3rd and 4th layers of the encoder. The formula is as follows: where h and Represent convolutional layers and upsampling respectively; Finally, calculate the final output of each layer of the encoder As shown below: The decoder is a symmetrical structure of VGG-19; the coordinated encoder features As the input of the decoder or connected with the decoder features through skip connection to obtain the final generator output; Discriminator Network Model D g The inputs are the painting background image I s And the harmonious image I output by the generator h , respectively, through seven convolutional layers and self-attention layers for feature extraction, and LeakyReLU layers are applied after all convolutional layers before the last convolutional layer to generate the probability D of the discriminator output prediction respectively. g (I s ) and D g (I h ) is used to calculate the adversarial loss.

4. The method for painting image harmony based on attention-generated adversarial network according to claim 3, characterized in that: Step 3: The generator and discriminator are optimized by alternating training, using a hinged version of the adversarial loss function, which is formulated as follows: Where D g is the discriminator, responsible for judging whether the input image is a painting background image or a harmonious image (generated by the generator G); when minimizing The loss function is minimized when the harmonious image can be accurately identified. The loss function encourages the generator to generate images that can fool the discriminator, thereby improving the realism of the generated images; The Adam optimizer is used for training with β1 = 0, β2 = 0.9; by default, the learning rate for the discriminator is 0.0004 and the learning rate for the generator is 0.0001; during training and testing, the input images are resized to 256 × 256.

5. The method for painting image harmony based on attention-generated adversarial network according to claim 4, characterized in that: Step 4: First, load the weights of the painting image harmony model trained in step 3 and update the parameters in the model; secondly, pass the image that needs to be harmonized, which is obtained by pasting the foreground image onto the background image according to the instance segmentation mask, as input data into the trained network model. The input data continuously updates the parameters through the generator and the discriminator, and finally the generator obtains the harmonized output image.

Images