A pose-guided character image synthesis method based on transformer

Abstract

The present invention discloses a method for synthesizing character images guided by a posture of a transformer. The method first preprocesses multiple source character images and multiple target posture images to construct an image dataset. A hypergraph prediction transformer model framework is then constructed, comprising a multi-scale feature extraction module, a hypergraph learning module, and a fusion prediction module. Finally, the hypergraph prediction transformer is trained to obtain a trained hypergraph prediction transformer, which is then used to synthesize character images guided by the target posture to obtain a synthesized character image. The method effectively integrates multimodal information (character posture, target posture, and background information) to improve the quality and detail of the generated image, thereby generating a more refined posture-guided synthesized character image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a posture-guided character image synthesis method based on a transformer. Background Art

[0002] With the rapid development of deep learning and computer vision, image synthesis has been widely used in various fields. Pose-guided person image generation has become an important research direction, particularly in virtual character generation and augmented reality (AR). Most existing image synthesis methods are based on traditional generative adversarial networks (GANs) or autoencoders. While these methods have made some progress, they still face challenges, particularly in achieving pose consistency and detail quality in the generated images.

[0003] Many existing methods rely on simple convolutional neural network (CNN) architectures for image generation. These methods fail to effectively capture the relationship between pose variations and person poses, resulting in inconsistencies between the poses in the generated images and the target images. Furthermore, traditional image synthesis methods often ignore high-order relationships between multimodal data, resulting in poor visual quality in the synthesized results. In particular, the generated images often exhibit blurriness and distortion, particularly in terms of texture detail and realism.

[0004] To overcome these challenges, researchers have attempted to introduce transformer models for image generation. Compared to traditional generative adversarial networks, transformer models can better learn data distribution in the latent space, thereby generating more diverse and high-quality images. However, existing transformer models primarily focus on a single data source and lack the ability to effectively integrate multiple information, such as the person's pose, background details, and texture details, thus limiting the effectiveness of generated images.

[0005] In recent years, the rise of hypergraph learning and transformer technologies has provided new insights into multimodal information processing in image synthesis. Hypergraph learning effectively captures multi-layered relationships in data, while transformer architectures are capable of processing long-range dependencies, making them suitable for complex image generation tasks. However, existing models do not effectively combine the strengths of these two approaches, resulting in insufficient accuracy and loss of texture details in complex image synthesis tasks, particularly in pose-guided person image synthesis (PGPIS). Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a posture-guided character image synthesis method based on a transformer, which effectively integrates multimodal information (character posture, target posture and background information) to improve the quality and detail expression of the generated image, thereby generating a more refined posture-guided character synthetic image.

[0007] The technical solution of the present invention is:

[0008] A method for synthesizing a character image guided by a posture of a transformer, comprising the following steps:

[0009] (1) Preprocessing a plurality of source person images with different person poses and different backgrounds, and a plurality of target pose images, to construct an image dataset, and selecting any source person image and any target pose image in the image dataset to combine and obtain a paired data set;

[0010] (2) Constructing the model framework of the hypergraph prediction transformer, including a multi-scale feature extraction module, a hypergraph learning module, and a fusion prediction module;

[0011] The multi-scale feature extraction module performs multi-scale feature extraction on the source person image and the target posture image respectively, and obtains the feature information of the source person image and the target posture image at different scales;

[0012] Based on the feature information extracted by the multi-scale feature extraction module, the hypergraph learning module is used to construct the source person hypergraph and the target posture hypergraph, and the hypergraph representation learning is performed separately;

[0013] The fusion prediction module performs multi-scale feature fusion on the character image features learned from the hypergraph and the target pose image features, and enables the character to learn the target pose, generating a synthetic image of the character guided by the target pose;

[0014] (3) The hypergraph prediction transformer is trained to obtain a trained hypergraph prediction transformer, and the trained hypergraph prediction transformer is used to synthesize a character image under the guidance of a target posture to obtain a character synthesized image.

[0015] The source character image is a character posture image under a real background.

[0016] The image preprocessing is to first perform uniform resizing and normalization on the input image data, adjust the pixel values ​​of the image to the range of [0, 1], and then perform data enhancement to increase the diversity of the data set.

[0017] The multi-scale feature extraction module includes a Vision Transformer model trained based on the DINO model and a VAE feature extraction model. Specifically, the target posture image uses the Vision Transformer model trained based on the DINO model to perform multi-scale feature extraction of the target posture, and obtains the target posture feature information at different scales; the source person image uses the Vision Transformer model trained based on the DINO model to perform multi-scale feature extraction of the person posture, and obtains the person posture feature information at different scales; the source person image uses the VAE feature extraction model to perform multi-scale feature extraction of the background, and obtains the background feature information at different scales.

[0018] The construction of the source character hypergraph is to use the character posture feature information corresponding to each pixel point in the source character hypergraph as a node to construct a hypergraph; the hypergraph representation learning of the source character hypergraph is to extract the positional relationship between different nodes of the source character hypergraph and learn the relative positional relationship of the character posture.

[0019] The target posture hypergraph is constructed by taking the target posture feature information corresponding to each pixel point in the target posture hypergraph as a node to construct a hypergraph; the hypergraph representation learning of the target posture hypergraph is to extract the positional relationship between different nodes of the target posture hypergraph and learn the relative positional relationship of the target posture.

[0020] The fusion prediction module includes a first-layer multi-layer perceptron, a residual module, a second-layer multi-layer perceptron and a decoder. The character image features and target posture image features after hypergraph learning are input into the first-layer multi-layer perceptron for training and learning, so that the character learns the target posture. Then, the output of the first-layer multi-layer perceptron and the background feature information extracted by the VAE feature extraction model are residually connected using the residual module. Finally, after processing by the second-layer multi-layer perceptron, the detailed features of the character composite image are amplified in the background image. Finally, after decoding by the decoder, a character composite image guided by the target posture is obtained.

[0021] Advantages of the present invention:

[0022] (1) The hypergraph prediction transformer of the present invention extracts image features of different scales through a multi-scale feature extraction module, so that the model can fully capture the details and overall structure of the image, thereby generating a more natural synthetic image.

[0023] (2) The hypergraph prediction transformer of the present invention captures the high-order correlation information between different elements in each image by constructing source character images and target posture images and using a hypergraph learning module to perform hypergraph representation learning, so that the model can more accurately control the relationship between the character's posture, posture and background during the synthesis process, thereby generating a more natural and consistent character synthesis image.

[0024] (3) The hypergraph prediction transformer of the present invention ensures the effective combination of information of different scales in the image generation process by fusing the prediction module, so that the generated image retains the detail information and has strong overall consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a framework diagram of the hypergraph prediction converter of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] A method for synthesizing a character image guided by a posture of a transformer, comprising the following steps:

[0028] (1) Preprocess multiple source person images with different person poses and different backgrounds, and multiple target pose images to construct an image dataset, and select any source person image and any target pose image in the image dataset to obtain a paired data set; wherein, the source person image is a person pose image under a real background, and the image preprocessing is to first uniformly resize and normalize the input image data, adjust the pixel value of the image to the range of [0,1], and then perform data enhancement (random rotation, scaling, cropping, flipping, etc.) to increase the diversity of the dataset;

[0029] The image dataset is divided into a training set and a test set in a ratio of 8:2, and both the training set and the test set include multiple pairs of data groups;

[0030] (2) See Figure 1 ,construct the model framework of hypergraph prediction transformer, including multi-scale feature extraction module, hypergraph learning module and fusion prediction module;

[0031] The framework structure of the hypergraph prediction transformer is expressed as follows:

[0032]

[0033] In formula (1), q represents the conditional probability distribution, represents a normal distribution parameterized by mean u and covariance function Σ, X p represents the target pose image, X srepresents the source person image, t represents the time step of the hypergraph prediction transformer, X t Represents the composite images of people at different time steps;

[0034] S21, the multi-scale feature extraction module includes a Vision Transformer model trained based on the DINO model and a VAE feature extraction model, specifically: the target posture image uses the Vision Transformer model (ViT-B) trained based on the DINO model to perform multi-scale feature extraction of the target posture, and obtains the target posture feature information at different scales; the source person image uses the Vision Transformer model (ViT-G) trained based on the DINO model to perform multi-scale feature extraction of the person posture, and obtains the person posture feature information at different scales; the source person image uses the VAE feature extraction model (VAEEnc. model) to perform multi-scale feature extraction of the background, and obtains the background feature information at different scales;

[0035] S22. Based on the feature information extracted by the multi-scale feature extraction module, a source character hypergraph and a target posture hypergraph are constructed using a hypergraph learning module. The source character hypergraph is constructed by using the character posture feature information corresponding to each pixel in the source character hypergraph as a node to construct a hypergraph; the target posture hypergraph is constructed by using the target posture feature information corresponding to each pixel in the target posture hypergraph as a node to construct a hypergraph; then, the source character hypergraph is subjected to hypergraph representation learning, the positional relationship between different nodes of the source character hypergraph is extracted, and the relative positional relationship of the character posture is learned; the target posture hypergraph is subjected to hypergraph representation learning, the positional relationship between different nodes of the target posture hypergraph is extracted, and the relative positional relationship of the target posture is learned;

[0036] In a hypergraph, spatial convolution propagates information through hyperedges. Each node in the hypergraph forms a complex coupling relationship with multiple other nodes through hyperedges. The information propagation from node to hyperedge is defined as follows:

[0037]

[0038] In formula (2), z e The eigenvector representing the hyperedge e is obtained by aggregating information from nodes to hyperedges; is the number of nodes associated with the hyperedge e, indicating the degree of the hyperedge; Is a vector of all 1s, used to sum the features of each node; X is the node feature matrix, each row represents the feature of a node; H e,: represents the indicator vector of all nodes associated with the hyperedge e; W is a learnable weight matrix for feature transformation; H v,: represents the indicator vector of all hyperedges associated with node v; represents the number of hyperedges associated with node v (the degree of the node); U represents the update matrix, which is used for feature update; x v represents the features of node v before update; x v ′ represents the updated features of node v;

[0039] S23, the fusion prediction module performs multi-scale feature fusion on the character image features and the target posture image features after hypergraph learning, and enables the character to learn the target posture, thereby generating a character composite image guided by the target posture; the fusion prediction module includes a first-layer multi-layer perceptron, a residual module, a second-layer multi-layer perceptron and a decoder, the character image features and the target posture image features after hypergraph learning are input into the first-layer multi-layer perceptron (MLP) for training and learning, so that the character learns the target posture, and then the output of the first-layer multi-layer perceptron and the background feature information extracted by the VAE feature extraction model are residually connected using the residual module, and finally processed by the second-layer multi-layer perceptron (MLP), the detail features of the character composite image are amplified in the background image, and finally decoded by the decoder (Dec.) to obtain the character composite image guided by the target posture;

[0040] S3. Use the training set to train the hypergraph prediction transformer and the test set to test it to obtain a trained hypergraph prediction transformer. Use the trained hypergraph prediction transformer to synthesize a character image under the guidance of the target posture to obtain a synthesized character image.

[0041] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for synthesizing human images guided by posture based on a transformer, characterized by: The specific steps include: (1) Preprocessing a plurality of source person images with different person poses and different backgrounds, and a plurality of target pose images, to construct an image dataset, and selecting any source person image and any target pose image in the image dataset to combine and obtain a paired data set; (2) Constructing the model framework of the hypergraph prediction transformer, including a multi-scale feature extraction module, a hypergraph learning module, and a fusion prediction module; The multi-scale feature extraction module performs multi-scale feature extraction on the source person image and the target posture image respectively, and obtains the feature information of the source person image and the target posture image at different scales; The multi-scale feature extraction module includes a Vision Transformer model trained based on the DINO model and a VAE feature extraction model. Specifically, the target pose image uses the Vision Transformer model trained based on the DINO model to extract multi-scale features of the target pose, and obtains target pose feature information at different scales; the source person image uses the Vision Transformer model trained based on the DINO model to extract multi-scale features of the person pose, and obtains person pose feature information at different scales; the source person image uses the VAE feature extraction model to extract multi-scale features of the background, and obtains background feature information at different scales; Based on the feature information extracted by the multi-scale feature extraction module, the hypergraph learning module is used to construct the source person hypergraph and the target posture hypergraph, and the hypergraph representation learning is performed separately; The source character hypergraph is constructed by taking the character posture feature information corresponding to each pixel in the source character hypergraph as a node to construct a hypergraph; the hypergraph representation learning of the source character hypergraph is to extract the positional relationship between different nodes of the source character hypergraph and learn the relative positional relationship of the character posture; The fusion prediction module performs multi-scale feature fusion on the character image features learned from the hypergraph and the target pose image features, and enables the character to learn the target pose, generating a synthetic image of the character guided by the target pose; (3) The hypergraph prediction transformer is trained to obtain a trained hypergraph prediction transformer, and the trained hypergraph prediction transformer is used to synthesize a character image under the guidance of a target posture to obtain a character synthesized image.

2. The method for synthesizing a character image guided by a posture of a transformer according to claim 1, characterized in that: The source character image is a character posture image under a real background.

3. The method for synthesizing a character image guided by a posture of a transformer according to claim 1, characterized in that: The image preprocessing is to first perform uniform resizing and normalization on the input image data, adjust the pixel values ​​of the image to the range of [0, 1], and then perform data enhancement to increase the diversity of the data set.

4. The method for synthesizing a character image guided by a posture of a transformer according to claim 1, wherein: The target posture hypergraph is constructed by taking the target posture feature information corresponding to each pixel point in the target posture hypergraph as a node to construct a hypergraph; the hypergraph representation learning of the target posture hypergraph is to extract the positional relationship between different nodes of the target posture hypergraph and learn the relative positional relationship of the target posture.

5. The method for synthesizing a character image guided by a posture of a transformer according to claim 4, characterized in that: The fusion prediction module includes a first-layer multi-layer perceptron, a residual module, a second-layer multi-layer perceptron and a decoder. The character image features and target posture image features after hypergraph learning are input into the first-layer multi-layer perceptron for training and learning, so that the character learns the target posture. Then, the output of the first-layer multi-layer perceptron and the background feature information extracted by the VAE feature extraction model are residually connected using the residual module. Finally, after processing by the second-layer multi-layer perceptron, the detailed features of the character composite image are amplified in the background image. Finally, after decoding by the decoder, a character composite image guided by the target posture is obtained.

Images