Human body image generation method based on graph neural network
Through the human body image generation method based on graph neural network, the isotropic feature extraction module and graph information module are used to solve the limitations of traditional networks in capturing complex features of human body images, and achieve higher feature extraction accuracy and detailed capture capabilities. , achieving the most advanced generation performance.
Patent Information
- Application Number
- CN202411201472.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional convolutional neural networks and transformer networks have limitations in capturing irregular and complex human image features, making it difficult to effectively deal with non-uniform features and multi-scale attributes.
A human image generation method based on graph neural network was designed, including upsampling module, self-attention layer, isotropic feature extraction module, graph information module and graph neural network, to capture multi-scale features through graph convolution operation, and Generate human images of target poses through feature transformation and fusion.
The accuracy and flexibility of feature extraction are improved and the ability to capture details is enhanced. The experimental results show that the most advanced performance is achieved in the human image generation task, with higher discrimination ability and flexibility.
Smart Images

Figure CN118941670A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a method for generating human body images based on a graph neural network. Background Art
[0002] With the rapid development of artificial intelligence technology, human image generation (HIG) technology in the field of computer vision has become a research hotspot. Human image generation (HIG) is a valuable data augmentation technology and a fundamental step in many tasks, such as movie production and pedestrian re-identification. In order to extract pose and appearance representations from human images, existing methods usually use attention-based or transformer-based methods to fuse and update pose and appearance information.
[0003] However, traditional convolutional neural networks and transformer networks usually view images as grid or sequence structures, which limits their ability to capture irregular and complex objects. Summary of the invention
[0004] The purpose of the present invention is to design a human body image generation method based on graph neural network in order to solve the above problems.
[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:
[0006] The human image generation method based on graph neural network includes:
[0007] S1. Construct an initial human image generation model, which includes an upsampling module, a self-attention layer, an isotropic feature extraction module, a graph information module, a graph neural network and a downsampling module. The isotropic feature extraction module is used to extract the features of the human image; the graph information module captures multi-scale features through graph convolution operations; the graph neural network is used to perform feature conversion and fusion on the features of the human image and the multi-scale features to generate a human image of the target posture; the output of the upsampling module is used as the input of the self-attention layer, the output of the self-attention layer is used as the input of the isotropic feature extraction module and the input of the graph information module respectively, the output of the isotropic feature extraction module and the output of the graph information module are both used as the input of the graph neural network, the output of the graph neural network is used as the input of the downsampling module, and the output of the downsampling module is the generated human figure;
[0008] S2, obtain the experimental data set;
[0009] S3. Experimental dataset imports the initial human image generation model and constructs the loss function L fullAs the constraint condition of the human image generation model, the optimized human image generation model is obtained through training optimization through back propagation and gradient descent algorithm;
[0010] S4, obtaining a human body image to be generated;
[0011] S5. Generate a human figure of a human image to be generated using the optimized human image generation model.
[0012] The beneficial effects of the present invention are as follows: the isotropic feature extraction module and the graphic information module are used to effectively process the non-uniform features and multi-scale attributes of human images; the hierarchical structure of the isotropic feature extraction module matches the heterogeneous components of a person (such as arms, legs, body and head), which improves the accuracy and flexibility of feature extraction; the design of the graphic information module encourages the conversion of features between different levels, and enhances the ability to capture details, including clothing textures, etc. Experimental results show that the method of the present invention achieves state-of-the-art performance in the task of human image generation, with higher discrimination ability and flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a schematic diagram of the comparison between the original human body image, the sequence image and the image represented by the graph structure;
[0014] Figure 2 It is a structural schematic diagram of a human body image generation method based on a graph neural network of the present invention;
[0015] Figure 3 It is a schematic diagram of the comparison between the images generated by the present invention and the images generated by the prior art on the public dataset DeepFashion. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0017] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0018] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0019] In the description of the present invention, it should be understood that the terms "upper", "lower", "inside", "outside", "left", "right", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the product of the invention is usually placed when in use, or are directions or positional relationships commonly understood by those skilled in the art. These directions or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.
[0020] Furthermore, the terms “first”, “second”, etc. are merely used for distinguishing descriptions and should not be understood as indicating or implying relative importance.
[0021] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0022] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.
[0023] like Figure 2 As shown, the human image generation method based on graph neural network includes:
[0024] S1. Construct an initial human image generation model. The human image generation model includes an upsampling module, a self-attention layer, an isotropic feature extraction module, a graph information module, a graph neural network and a downsampling module. The isotropic feature extraction module is used to extract the features of the human image; the graph information module captures multi-scale features through graph convolution operations; the graph neural network is used to perform feature conversion and fusion on the features of the human image and the multi-scale features to generate a human image of the target posture; the output of the upsampling module is used as the input of the self-attention layer, and the output of the self-attention layer is used as the input of the isotropic feature extraction module and the input of the graph information module respectively, the output of the isotropic feature extraction module and the output of the graph information module are both used as the input of the graph neural network, the output of the graph neural network is used as the input of the downsampling module, and the output of the downsampling module is the generated human figure. The upsampling module includes two upsampling layers, and the downsampling module includes two downsampling layers;
[0025] The isotropic feature extraction module includes the first padding layer, the first graph convolution layer, the first normalization layer, the first activation layer, the second padding layer, the second graph convolution layer and the second normalization layer from input to output. The first padding layer and the second padding layer are used to increase the boundary of the input feature map; the first graph convolution layer and the second graph convolution layer are used to learn the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood; the first normalization layer and the second normalization layer are used to perform normalization processing to ensure that the outputs of different layers have a stable distribution; the first activation layer introduces nonlinearity to ensure that the gradient flow is maintained between positive and negative eigenvalues;
[0026] The graph information module consists of the third graph convolution layer, the first batch normalization layer, the second activation layer, the fourth graph convolution layer and the second batch normalization layer. The third graph convolution layer is used to learn the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood. The first batch normalization layer and the second batch normalization layer are used to normalize the two-dimensional feature graph. The second activation layer introduces nonlinearity to ensure that the gradient flow is maintained between positive and negative feature values. The fourth graph convolution layer performs graph convolution operations to further capture the complex patterns and relationships in the graph.
[0027] S2. Obtain the experimental dataset.
[0028] S3. Experimental dataset imports the initial human image generation model and constructs the loss function L full As the constraint condition of the human image generation model, the optimized human image generation model is obtained through training optimization through back propagation and gradient descent algorithm; the loss function L full It is expressed as: , where L GAN Is to distinguish between real data and generated data, including generator loss and discriminator loss; L percp is the perceptual loss, which measures how visually similar the generated image is to the real image; L L1 The loss is the mean absolute error loss, which measures the pixel-wise difference between the generated image and the real image. , and Respectively represent L GAN , L percp , L L1 For L full The weight contribution of .
[0029] S4. Obtain a human body image to be generated.
[0030] S5. Generate a human figure of a human image to be generated using the optimized human image generation model.
[0031] Example 1
[0032] This example uses an NVIDIA GTX4060ti GPU to run the experiment. The training data is the DeepFashion dataset collected from the Internet. The dataset contains 200,000 image pairs, and the size of all images is 256×256 pixels. We select 101,966 pairs of images as training data and 12,000 pairs of images as verification data.
[0033] like Figure 2 As shown, the following steps are included:
[0034] Step 1: Preprocessing of human image data. This step uses a network framework for image parsing and feature extraction, including the following sub-steps:
[0035] Step 1.1: Divide the human image dataset into training set and test set.
[0036] Step 1.2: Use the upsample module to enlarge the original image and enhance the image quality.
[0037] Step 1.3: The self-attention layer extracts human image features and converts the input human image into a graph structure, where each key area of the image is regarded as a node in the graph, and the relationship between nodes is represented by edges, such as Figure 1 shown.
[0038] like Figure 1 Shown, graph-structured representation of human images.
[0039] Step 2: Isotropic Feature Extraction Module IFEM: The isotropic feature extraction module IFEM extracts the features of the human body image in a hierarchical manner to adapt to the non-uniform characteristics of the human body structure. It specifically includes the following sub-steps:
[0040] Step 2.1: The first padding layer ReflectionPad2d is a padding operation used to increase the boundary of the input feature map before the two-dimensional convolution layer. It maintains the boundary information of the feature map and reduces the unnatural changes introduced by padding.
[0041] Step 2.2: The first graph convolution layer learns the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood.
[0042] Step 2.3: The first normalization layer Layer Norm is used to normalize the activation output in the neural network to ensure that the outputs of different layers have a stable distribution. It ensures that different layers in the network can learn effective feature representations and reduce internal covariate shift.
[0043] Step 2.4: The first activation layer LeakyRelu is an activation function that introduces nonlinearity while ensuring that the gradient flows between positive and negative eigenvalues.
[0044] Step 2.5: The second padding layer ReflectionPad2d again - the second graph convolution layer GraphConvolution - the second normalization layer Layer Norm Through multi-layer processing, the feature representation of the node is further refined and enriched to capture more complex graph structure patterns.
[0045] Step 3: Graph Information Module GIM: The Graph Information Module GIM works together with the Isotropic Feature Extraction Module IFEM to capture multi-scale features through graph convolution operations and enhance the expressiveness of features. Specifically, it includes:
[0046] Step 3.1: The third graph convolution layer Graph Convolution learns the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood.
[0047] Step 3.2: First batch normalization layer BatchNorm2d is a technique used to improve training speed, stability, and network generalization ability. BatchNorm2d is commonly used in convolutional neural networks to normalize two-dimensional feature maps and reduce internal covariate shift.
[0048] Step 3.3: The second activation layer LeakyRelu is an activation function that introduces nonlinearity while ensuring that the gradient flows between positive and negative eigenvalues to avoid neuron death problems.
[0049] Step 3.4: The fourth graph convolution layer Graph Convolution After applying the activation function, another graph convolution operation can further capture the complex patterns and relationships in the graph. This continuous graph convolution layer can deepen the network and enable the model to learn higher-level abstract features.
[0050] Step 3.5: The second batch normalization layer BatchNorm2d uses batch normalization again after the fourth graph convolutional layer to continue to stabilize the learning process of the network, help the features maintain a stable distribution during training, and accelerate convergence.
[0051] Step 4: Feature conversion and fusion: Using the features extracted by the isotropic feature extraction module IFEM and the graph information module GIM, feature conversion and fusion are performed through the graph neural network to generate a human body image of the target posture.
[0052] Construct the loss function L based on the human image Input and the generated human figure G full :
[0053]
[0054] Loss function L full As the constraint of the model, through back propagation and gradient descent algorithm, minimize L full , thereby training the graph neural network model.
[0055] The downsampling module subsamples the image so that it conforms to the size of the display area and generates a thumbnail of the corresponding image.
[0056] Step 5: Evaluation of human image generation performance, such as Figure 3 As shown;
[0057] Step 5.1: Use the test set obtained in step 1.1 to test the algorithm performance. The evaluation indicators during the test are SSIM, DS, IS, and PCKh.
[0058] Step 5.2, the comparison algorithms are Siarohin et al., Ma et al., Esser et al., Zhu et al., Tang et al.. After sufficient training, the corresponding test set is tested to compare the objective indicators. The test shows that the experimental evaluation indicators of the present invention are better and the generated image effect is better.
[0059] As shown in Table 1, in terms of objective indicators such as SSIM, DS, IS, and PCKh, it is generally better than other comparison algorithms.
[0060] Table 1 Objective indicators of human image generation methods based on graph neural networks
[0061]
[0062] It can be seen that this method has achieved state-of-the-art performance in the task of human image generation and has higher discriminative ability and flexibility.
[0063] The technical solution of the present invention is not limited to the above-mentioned specific embodiments. All technical variations made according to the technical solution of the present invention fall within the protection scope of the present invention.
Claims
1. A human image generation method based on graph neural network, characterized in that: include: S1. Construct an initial human image generation model, which includes an upsampling module, a self-attention layer, an isotropic feature extraction module, a graph information module, a graph neural network and a downsampling module. The isotropic feature extraction module is used to extract the features of the human image; the graph information module captures multi-scale features through graph convolution operations; the graph neural network is used to perform feature conversion and fusion on the features of the human image and the multi-scale features to generate a human image of the target posture; the output of the upsampling module is used as the input of the self-attention layer, the output of the self-attention layer is used as the input of the isotropic feature extraction module and the input of the graph information module respectively, the output of the isotropic feature extraction module and the output of the graph information module are both used as the input of the graph neural network, the output of the graph neural network is used as the input of the downsampling module, and the output of the downsampling module is the generated human figure; S2, obtain the experimental data set; S3. Experimental dataset imports the initial human image generation model and constructs the loss function L full As the constraint condition of the human image generation model, the optimized human image generation model is obtained through training optimization through back propagation and gradient descent algorithm; S4, obtaining a human body image to be generated; S5. Generate a human figure of a human image to be generated using the optimized human image generation model.
2. The human body image generation method based on graph neural network according to claim 1 is characterized in that: The isotropic feature extraction module includes, from input to output, a first padding layer, a first graph convolution layer, a first normalization layer, a first activation layer, a second padding layer, a second graph convolution layer and a second normalization layer. The first padding layer and the second padding layer are used to increase the boundary of the input feature map; the first graph convolution layer and the second graph convolution layer are used to learn the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood; the first normalization layer and the second normalization layer are used to perform normalization processing to ensure that the outputs of different layers have a stable distribution; the first activation layer introduces nonlinearity to ensure that the gradient flow is maintained between positive and negative eigenvalues.
3. The human body image generation method based on graph neural network according to claim 1 is characterized in that: The graph information module consists of the third graph convolution layer, the first batch normalization layer, the second activation layer, the fourth graph convolution layer and the second batch normalization layer. The third graph convolution layer is used to learn the representation of nodes on the graph structure so that the node features contain the information of their local neighborhood; the first batch normalization layer and the second batch normalization layer are used to normalize the two-dimensional feature map; the second activation layer introduces nonlinearity to ensure that the gradient flows between positive and negative eigenvalues; the fourth graph convolution layer performs graph convolution operations to further capture the complex patterns and relationships in the graph.
4. The human body image generation method based on graph neural network according to claim 1, characterized in that: Loss function L full It is expressed as: , where L GAN Is to distinguish between real data and generated data, including generator loss and discriminator loss; L percp is the perceptual loss, which measures how visually similar the generated image is to the real image; L L1 The loss is the mean absolute error loss, which measures the pixel-wise difference between the generated image and the real image. , and Respectively represent L GAN , L percp , L L1 For L full The weight contribution of .
