Image generation method based on perceptual measurement guidance
By constructing a dual-branch evaluation network and a perceptual metric-guided image generation method, the problems of insufficient realism and loss of texture details in generated images are solved, achieving high perceptual quality and consistency in generated images.
Patent Information
- Application Number
- CN202511775159.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-02-06
AI Technical Summary
Existing technologies suffer from insufficient realism, loss of texture details, and poor consistency between pose and appearance when generating images, mainly due to the failure to effectively utilize perceptual feedback for optimization during the generation stage.
We construct an image generation method guided by perceptual metrics. We integrate multiple perceptual metrics through a dual-branch evaluation network, optimize the generation process using a ranking loss function, and combine a generative adversarial network and a pose feature transfer module to optimize the image feature transfer process.
The realism and detail integrity of the generated images are significantly improved, and the goals of the generation and evaluation stages are highly aligned, resulting in higher perceptual quality.
Smart Images
Figure CN121482828A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence, and in particular to an image generation method based on perceptual measurement guidance. Background Technology
[0002] With the rapid development of artificial intelligence technology, human pose transfer technology in the field of computer vision has become a research hotspot.
[0003] Pose transfer is a key image data augmentation technique and a fundamental processing step in many computer vision tasks such as image editing and pedestrian re-identification.
[0004] In the evaluation phase, existing methods typically use structural similarity (SSIM) and other perceptual metrics to assess the difference between the generated human image and the target human image. However, in the generation phase, mean squared error loss (MSELoss) and L1 norm loss are mainly used as optimization targets. This misalignment of targets prevents the generation model from directly utilizing the feedback of perceptual metrics to optimize the generation process, resulting in problems such as insufficient realism of the generated image, loss of texture details, and poor consistency between pose and appearance. Summary of the Invention
[0005] To address the problems mentioned in the background art, the present invention provides an image generation method based on perceptual measurement guidance, which solves the problems of insufficient realism, loss of texture details, and poor consistency between pose and appearance when generating images in the prior art.
[0006] The present invention achieves the above objectives through the following technical solutions:
[0007] The image generation method based on perceptual measurement guidance is characterized by including:
[0008] S1: Obtain a source image containing human targets, perform human pose estimation on the source image to obtain the pose corresponding to the source image, and determine the target pose to which the human targets in the source image need to be migrated.
[0009] S2: Based on the source image and target pose, generate target human images on a public pose transfer dataset using at least two pose transfer methods, and select at least two image perception quality evaluation metrics to calculate the quality score of the target human images.
[0010] Two images containing the same person and corresponding to the same target pose are selected from the generated target human images to form a pair of images. The pairs of images are prioritized according to their quality scores and labeled with corresponding sorting labels. The sorted dataset is constructed using the pairs of images and sorting labels.
[0011] S3: Use the paired images in the sorted dataset as input to the dual-branch evaluation network. With the training objective of "making the ranking score output by the network consistent with the quality score ranking of the paired images", the dual-branch evaluation network is trained using the ranking loss function. After training, the dual-branch evaluation network has the ability to output the quality ranking score of any two portrait images.
[0012] S4: Assemble the trained bi-branch evaluation network as an evaluator into the generative adversarial network PIGGAN;
[0013] During the training of the Generative Adversarial Network (PIGGAN), the input to PIGGAN includes the source image, the pose corresponding to the source image, the target pose, and real human image samples from the common pose transfer dataset. The generator network of PIGGAN outputs candidate images, the evaluator outputs a ranking score for the candidate images, a ranking loss function is constructed based on the ranking score, and an adversarial loss is constructed by combining the output of the discriminator of PIGGAN. The parameters of the generator network are jointly optimized through backpropagation.
[0014] S5: The optimized generative network takes the source image as the conditional image as input, and the pose corresponding to the source image and the target pose as input. Through its integrated pose feature transfer module and texture feature optimization module, it coordinates to optimize the image feature transfer process and outputs the human image corresponding to the target pose.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0016] This application constructs a dual-branch evaluation network (Evaluator) to integrate the evaluation logic of multiple perception indicators into the generation process. It uses ranking loss to directly guide the generator to optimize perception quality, making the goals of the generation stage and the evaluation stage highly unified in its method, and significantly improving the realism and detail integrity of the generated image. Attached Figure Description
[0017] Figure 1 This is a structural diagram of PIGGAN in this application;
[0018] Figure 2 This is a structural diagram of the Evaluator in this application;
[0019] Figure 3 These are sample images generated by PIGGAN based on Market-1501;
[0020] Figure 4 This is a schematic diagram comparing the image generated in this application with images generated by existing technologies on the public dataset Market-1501;
[0021] Figure 5This is a schematic diagram comparing images generated by this application with those generated by existing technologies on the public dataset DeepFashion. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.
[0024] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0025] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0026] Example 1
[0027] This embodiment uses one NVIDIA GTX4060ti GPU to run the experiments, employing the Deepfashion and Market-1501 datasets. The Deepfashion dataset (an in-store clothing retrieval benchmark) contains 52,172 in-store clothing images; we selected 101,966 image pairs as training data and 12,000 image pairs as validation data. The Market-1501 dataset contains 32,668 images of 1,501 individuals captured from six disjoint surveillance cameras. These images have a resolution of 128×64 pixels. 263,632 pairs were used for training, and 12,000 pairs were used for testing on the Market-1501 dataset.
[0028] like Figure 1 As shown, it includes the following steps:
[0029] Step 1: Pose extraction, for the input conditional image Human pose estimation is performed to obtain a pair of target poses (P) composed of 2D joints. C， P t ), output the generated image.
[0030] Step 2: Construct the sorted dataset, which includes the following sub-steps:
[0031] Step 2.1: Generate target human images on the Market-1501 and DeepFashion datasets using different pose transfer methods (such as PG², DSC, PATN).
[0032] Step 2.2: Apply the selected perceptual metric to the generated images. To obtain optimal performance, this application employs four perceptual metrics (SSIM, IS, DS, PCKh). Paired images are sorted according to the quality scores calculated based on the selected perceptual metric. The sorting logic is as follows: Given two images x 1, x2, the sorting score s is obtained in the following way. 1， s2,
[0033] ;
[0034] ;
[0035] in Let R(.) represent the network weights, R(.) represent the mapping function of the evaluator, and the ranking score can be expressed as:
[0036] ;
[0037] in and The quality scores of image x1 and image x2 are represented respectively. Candidate images of the same person and the same target pose are paired up. The better index is ranked as 1 and the worse index is ranked as -1, forming a ranked sample pair.
[0038] Step 3: Train the evaluator, such as Figure 2 The structure of the evaluator is shown in the figure. The evaluator is built based on a two-branch Siamese CNN network. Each branch of the network consists of fully connected layers, convolutional layers, Mish activation layers, and pooling layers. The ranked sample pairs obtained in step 2 are used as training data, and the following ranking loss function is used for training:
[0039] ;
[0040] ;
[0041] in and This represents the ranking score of the generated images. This represents the sorting label of the generated image; the margin z can be controlled. and The distance between them.
[0042] Step 4: Train PIGGAN, which consists of one generator and six discriminators;
[0043] The generator integrates the Pose Attention Transfer Block (PATB) and the Texture Attention Module (TAM). PATB is responsible for transferring information from the conditional image. Texture acquisition; the input to TAM is the input feature F obtained after processing by the PATB module. t-1 The model uses TAM to obtain a reasonable refined feature F. t It is responsible for guiding the texture transfer of image features.
[0044] The six discriminators are appearance discriminator D A Attitude discriminator D S Evaluator SSIM Evaluator IS Evaluator DS and Evaluator PCKh The appearance discriminator is used to determine whether the generated image and the conditional image are of the same person, and the pose discriminator is used to determine whether the generated image and the target pose are consistent. The latter four discriminators are trained based on corresponding perceptual indicators.
[0045] The generator's total loss function includes the following steps:
[0046] Step 4.1: GAN Loss ℒ GAN The calculation formula is:
[0047] ;
[0048] in For the expectation, It is the target posture. For attitude space, It is a conditional image. These are real target images from a public pose transfer dataset that match the person in the source image. It is the distribution of real human body images. For a fully convolutional discriminator, G(.) represents the generator. As the source posture, The distribution of human postures. The pose corresponding to the conditional image. Output the poses corresponding to the candidate images to the generator. It is the distribution of fake human body images. To generate an image.
[0049] Step 4.2: The formula for calculating the combL1 loss is:
[0050] ;
[0051] in , These are the weighting coefficients for the loss term. For the generated pose, For the target posture, The definition is as follows:
[0052] ;
[0053] in For the representation layer index, W, H, and C are respectively Ø ρ Height, depth, and width of space For the generated pose, For the target posture, For the first The weight coefficients of the layer feature loss, Ø ρ The term "a" represents the output of a certain layer, where "a certain layer" refers to a convolutional or fully connected layer in the PIGGAN generator used to extract pose features.
[0054] Step 4.3:
[0055] The complete loss function of the pose transfer network is:
[0056] + + ;
[0057] in , These are weighting coefficients. To combat the losses, Indicates the L1 loss of the combination. This represents the sorting loss function.
[0058] Step 4.4: Obtain the generated image, such as Figure 3 As shown, this is a sample image generated by PIGGAN based on Market-1501.
[0059] Step 5: Evaluate the final generated human image, such as... Figure 4 , Figure 5 As shown;
[0060] The obtained test set was used to test the algorithm's performance, with evaluation metrics including SSIM, DS, IS, and PCKh. Comparison algorithms included those by Siarohin et al., Ma et al., Esser et al., Zhu et al., Tang et al., Yang et al., and Li et al. Under sufficient training conditions, the algorithm was tested on the corresponding test set, and objective metrics were compared. The tests showed that the experimental evaluation metrics of the present invention are better, and the generated image quality is superior.
[0061] As shown in Table 1, on the Market-1501 dataset, the algorithm outperforms other comparative algorithms in terms of objective metrics such as SSIM, DS, IS, and PCKh.
[0062] As shown in Table 2, on the DeepFashions dataset, the algorithm outperforms other comparative algorithms in terms of objective metrics such as SSIM, DS, IS, and PCKh.
[0063] Table 1
[0064] Table 2
[0065] It is evident that this method achieves state-of-the-art performance in human pose transfer tasks, exhibiting superior discrimination capabilities and flexibility.
[0066] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.
Claims
1. An image generation method guided by perceptual measurement, characterized in that, include: S1: Obtain a source image containing human targets, perform human pose estimation on the source image to obtain the pose corresponding to the source image, and determine the target pose to which the human targets in the source image need to be migrated. S2: Based on the source image and target pose, generate target human images on a public pose transfer dataset using at least two pose transfer methods, and select at least two image perception quality evaluation metrics to calculate the quality score of the target human images. Two images containing the same person and corresponding to the same target pose are selected from the generated target human images to form a pair of images. The pairs of images are prioritized according to their quality scores and labeled with corresponding sorting labels. The sorted dataset is constructed using the pairs of images and sorting labels. S3: Use the paired images in the sorted dataset as input to the dual-branch evaluation network. With the training objective of "making the ranking score output by the network consistent with the quality score ranking of the paired images", the dual-branch evaluation network is trained using the ranking loss function. After training, the dual-branch evaluation network has the ability to output the quality ranking score of any two portrait images. S4: Assemble the trained bi-branch evaluation network as an evaluator into the generative adversarial network PIGGAN; During the training of the Generative Adversarial Network (PIGGAN), the input to PIGGAN includes the source image, the pose corresponding to the source image, the target pose, and real human image samples from the common pose transfer dataset. The generator network of PIGGAN outputs candidate images, the evaluator outputs a ranking score for the candidate images, a ranking loss function is constructed based on the ranking score, and an adversarial loss is constructed by combining the output of the discriminator of PIGGAN. The parameters of the generator network are jointly optimized through backpropagation. S5: The optimized generative network takes the source image as the conditional image as input, and the pose corresponding to the source image and the target pose as input. Through its integrated pose feature transfer module and texture feature optimization module, it coordinates to optimize the image feature transfer process and outputs the human image corresponding to the target pose.
2. The image generation method based on perceptual measurement guidance according to claim 1, characterized in that, The human pose estimation in S1 uses a 2D joint detection method. The poses corresponding to the source image and the target pose are both composed of 2D joint sets, and the target pose is consistent with the identity of the person in the source image.
3. The image generation method based on perceptual measurement guidance according to claim 1, characterized in that, The at least two attitude transfer methods include any two or more of PG², DSC, and PATN.
4. The image generation method based on perceptual measurement guidance according to claim 1, characterized in that, The at least two image perception quality assessment metrics mentioned in S2 include any two or more of the following: SSIM, IS, DS, and PCKh. The quality score of the target human image is calculated by weighted average of the four metrics, where SSIM is structural similarity, IS is inception score, DS is density score, and PCKh is keypoint detection accuracy.
5. The image generation method based on perceptual measurement guidance according to claim 1, characterized in that, The sorting labels mentioned in S2 are three-level numerical labels, including high-priority labels, medium-priority labels and low-priority labels. The high-priority label corresponds to the image with the highest quality score, and the value is 1. The medium priority label corresponds to images with quality scores between the highest and lowest, with a value of 10. The low-priority label corresponds to the image with the lowest quality score, with a value of 100.
6. The image generation method based on perceptual measurement guidance according to claim 1, characterized in that, In S3, the dual-branch evaluation network, which serves as the evaluator, is constructed based on a dual-branch Siamese CNN network. Each branch includes a fully connected layer, a convolutional layer, a Mish activation layer, and a pooling layer. Given two images x in a pair of images 1, x2, the sorting score s is obtained in the following way. 1， s2, ； ； in Let R(.) represent the network weights, R(.) represent the mapping function of the evaluator, and the ranking score can be expressed as: ； in and These represent the quality scores of image x1 and image x2, respectively.
7. The image generation method based on perceptual measurement guidance according to claim 6, characterized in that, The formula for calculating the ranking loss function is: ； ； This represents the sorting label of the generated images, where z is the margin. and The controllable distance between them.
8. The image generation method based on perceptual measurement guidance according to claim 7, characterized in that, The generative adversarial network PIGGAN consists of one generator and six discriminators; The six discriminators are appearance discriminator D A Attitude discriminator D S Evaluator SSIM Evaluator IS Evaluator DS and Evaluator PCKh Appearance discriminator D A The generated image and the conditional image are used to determine whether they are the same person. The pose discriminator is used to determine whether the generated image and the target pose are consistent. The other discriminators are trained based on the corresponding image perception quality evaluation index. The generator integrates the Pose Attention Transfer Block (PATB) and the Texture Attention Module (TAM). PATB is responsible for extracting the pose and appearance features of the person, while the input to TAM is the input feature F obtained after processing by the PATB module. t-1 The output is the refined feature F. t It is used to guide the texture transfer of image features.
9. The image generation method based on perceptual measurement guidance according to claim 8, characterized in that, Combating Losses GAN The calculation formula is: ； in For the expectation, It is the target posture. For attitude space, It is a conditional image. These are real target images from a public pose transfer dataset that match the person in the source image. It is the distribution of real human body images. For a fully convolutional discriminator, G(.) represents the generator. As the source posture, The distribution of human postures. The pose corresponding to the conditional image. Output the poses corresponding to the candidate images to the generator. It is the distribution of fake human body images. To generate an image.
10. The image generation method based on perceptual measurement guidance according to claim 9, characterized in that, The generative adversarial network PIGGAN also needs to calculate the combL1 loss, specifically: ； in , These are the weighting coefficients for the loss term. The definition is as follows: ； in For the representation layer index, W, H, and C are respectively Ø ρ Height, depth, and width of space For the first The weight coefficients of the layer feature loss, Ø ρ represents the output of a certain layer; The complete loss function for backpropagation is expressed as: + + ； in , These are weighting coefficients. This represents the sorting loss function.