Method for generating multiple hands of characters in text-to-image synthesis based on region perception

Abstract

This invention belongs to the field of image generation technology, specifically a method for optimizing the generation of hands in multi-person text-to-image synthesis based on region awareness. The invention proposes a Region Aware Group Relative Policy Optimization (RA-GRPO) framework, which, by introducing a spatial mask reward mechanism, positions policy optimization to the detected hand regions, achieving targeted anatomical structure optimization while maintaining overall image quality. The invention designs a Keypoint Confidence Reward (KCR) based on hand pose estimation, using HRNet to predict hand keypoint heatmaps and calculate confidence scores to quantitatively evaluate the structural correctness of the generated hands. In each reinforcement learning iteration, the model generates N images for each prompt text, obtains reward scores through a reward model, calculates the advantage value within the same prompt group using normalization, and finally obtains the region-aware policy loss through mask weighting. Training results show that this method can reduce the probability of deformed generated hands.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and natural language processing technology. Specifically, it relates to an optimized method for generating hands of multiple people in text-to-image synthesis based on region awareness. Background Technology

[0002] In recent years, text-to-image generation technology has made significant progress with the development of diffusion models. However, generating realistic human hands remains an ongoing challenge, mainly due to the complex joint structures and high structural sensitivity of hands, especially in images containing multiple individuals. Existing state-of-the-art models such as FLUX, QwenImage, and HunyuanImage, while capable of generating highly realistic images, still have shortcomings in hand generation.

[0003] To address the problem of hand generation, researchers have proposed several methods. Hand1000 optimizes the diffusion model by incorporating 2D hand information into text embeddings [Zhang, Haozhuo, et al. "Hand1000: Generating realistic hands from text with only 1,000 images." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 9. 2025]; Handiffuser encodes 3D hand mesh information and trains a dedicated encoder [Narasimhaswamy, Supreeth, et al. "Handiffuser: Text-to-image generation with realistic hand appearances." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2024]; HandRefiner uses a hand mesh reconstruction model to correct malformed hands within a conditional inpainting framework [Lu, Wenquan, et al. "Handrefiner: Refining malformed hands in generated images by diffusion-based conditional inpainting." Proceedings of the 32nd ACM International Conference on Multimedia].

[2024] ; Attentionhand enhances hand generation through text and visual attention mechanisms [Park, Junho, Kyeongbo Kong, and Suk-Ju Kang. "AttentionHand: Text-driven controllable hand image generation for 3D hand reconstruction in the wild." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024]; RHanD employs a decoupled, mesh-based guided and style-preserving two-stage conditional diffusion framework [Wang, Chengrui, et al.]."Rhands: Refining malformed hands for generated images with decoupled structure and style guidance." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 7. 2025]. .

[0004] However, these methods have obvious limitations: they often rely on the introduction of auxiliary control mechanisms (such as depth maps or dedicated encoding), which increases model complexity and computational overhead; at the same time, existing methods mainly focus on single-person scenes, and it is difficult to maintain realism and structural correctness when processing images containing multiple people's hands, especially when the hands overlap or interact closely, existing technologies cannot accurately represent complex occlusion relationships and spatial layouts.

[0005] Reinforcement learning has been proven effective in improving the quality of generated images. The Group Relative Policy Optimization (GRPO) framework [Xue, Zeyue, et al. "Dancegrpo: Unleashing grpo on visual generation." arXiv preprint arXiv:2505.07818 (2025)] was initially used for language model alignment and has since been applied to visual generation tasks. However, traditional GRPO performs uniform optimization across the entire image, which can lead to unintended modifications to non-hand regions in hand generation tasks, degrading the overall image quality. Therefore, there is a pressing need in this field for a method that can accurately optimize the hand region while maintaining overall image quality, especially for hand generation problems in multi-person scenes. Summary of the Invention

[0006] This invention aims to address the problems existing in the prior art by providing a region-aware group relative policy optimization (RA-GRPO) method. By focusing policy optimization on the hand region, it directly enhances the hand generation capability of the diffusion model. This invention is particularly suitable for the refined generation of hand structures in complex scenes containing multiple characters.

[0007] The present invention is achieved through the following technical solution.

[0008] This invention provides an optimized method for generating hands of multiple people in text-to-image synthesis based on region awareness, comprising the following steps:

[0009] Step 1: Construct a multi-person hand gesture dataset, containing multiple image-text pairs;

[0010] Step 2: Use a multi-person hand gesture dataset to perform supervised fine-tuning of the basic diffusion model, so that it initially has the ability to generate hands for multiple people;

[0011] Step 3: Perform group relative policy optimization (GRPO) reinforcement learning on the supervised fine-tuned model;

[0012] In each iteration, for each prompt text, multiple images are generated by denoising multiple initial noise vectors;

[0013] The hand detection module identifies the hand region in the generated image and generates a corresponding spatial mask.

[0014] For each identified hand region, a hand quality reward value is obtained through a hand reward model. Images generated with the same text but different initial noise are considered as the same cue group. Within the same cue group, all hand quality reward values ​​are normalized, and the advantage value of each hand region is calculated.

[0015] The spatial mask is downsampled to match the spatial resolution of the latent representation of the variational autoencoder;

[0016] The reverse diffusion process based on the diffusion model is used to calculate the transition log probability.

[0017] The downsampled mask is multiplied element-wise with the transition log probability to obtain the localized transition log probability;

[0018] Based on the localized transition log probabilities and corresponding dominance values, the strategy loss is calculated, the model parameters are updated, and the model's performance in hand generation is improved, resulting in better hand generation and achieving the effect of hand generation optimization.

[0019] In this invention, step one, the method for constructing a multi-person hand gesture dataset, includes:

[0020] A large language model is used to generate diverse descriptors covering subject features P, clothing attributes A, and hand gestures H; the number of people in each image is also considered. Limited to 1-4 people, these descriptors are randomly sampled and combined to systematically construct prompt text, with the template as follows: "There are..." The image shows a person wearing a hand gesture. The text prompt is: "P wearing A is giving a hand gesture H;...". After obtaining the first batch of generated text prompt data, images are generated using an industry-leading (State-of-the-Art, SOTA) generative model. After generation, images are manually selected to remove those with obvious distortions in the hands or body parts, resulting in the final image dataset. This dataset is then divided into training and testing sets according to a set ratio for use. In a specific embodiment, the large language model is GPT-4o, and the generative models include FLUX.1-dev and Gemini 2.5 Pro.

[0021] In this invention, in step two, the basic diffusion model is Stable Diffusion3; during supervised fine-tuning, it is trained using LORA, and the mean square error (MSE) between the predicted noise and the actual noise is minimized during training to minimize the difference between the predicted value and the actual value.

[0022] In this invention, in step three, the hand reward model is a keypoint confidence reward KCR model based on CNN-based hand pose estimation. For each hand region, a keypoint heatmap is predicted using a hand pose estimator, and keypoint confidence scores are extracted from the heatmap. The hand quality reward value is obtained by calculating the average of the confidence scores of each keypoint, using the following formula:

[0023]

[0024] in, For the first Confidence level of key points on the hand.

[0025] In this invention, in step three, the advantage value of each hand area within the same prompt group is calculated using the following formula:

[0026]

[0027] in, The reward value for the i-th hand region corresponds to the hand. , where n is the number of hands generated within the same prompt group.

[0028] In this invention, in step three, the transition log probability is calculated using the following formula:

[0029]

[0030] in, For the potential state sampled at the next time step, For the deterministic next state predicted by the model, To explore the standard deviation of injected noise;

[0031] The log probability of localization transitions is calculated using the following formula:

[0032]

[0033] in, This is the mask after downsampling. This indicates element-wise multiplication.

[0034] In this invention, in step three, the loss function of GRPO adopts the PPO strategy loss with pruning, which multiplies the localized transition log probability and the corresponding advantage value as the loss function. The calculation formula is as follows:

[0035]

[0036] in, , These are pruning hyperparameters used to limit the magnitude of policy updates.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention proposes a Region-Aware Group Relative Policy Optimization (RA-GRPO) framework. By introducing a spatial mask reward mechanism, policy optimization is localized to the detected hand region, achieving targeted anatomical structure optimization while maintaining global image quality.

[0039] This invention designs a Keypoint Confidence Reward (KCR) based on hand pose estimation. It uses HRNet to predict hand keypoint heatmaps and calculates confidence scores to quantitatively evaluate the structural correctness of the generated hand. In each reinforcement learning iteration, the model generates N images for each prompt text, obtains a reward score through a reward model, calculates the advantage value within the same prompt group using normalization, and finally obtains a region-aware policy loss through mask weighting. Training results show that this method can reduce the probability of deformed generated hands. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the GRPO training framework for region awareness according to the present invention.

[0041] Figure 2 This is a schematic diagram of the MPJPE calculation process. Detailed Implementation

[0042] The present invention will be further described below through specific embodiments.

[0043] This invention provides an optimized method for generating hands of multiple people in text-to-image synthesis based on region awareness. The specific steps are as follows:

[0044] Step 1: Dataset Construction

[0045] This embodiment constructs a multi-person hand gesture dataset, MultiHand-11K. The specific steps are as follows:

[0046] First, GPT-4o is used to generate diverse descriptors covering subject features P (e.g., "a young woman"), clothing attributes A (e.g., "black leather jacket"), and hand gestures H (e.g., "OK", "hi"). The number of people in each image is then calculated. Limited to 1-4 people, these descriptors are randomly sampled and combined to systematically construct prompt text. The text template is as follows: "There are..." person in the image. P wearing A is giving a hand gestureH...".

[0047] To enhance data diversity, two different models, FLUX.1-dev and Gemini 2.5 Pro, were used to generate images. After generation, the images were manually screened to remove those with obvious distortions in the hands or body parts, resulting in the final image dataset. This dataset was then divided into training and test sets according to a set ratio for use.

[0048] Step 2, Monitor and fine-tune

[0049] The basic diffusion model was fine-tuned under supervision using the constructed MultiHand-11K dataset, enabling it to initially generate hands from multiple figures. The specific steps are as follows:

[0050] Using Stable Diffusion 3 as the base model, we load its pre-trained weights on large image-text pairs. Supervised fine-tuning is performed using image-text pairs from the MultiHand-11K training set. For each training sample, a text cue is used as conditional input, and the model generates the corresponding image through a denoising process. The standard training objective of the diffusion model is adopted, namely the mean squared error loss function between the predicted added noise and the model's predicted noise.

[0051]

[0052] in For the velocity field predicted by the model, For the actual target velocity field, For the potential representation of the image, This is the time step. After supervised fine-tuning, the model has initially mastered the ability to generate images containing multiple people based on text descriptions, and can generate basically reasonable hand structures, providing good initialization parameters for subsequent region-aware reinforcement learning optimization.

[0053] Step 3: Optimization of Region Awareness Strategy

[0054] After supervised fine-tuning, the model initially demonstrated the ability to generate multi-person images. To optimize for hands in the images, we designed a region-aware reinforcement learning optimization strategy. During reinforcement learning training, such as... Figure 1 As shown, in each iteration, the model generates N images for each prompt text by denoising N different initial noise vectors. Then, the hand detection module identifies hand regions in the generated images, generates corresponding spatial masks, and crops out each hand region. For each cropped hand region, a hand reward score is obtained using a hand scoring model, and the reward values ​​are standardized within the same prompt group to calculate the advantage score for each hand region. The formula for calculating the advantage score of a hand region is as follows:

[0055]

[0056] In the formula, The reward value for the i-th hand region corresponds to the hand. , where n is the number of hands generated within the same prompt group.

[0057] The spatial mask corresponding to the hand is downsampled to match the resolution of the VAE latent representation space, resulting in a downsampled mask. Then, according to the transition log probability calculation method of the diffusion model, the downsampled mask is multiplied element-wise with the transition log probability to obtain the localized transition log probability. Based on the localized transition log probability and the corresponding advantage value, the policy loss is calculated to update the model parameters. The transition log probability is calculated using the following formula:

[0058]

[0059] in, For the potential state sampled at the next time step, For the deterministic next state predicted by the model, To explore the standard deviation of the injected noise, the localized transition log probability is calculated using the following formula:

[0060]

[0061] in, This is the mask after downsampling. This indicates element-wise multiplication.

[0062] For each hand quality reward value, the CNN-based hand pose estimator HRNet [Wang, Jingdong, et al. "Deep high-resolution representation learning for visual recognition." IEEE transactions on pattern analysis and machine intelligence 43.10 (2020): 3349-3364] is used to predict the heatmap of each keypoint. Keypoint confidence scores are extracted from the heatmaps, and the hand quality reward value is calculated based on the average of the confidence scores of each keypoint. The calculation formula is as follows:

[0063]

[0064] in, For the first Confidence level of key points on the hand.

[0065] In a specific embodiment, the present invention constructs a multi-person hand gesture dataset MultiHand-11K as described above. First, GPT-4o is used to generate diverse descriptors covering subject features (such as "An old woman", "A cutekid"), clothing attributes (such as "orange knitted sweater", "yellow raincoat"), and hand gestures (such as "OK", "fist", "five").

[0066] To enhance data diversity, images were generated using two different models: FLUX.1-dev and Gemini 2.5 Pro. After generation, images were manually screened to remove those with significant distortions in hand or body parts. A total of 11,530 high-quality images were obtained, with 6,103 from FLUX.1-dev and 5,427 from Gemini. Text summaries for all images were automatically generated using Gemini 2.5 Pro. In the experiments, 6,530 image-text pairs were used as the training set, and 5,000 image-text pairs were used as the test set.

[0067] After LORA-supervised fine-tuning, GRPO training was performed. In each iteration, for each c c c, the text-to-image model generated 8 images by denoising 8 different initial noise vectors. The MediaPipe hand detection module [Lugaresi, Camillo, et al. "Mediapipe: A framework for building perceptionpipelines." arXiv preprint arXiv:1906.08172 (2019)] was used to identify hand regions in the generated images, generate corresponding binary space masks, and crop out each hand region. In the binary mask, the pixel value of the hand region is 1, and the pixel value of the non-hand region is 0. For each cropped hand image A hand pose estimator based on HRNet was used to predict heatmaps of 21 hand keypoints. The intensity value on each heatmap represents the probability of the corresponding keypoint's presence. The maximum confidence score for each keypoint was extracted from the heatmap. The average value is used to obtain the hand quality bonus value.

[0068] To verify the effectiveness of this invention, evaluation metrics included H-FID, H-KID, HandConf, and MPJPE. MPJPE was calculated by determining the average Euclidean distance difference between 21 keypoints in the generated image and the 21 keypoints in the correct hand model that best approximates the generated hand. Figure 2 As shown, key points in the generated image are extracted from the generated image using the MediaPipe hand detection model. Correct hand modeling key points are obtained by reconstructing a correct 3D hand mesh using MeshGraphormer [Lin, Kevin, Lijuan Wang, and Zicheng Liu. "Mesh graphormer." Proceedings of the IEEE / CVF international conference on computer vision. 2021.]. The calculation formula is as follows:

[0069]

[0070] in To detect the x and y coordinates of 21 key points in the generated image, The horizontal and vertical coordinates of 21 key points are used to correctly model the hand.

[0071] The experimental results are evaluated as shown in Table 1:

[0072] Table 1

[0073] 。

Claims

1. An optimized method for generating hands of multiple people in text-to-image synthesis based on region awareness, characterized in that, Includes the following steps: Step 1: Construct a multi-person hand gesture dataset, containing multiple image-text pairs; Step 2: Use a multi-person hand gesture dataset to perform supervised fine-tuning of the basic diffusion model, so that it initially has the ability to generate hands for multiple people; Step 3: Perform group relative policy optimization (GRPO) reinforcement learning on the supervised fine-tuned model; In each iteration, for each prompt text, multiple images are generated by denoising multiple initial noise vectors; The hand detection module identifies the hand region in the generated image and generates a corresponding spatial mask. For each identified hand region, a hand quality reward value is obtained through a hand reward model; Images generated with the same text but different initial noise are considered as the same cue group. Within the same cue group, all hand quality reward values ​​are normalized, and the dominance value of each hand region is calculated. The spatial mask is downsampled to match the spatial resolution of the latent representation of the variational autoencoder; The reverse diffusion process based on the diffusion model is used to calculate the transition log probability. The downsampled mask is multiplied element-wise with the transition log probability to obtain the localized transition log probability; The strategy loss is calculated based on the localized transition log probability and the corresponding advantage value. The model parameters are updated by multiplying the localized transition log probability and the corresponding advantage value as the loss function, thereby improving the model's performance in hand generation.

2. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step one, the method for constructing a multi-person hand gesture dataset includes: A large language model is used to generate diverse descriptors covering subject features P, clothing attributes A, and hand gestures H; the number of people in each image is also considered. Limited to 1-4 people, these descriptors are randomly sampled and combined to systematically construct prompt text, with the template as follows: "There are..." The image shows a person wearing a hand gesture. The text prompts are: P wearing A is giving a hand gesture; ... After obtaining the first batch of text prompt data, images are generated using an industry-leading generative model. After generation, images are manually selected to remove those with obvious distortions in the hands or body parts, resulting in the final image dataset. This dataset is then divided into training and testing sets according to a set ratio for use.

3. The multi-person hand region perception optimization method according to claim 2, characterized in that, The large language model is GPT-4o, and the generative models include FLUX.1-dev and Gemini 2.5 Pro.

4. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step two, the basic diffusion model is Stable Diffusion3; during supervised fine-tuning, it is trained using LORA, and the mean square error (MSE) between the predicted noise and the actual noise is minimized during training to minimize the difference between the predicted value and the actual value.

5. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step three, the hand reward model is a keypoint confidence reward KCR model based on CNN-based hand pose estimation. For each hand region, a keypoint heatmap is predicted using a hand pose estimator. Keypoint confidence scores are extracted from the heatmap, and the hand quality reward value is obtained by calculating the average of the confidence scores of each keypoint using the following formula: ； in, For the first Confidence level of key points on the hand.

6. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step three, within the same cue group, the dominance value for each hand area is calculated using the following formula: ； in, The reward value for the i-th hand region corresponds to the hand. , where n is the number of hands generated within the same prompt group.

7. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step three, the transition log probability is calculated using the following formula: ； in, For the potential state sampled at the next time step, For the deterministic next state predicted by the model, To explore the standard deviation of injected noise; The log probability of localization transitions is calculated using the following formula: ； in, This is the mask after downsampling. This indicates element-wise multiplication.

8. The multi-person hand region perception optimization method according to claim 1, characterized in that, In step three, the loss function of GRPO adopts the PPO policy loss with pruning, which multiplies the localized transition log probability and the corresponding advantage value as the loss function. The calculation formula is as follows: ； in, , These are pruning hyperparameters used to limit the magnitude of policy updates.

Images