Pedestrian associated feature enhancement method based on controllable image generation

Abstract

The invention discloses a pedestrian correlation feature enhancement method based on controllable image generation, and belongs to the technical field of computer vision, and the method comprises the following steps: S1, obtaining a plurality of pedestrian images with selected target postures based on a to-be-correlated pedestrian image and a pre-trained controllable image generation model; and S2, extracting pedestrian features of the to-be-associated pedestrian image and the plurality of selected target attitude pedestrian images, and performing feature fusion to obtain enhanced features of the to-be-associated pedestrian image. Based on the enhanced features obtained by the method, the accuracy of pedestrian association can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more particularly to a method for enhancing pedestrian-related features based on controllable image generation. Background Art

[0002] Currently, research in the field of person re-identification (Person ReID) technology focuses primarily on improving model architecture and optimizing loss functions, while paying insufficient attention to the person features extracted by the model. However, as the core basis for person matching, the quality of person features directly determines the accuracy of person matching. In practical applications, due to complex backgrounds, occlusion interference, and inherent model defects, the extracted pedestrian features are inevitably mixed with noise, resulting in reduced matching performance. Although existing research has attempted to improve feature representation capabilities through data augmentation (such as using generative adversarial networks to generate images) or local feature enhancement (such as enhancing the head region), these methods have significant limitations: the former requires training with large amounts of task-specific data, while the latter only optimizes specific body parts. Both methods are difficult to generalize to different task scenarios and model architectures, and both require additional training costs. Therefore, how to systematically improve the person re-identification capabilities of various models in any task without relying on additional training has become a bottleneck that needs to be overcome.

[0003] However, existing technologies fail to effectively address the synergistic issues of scarce positive samples and feature noise suppression. On the one hand, multi-view, multi-pose positive samples of the same pedestrian are extremely difficult to obtain in real-world scenarios, resulting in a lack of data foundation for feature aggregation. On the other hand, existing feature aggregation methods (such as direct averaging or weighted fusion) do not fully utilize the statistical properties of pedestrian identity distribution, making it difficult to effectively approximate the true identity center. While diffusion models have shown great potential in the field of controllable image generation, there is still a lack of systematic solutions for how to apply them to pedestrian association tasks, generate high-quality, identity-consistent, diverse positive samples, and design a feature aggregation strategy that adapts to them.

[0004] Therefore, how to provide a pedestrian association feature enhancement method, wherein the enhanced features obtained can improve the accuracy of pedestrian association, is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] In view of this, an object of the present invention is to provide a method for enhancing pedestrian-related features based on controllable image generation.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for enhancing pedestrian-related features based on controllable image generation includes the following steps:

[0008] S1: Based on the pedestrian image to be associated and the pre-trained controllable image generation model, multiple pedestrian images of the selected target posture are obtained;

[0009] S2: extracting pedestrian features from the pedestrian image to be associated and the multiple pedestrian images of the selected target postures, and performing feature fusion to obtain enhanced features of the pedestrian image to be associated.

[0010] Preferably, the pre-trained controllable image generation model is obtained based on the following steps:

[0011] Step 1: Input the target image into the VAE encoder to obtain the implicit features of the target image;

[0012] Adding Gaussian noise to the implicit features of the target image to obtain the noisy implicit features of the target image;

[0013] Inputting the target image into a posture extractor and a posture encoder in sequence to obtain posture features of the target image;

[0014] Inputting a reference image into a VAE encoder to obtain implicit features of the reference image; wherein the target image and the reference image are images of the same person in different poses;

[0015] Step 2: splicing the noisy implicit features of the target image and the posture features of the target image and inputting them into a controllable image generation model;

[0016] Inputting the implicit features of the reference image into a controllable image generation model;

[0017] Step 3: Minimize the gap between Gaussian noise and predicted noise to obtain the pre-trained controllable image generation model; wherein the predicted noise is output by the controllable image generation model.

[0018] Preferably, step 2 further includes:

[0019] The pedestrian features of the reference image are input as K and V into the cross-attention module of the controllable image generation model; wherein the reference image is input into the pedestrian association model to obtain the pedestrian features of the reference image, K represents the key vector, and V represents the value vector.

[0020] Preferably, S1 specifically includes the following steps:

[0021] S11: Inputting the pedestrian image to be associated into the VAE encoder to obtain implicit features of the pedestrian image to be associated;

[0022] Inputting the selected target posture into the posture encoder to obtain the posture characteristics of the selected target posture;

[0023] S12: inputting the posture features of the selected target posture with Gaussian noise added and the implicit features of the pedestrian image to be associated into the pre-trained controllable image generation model to obtain denoised features;

[0024] Inputting the denoised features into a VAE decoder to obtain an image of a pedestrian in a selected target pose;

[0025] S13: Update the selected target posture and repeat S11-S12 continuously to obtain the multiple pedestrian images with the selected target posture.

[0026] Preferably, S12 specifically further includes:

[0027] The pedestrian features of the pedestrian image to be associated are input as K and V into the cross-attention module of the pre-trained controllable image generation model; wherein, the pedestrian image to be associated is input into the pedestrian association model to obtain the pedestrian features of the pedestrian image to be associated, K represents the key vector, and V represents the value vector.

[0028] Preferably, S2 specifically includes the following steps:

[0029] S21: inputting the pedestrian image to be associated into a pedestrian association model to obtain pedestrian features of the pedestrian image to be associated;

[0030] Inputting the plurality of selected target pose pedestrian images into a pedestrian association model to obtain pedestrian features of the plurality of selected target pose pedestrian images;

[0031] S22: Fusing the pedestrian features of the pedestrian image to be associated with the pedestrian features of the multiple selected target posture pedestrian images to obtain enhanced features of the pedestrian image to be associated.

[0032] Preferably, S22 is implemented based on the following formula:

[0033]

[0034] Where, represents the enhanced features of the pedestrian image to be associated; f represents the pedestrian features of the pedestrian image to be associated; f represents the pedestrian features of the i-th selected target posture pedestrian image; represents the adjustment coefficient, which is determined according to the quality of the selected target posture pedestrian image; N represents the number of selected target posture pedestrian images included in the multiple selected target posture pedestrian images; ||·||2 represents L2 regularization.

[0035] Preferably, the controllable image generation model is a denoising diffusion probability model.

[0036] Preferably, the pose extractor is OpenPose or Dwpose.

[0037] Preferably, the pedestrian association model is ResNet50 or ViT.

[0038] It can be seen from the above technical solution that, compared with the prior art, the present invention discloses a pedestrian association feature enhancement method based on controllable image generation. The enhanced features obtained by the present invention can increase the representation ability of pedestrians and improve the accuracy of pedestrian association. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0040] Figure 1 A flowchart of a pedestrian-related feature enhancement method based on controllable image generation provided by the present invention;

[0041] Figure 2 A schematic diagram of the training process of the controllable image generation model provided by the present invention;

[0042] Figure 3 A schematic diagram of obtaining a pedestrian image of a selected target posture provided by the present invention;

[0043] Figure 4 The present invention provides multiple pedestrian images of selected target postures obtained using different pedestrian image datasets. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, an embodiment of the present invention discloses a method for enhancing pedestrian-related features based on controllable image generation, comprising the following steps:

[0046] S1: Based on the pedestrian image to be associated and the pre-trained controllable image generation model, multiple pedestrian images of the selected target posture are obtained;

[0047] It should be noted that each selected target pose pedestrian image and the to-be-associated pedestrian image have the same pedestrian appearance features but different pedestrian pose features.

[0048] In one embodiment, if Figure 2 As shown, the pre-trained controllable image generation model is obtained based on the following steps:

[0049] Step 1: Input the target image into the VAE encoder to obtain the implicit features of the target image;

[0050] Adding Gaussian noise to the implicit features of the target image to obtain the noisy implicit features of the target image;

[0051] Inputting the target image into a posture extractor and a posture encoder in sequence to obtain posture features of the target image;

[0052] Inputting a reference image into a VAE encoder to obtain implicit features of the reference image; wherein the target image and the reference image are images of the same person in different poses;

[0053] It should be noted that the target image and the reference image are provided by pedestrian image datasets (such as Market1501, Occluded-ReID, and SYSU-MM01).

[0054] Step 2: splicing the noisy implicit features of the target image and the posture features of the target image and inputting them into a controllable image generation model;

[0055] Inputting the implicit features of the reference image into a controllable image generation model;

[0056] In one embodiment, step 2 further includes:

[0057] The pedestrian features of the reference image are input as K and V into the cross-attention module of the controllable image generation model; wherein the reference image is input into the pedestrian association model to obtain the pedestrian features of the reference image, K represents the key vector, and V represents the value vector.

[0058] It is understandable that the present invention can only input the noise-added implicit features of the target image and the spliced ​​features of the posture features of the target image, and the implicit features of the reference image into the controllable image generation model to train the controllable image generation model;

[0059] Alternatively, the noise-added implicit features of the target image and the concatenated features of the target image's posture features, the implicit features of the reference image, and the pedestrian features of the reference image may be input into a controllable image generation model to train the controllable image generation model.

[0060] It can be understood that the pedestrian features of the reference image are input as K and V to the cross-attention module of the controllable image generation model to guide image generation.

[0061] Step 3: Minimize the gap between Gaussian noise and predicted noise to obtain the pre-trained controllable image generation model; wherein the predicted noise is output by the controllable image generation model.

[0062] In one embodiment, if Figure 3 As shown, S1 specifically includes the following steps:

[0063] S11: Inputting the pedestrian image to be associated into the VAE encoder to obtain implicit features of the pedestrian image to be associated;

[0064] Inputting the selected target posture into the posture encoder to obtain the posture characteristics of the selected target posture;

[0065] S12: inputting the posture features of the selected target posture with Gaussian noise added and the implicit features of the pedestrian image to be associated into the pre-trained controllable image generation model to obtain denoised features;

[0066] Inputting the denoised features into a VAE decoder to obtain an image of a pedestrian in a selected target pose;

[0067] In one embodiment, S12 specifically further includes:

[0068] The pedestrian features of the pedestrian image to be associated are input as K and V into the cross-attention module of the pre-trained controllable image generation model; wherein, the pedestrian image to be associated is input into the pedestrian association model to obtain the pedestrian features of the pedestrian image to be associated, K represents the key vector, and V represents the value vector.

[0069] It is understandable that the present invention can only input the posture features of the selected target posture with Gaussian noise added and the implicit features of the pedestrian image to be associated into the pre-trained controllable image generation model to obtain denoised features;

[0070] Alternatively, the posture features of the selected target posture, to which Gaussian noise is added, the implicit features of the pedestrian image to be associated, and the pedestrian features of the pedestrian image to be associated are input into the pre-trained controllable image generation model to obtain denoised features;

[0071] The number of denoising iterations set in the present invention is M times (for example, 2000 times);

[0072] The first iteration of denoising, the controllable image generation model outputs feature X T -Z1; where X T represents the Gaussian noise added to the posture feature of the selected target posture; Z1 represents the predicted noise of the first iteration denoising controllable image generation model;

[0073] Second iteration denoising, controllable image generation model output feature XT -Z1-Z2; where Z2 represents the predicted noise of the second iteration of the denoising controllable image generation model;

[0074] The third iteration of denoising, the controllable image generation model outputs feature X T -Z1-Z2-Z3; where Z3 represents the predicted noise of the third iteration of the denoising controllable image generation model; ....

[0076] The Mth iteration denoising, controllable image generation model output feature X T -Z1-Z2-Z3-....-Z M Among them, Z M represents the predicted noise of the M-th iteration denoising controllable image generation model.

[0077] S13: Update the selected target posture and repeat S11-S12 continuously to obtain the multiple pedestrian images with the selected target posture.

[0078] S2: extracting pedestrian features from the pedestrian image to be associated and the multiple pedestrian images of the selected target postures, and performing feature fusion to obtain enhanced features of the pedestrian image to be associated.

[0079] In one embodiment, S2 specifically includes the following steps:

[0080] S21: inputting the pedestrian image to be associated into a pedestrian association model to obtain pedestrian features of the pedestrian image to be associated;

[0081] Inputting the plurality of selected target pose pedestrian images into a pedestrian association model to obtain pedestrian features of the plurality of selected target pose pedestrian images;

[0082] S22: Fusing the pedestrian features of the pedestrian image to be associated with the pedestrian features of the multiple selected target posture pedestrian images to obtain enhanced features of the pedestrian image to be associated.

[0083] In one embodiment, S22 is implemented based on the following formula:

[0084]

[0085] Where, represents the enhanced features of the pedestrian image to be associated; f represents the pedestrian features of the pedestrian image to be associated; f represents the pedestrian features of the i-th selected target posture pedestrian image; represents the adjustment coefficient, which is determined according to the quality of the selected target posture pedestrian image; N represents the number of selected target posture pedestrian images included in the multiple selected target posture pedestrian images; ||·||2 represents L2 regularization.

[0086] In one embodiment, the controllable image generation model is a denoising diffusion probability model, and the denoising diffusion probability model can be selected as DDPM. DDPM uses U-Net as a denoising network to implement iterative denoising of the controllable image generation model.

[0087] In one embodiment, the pose extractor is OpenPose or Dwpose.

[0088] In one embodiment, the pedestrian association model is ResNet50 or ViT.

[0089] The present invention is based on the Stable Diffusion architecture and multiple pedestrian images of selected target postures obtained from different pedestrian image datasets. Figure 4 As shown;

[0090] Finally, the enhanced features obtained by the method of the present invention were compared with the enhanced features obtained by the original methods Clip-ReID, KPR, and SAAI. As shown in Table 1, the experimental results on the Market1501, Occluded-ReID, and SYSU-MM01 datasets show that the enhanced features provided by the present invention improve the mAP accuracy of pedestrian association by 5.1% compared with the enhanced features provided by the original method Clip-ReID, the enhanced features provided by the present invention improve the mAP accuracy of pedestrian association by 10.29% compared with the enhanced features provided by the original method KPR, and the enhanced features provided by the present invention improve the mAP accuracy of pedestrian association by 4.63% compared with the enhanced features provided by the original method SAAI. This verifies that the pedestrian association feature enhancement method based on controllable image generation provided by the present invention effectively enhances the representation ability of the original features and significantly improves the accuracy of pedestrian association. Where rank1 represents the probability that the first one in the obtained ranked matching sequence is the matching target; mAP represents the average precision, which reflects the degree to which the retrieved person ranks at the top of the ranked list of all correct images in the database.

[0091] Table 1

[0092]

[0093] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0094] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pedestrian-related feature enhancement method based on controllable image generation, characterized in that: The following steps are involved: S1: Based on the pedestrian image to be associated and the pre-trained controllable image generation model, multiple pedestrian images of the selected target posture are obtained; S2: extracting pedestrian features from the pedestrian image to be associated and the multiple pedestrian images of the selected target postures, and performing feature fusion to obtain enhanced features of the pedestrian image to be associated.

2. The method for enhancing pedestrian-related features based on controllable image generation according to claim 1, characterized in that: The pre-trained controllable image generation model is obtained based on the following steps: Step 1: Input the target image into the VAE encoder to obtain the implicit features of the target image; Adding Gaussian noise to the implicit features of the target image to obtain the noisy implicit features of the target image; Inputting the target image into a posture extractor and a posture encoder in sequence to obtain posture features of the target image; Inputting a reference image into a VAE encoder to obtain implicit features of the reference image; wherein the target image and the reference image are images of the same person in different poses; Step 2: splicing the noisy implicit features of the target image and the posture features of the target image and inputting them into a controllable image generation model; Inputting the implicit features of the reference image into a controllable image generation model; Step 3: Minimize the gap between Gaussian noise and predicted noise to obtain the pre-trained controllable image generation model; wherein the predicted noise is output by the controllable image generation model.

3. The method for enhancing pedestrian-related features based on controllable image generation according to claim 2, characterized in that: Step 2 specifically includes: The pedestrian features of the reference image are input as K and V into the cross-attention module of the controllable image generation model; wherein the reference image is input into the pedestrian association model to obtain the pedestrian features of the reference image, K represents the key vector, and V represents the value vector.

4. The method for enhancing pedestrian-related features based on controllable image generation according to claim 1, characterized in that: S1 specifically includes the following steps: S11: Inputting the pedestrian image to be associated into the VAE encoder to obtain implicit features of the pedestrian image to be associated; Inputting the selected target posture into the posture encoder to obtain the posture characteristics of the selected target posture; S12: inputting the posture features of the selected target posture with Gaussian noise added and the implicit features of the pedestrian image to be associated into the pre-trained controllable image generation model to obtain denoised features; Inputting the denoised features into a VAE decoder to obtain an image of a pedestrian in a selected target pose; S13: Update the selected target posture and repeat S11-S12 continuously to obtain the multiple pedestrian images with the selected target posture.

5. The method for enhancing pedestrian-related features based on controllable image generation according to claim 4, characterized in that: S12 specifically includes: The pedestrian features of the pedestrian image to be associated are input as K and V into the cross-attention module of the pre-trained controllable image generation model; wherein, the pedestrian image to be associated is input into the pedestrian association model to obtain the pedestrian features of the pedestrian image to be associated, K represents the key vector, and V represents the value vector.

6. The method for enhancing pedestrian-related features based on controllable image generation according to claim 1, characterized in that: S2 specifically includes the following steps: S21: inputting the pedestrian image to be associated into a pedestrian association model to obtain pedestrian features of the pedestrian image to be associated; Inputting the plurality of selected target pose pedestrian images into a pedestrian association model to obtain pedestrian features of the plurality of selected target pose pedestrian images; S22: Fusing the pedestrian features of the pedestrian image to be associated with the pedestrian features of the multiple selected target posture pedestrian images to obtain enhanced features of the pedestrian image to be associated.

7. The method for enhancing pedestrian-related features based on controllable image generation according to claim 1, characterized in that: S22 is implemented based on the following formula: Where, represents the enhanced features of the pedestrian image to be associated; f represents the pedestrian features of the pedestrian image to be associated; f represents the pedestrian features of the i-th selected target posture pedestrian image; represents the adjustment coefficient, which is determined according to the quality of the selected target posture pedestrian image; N represents the number of selected target posture pedestrian images included in the multiple selected target posture pedestrian images; ||·||2 represents L2 regularization.

8. The method for enhancing pedestrian-related features based on controllable image generation according to claim 1, characterized in that: The controllable image generation model is a denoising diffusion probability model.

9. The method for enhancing pedestrian-related features based on controllable image generation according to claim 2, characterized in that: The pose extractor is OpenPose or Dwpose.

10. The method for enhancing pedestrian-related features based on controllable image generation according to claim 3, characterized in that: The pedestrian association model is ResNet50 or ViT.

Images